We were doing great til about block 5000, cat debug.log | grep bits
Something went wrong with DGW. Will check in the morning...
So just to explain whats going on over here, I believe DGW is not a cause but an effect. It looks like the problem started at block 4940 (that is the forkpoint).
Basically, we had 3 sancs online (I think all 3 were mine at the time as it was early in the game).
Sanc #1 had 8 triggers in memory for block 4940 (this was a gsc superblock), and 7 of them had some type of epoch time issue - (that is problem #1; first of all we need to fix this problem - I do have a log of this issue so I will fix this first, meaning the IsSuperblockTriggered will not complain about the event_block_height not matching the start_epoch blockStart).
Next problem, our minimum vote quorum level is 20% net yes in testnet, but this is only .60 in testnet with 3 sancs (we do have a floor of 2 for Creation, but not for Superblock Triggering) another words, this needs beefed up and standardized so that both creation requires a better minimum and also triggering matches the same number (it does not match). In prod this wont be a problem because we will be dealing with a much bigger number, like 80 sanctuaries for example. So I will move the floor to 3 in testnet, and standardize the agreement on the trigger also - this would have prevented Sanc #1 from actually emitting the superblock.
As far as the fork itself, Sanc #1 emitted the superblock, the others rejected it - (I can see from sanc #2s log at 4940 that it says, Superblock Not triggered - normal block) (This is where things get bad), so basically half of the network accepted the superblock, and this forked the chain at 4940, the other 2 sancs banned my sanc, and split the network.
Im also going to move sanc #1 to the cloud so as to standardize the connectivity (as I see a strange DNS problem on sanc #1 also).
As you can see I have a lot of work to do, so it is no longer necessary to continue testing until we have another release.
Thanks for everyones help so far.