Actually it looks like we are encountering a brand new Evo-specific issue as of a new block from yesterday:
2019-04-18 13:59:56 ERROR: ConnectBlock(): ProcessSpecialTxsInBlock for block 4b1205177d60daa173491b742e39501ae29a48605bf3ed617ade77149feb6fb2 failed with bad-protx-collateral (code 16)
It looks like two of us tried to create deterministic sancs before the spork was enabled. But what is interesting is this was legal yesterday but today the block is actually being rejected. (I can understand that partially since all the sancs are disabled right now), but what is interesting to me is that we reject the entire block if the Pro-reg-tx fails.
Let me take a look at this issue in detail. In the mean time I think the only way to sync is to re-sync every node to before 34510 occurred.
Ok. I see whats going on here. At first it seems a little strange that a single bad pro-reg tx could fork the chain (because business logic rejects the entire block that contains the bad pro-reg) -- but thats not whats really happening in the bigger picture.
In the bigger picture we will always have a supermajority of sancs online in prod to make the call, so with sancs actually online, the pro-reg would have been processed or the sanc-quorum would have rejected the block.
The rule goes like this: For chainlocks, llmqs, and pro-reg-special-transactions, all of these are forwarded to a processor that checks validity based on the llmq (long living masternode quorum), and if any fail, the entire block is rejected.
So what we need to do first everyone,
please re-sync your chain to before 34510 and please do not send any 'upgradesanc' commands or pro-regs until we successfully recreate our sanc quorum with 4.5 mm locks. Also just to be safe, lets ensure diff is > 1.0 and we solve the next GSC superblock after that with exec health showing true.
Then we can re-group and talk about testing the next layer(s).
