As an addition to this:
This appears to be serious as it appears there are 2 divergent chains!
Each mining pool is on a different divergent chain. The foundation pool is on the chain that the bbp explorer is on, but the miningpool.fun pool is on the chain that my wallet (1.6.2.8 ) is on.
So for one, I take security very seriously, and couldnt possibly imagine having RandomX + Chainlocks in our community and ending up with a sidechain, so I've been vehemently trying to find the root cause of this and I believe I found the root cause and it should be fixed now.
Basically the first part of the explanation is correct; we were transitioning over to chainlocks and the chainlocks started successfully, I can see this because yesterday morning we didnt have blocks locked at the tip and now we have hundreds of blocks locked and we are locked to the tip, so that part is good, the delay was due to normal quorum formation (which takes 2 days from my experience in testnet) and it took about 2 days in prod.
Now the second part of the issue, how we were jittering in and out of the go live and having a side chain form for a few hours (which, to my knowledge is the Only side chain we ever had since randomx started) - I honestly dont remember any sidechain even forming while on randomx because its a very expensive algorithm and it clearly keeps our consensus.
So what I found when debugging my server #6 which has multiple sancs on it was all the instances of BBP went down at the same time (during that height range of the sidechain), and fortunately I was able to find the problem in the debug logs on the server and this leads to a Fix..... The problem is in our memorize prayers function believe it or not. Its hard to fathom how that could cause a problem like this; but here is how. We serialize the prayers based on a filename location that works for Single nodes only. When more than one node instance on a sanctuary tries to write to that file at the same time it can crash the node (and that can cause database corruption because its an immediate non flushed berkeley db exit).
So the fix is now checked in, I have upgraded my sancs. The fix is working (cause I can see the prayers file has a distinct name now per server instance etc) and they are not clashing anymore.
Now we need to let everything run for a week and monitor chainlocks and instantsend and see if chainlocks keeps us on one main chain for long periods of time, and my feeling is that we are going to be successful this time; I think we have fixed our bugs and we have a solid chain to build on.
Obviously I invite as many people to run sancs as possible so we can keep this decentralized; but the issue is you have to sponsor an orphan from cameroon-one to receive full rewards for sancs (or you can run a tier-2 sanc); Please God send us more volunteers and investors who want to be sanctuary owners!
Praise God - I think we are getting back on track again.
On a side note, to my knowledge the 3 hour sidechain resolved itself and transactions went out on both chains anyway (I believe it can be considered a temporary reorganization).
8200 BBP
