Thank you, Rob. Perfect timing: I am going to try to resync using blocks.zip since the chain on 2/10 of my sancs on the temple has been halted >12 hours.
Output of `getblockcount` for each sanc
1. 444333
2. 444333
3. 444333
4. 444086 <---
5. 444333
6. 444333
7. 444333
8. 444333
9. 444159 <---
10. 444333
Mode(s) of the dataset: 444333
Sancs 4 and 9 are halted since 7:36 this morning. Time now 20:45
However, the 2 sancs in question report being ok:
Sanc 4
{
"AssetID": 999,
"AssetName": "MASTERNODE_SYNC_FINISHED",
"AssetStartTime": 1693214535,
"Attempt": 0,
"IsBlockchainSynced": true,
"IsSynced": true
}
and
"PoSePenalty": 0,
"PoSeBanHeight": -1,
"revocationReason": 0,
Sanc 9
{
"AssetID": 999,
"AssetName": "MASTERNODE_SYNC_FINISHED",
"AssetStartTime": 1693213491,
"Attempt": 0,
"IsBlockchainSynced": true,
"IsSynced": true
}
and
"PoSePenalty": 0,
"PoSeBanHeight": -1,
"revocationReason": 0,
I only checked as you previously mentioned a similar issue. I don't remember any of my sancs going out of sync with each other previously, but I did not have this many sancs before and certainly not all on one VPS.
Furthermore, Sanc 4 mined blocks at 00:35 and 12:16 (the former maybe it was in sync but the later, I recorded that it was AFTER it was halted and lagged by at least 50 block at 7:35) and Sanc 9 mined blocks at 07:34 and 18:35 (I am sure that both times, Sanc 9 was out of sync as I recorded it was lagging at least by 50 blocks at 08:16).
Is there any mechanism to ensure sancs in a temple are in sync with one another? I caught this as I have a script (attached) to find the mode (most common ie consensus) height of the sancs in the temple and then flag any that deviate from this by at least 50 blocks.
Blessings,
oncoapop
Hi Bro Oncoapop,
Glad you mentioned this; I observed the same conditions on my sancs (10%-15% halted at around the same block numbers, chain wont continue).
There is a lot of related detail to this answer. First of all, the chain is healthy. If a sanc gets halted at a block, that means its view of the chain wont allow it to continue due to InstantSend rules.
Since we have deterministic sancs enabled, LLMQ on, chainlocks On, instantsend On, there is a very high level of integrity to the chain itself (you cant add a block to the chain unless it passes the chainlocks rules for the whole supermajority).
However, an instantsend transaction that is 'questionable' can actually stop the chain on an individual sanc, if that sanc feels it is a dangerous lock.
I believe this is what happened because of Talismans transaction (certain sancs had it in memory) and when they get resyned with -erasechain=1, finally they are purged of the memory.
The other thing is with this Investor mode, we have a high level of sancs that are not keeping the quorums (because they are confused who to talk to), so without quorums, we have this propensity of not agreeing on IX tx's; when our quorums are "on", we all advance in lock step.
One other problem we have right now is since LLMQ quorums arent sticking, half of our investors are being paid full block rewards (in contrast to 50%).
Thankfully I believe there is a solution to all this without a mandatory upgrade. Im releasing a mandatory upgrade for the sancs later today that should make the sancs lock in the LLMQs and place investors in their investor brackets more accurately; then we can reassess the situation. The telltale sign is to check our LLMQ quorums over a couple weeks and see if we stay in quorum mode.
In the mean time, just relaunch any sanc that is out of sync with -erasechain=1.
Both chainz and SX are not affected.
Thanks!
