BiblePay | 10% to Orphan-Charity | RANDOMX MINING | Sanctuaries (Masternodes)

Thank you, Rob. Perfect timing: I am going to try to resync using blocks.zip since the chain on 2/10 of my sancs on the temple has been halted >12 hours.

Output of `getblockcount` for each sanc
1.   444333
2.   444333
3.   444333
4.   444086 <---
5.   444333
6.   444333
7.   444333
8.   444333
9.   444159 <---
10.   444333

Mode(s) of the dataset: 444333

Sancs 4 and 9 are halted since 7:36 this morning. Time now 20:45

However, the 2 sancs in question report being ok:

Sanc 4
{
  "AssetID": 999,
  "AssetName": "MASTERNODE_SYNC_FINISHED",
  "AssetStartTime": 1693214535,
  "Attempt": 0,
  "IsBlockchainSynced": true,
  "IsSynced": true
}
and     
    "PoSePenalty": 0,
    "PoSeBanHeight": -1,
    "revocationReason": 0,

Sanc 9
{
  "AssetID": 999,
  "AssetName": "MASTERNODE_SYNC_FINISHED",
  "AssetStartTime": 1693213491,
  "Attempt": 0,
  "IsBlockchainSynced": true,
  "IsSynced": true
}
and
    "PoSePenalty": 0,
    "PoSeBanHeight": -1,
    "revocationReason": 0,

I only checked as you previously mentioned a similar issue. I don't remember any of my sancs going out of sync with each other previously, but I did not have this many sancs before and certainly not all on one VPS.

Furthermore, Sanc 4 mined blocks at 00:35 and 12:16 (the former maybe it was in sync but the later, I recorded that it was AFTER it was halted and lagged by at least 50 block at 7:35) and Sanc 9 mined blocks at 07:34 and 18:35 (I am sure that both times, Sanc 9 was out of sync as I recorded it was lagging at least by 50 blocks at 08:16).

Is there any mechanism to ensure sancs in a temple are in sync with one another? I caught this as I have a script (attached) to find the mode (most common ie consensus) height of the sancs in the temple and then flag any that deviate from this by at least 50 blocks.

Blessings,
oncoapop

Hi Bro Oncoapop,

Glad you mentioned this; I observed the same conditions on my sancs (10%-15% halted at around the same block numbers, chain wont continue).
There is a lot of related detail to this answer.  First of all, the chain is healthy.  If a sanc gets halted at a block, that means its view of the chain wont allow it to continue due to InstantSend rules.
Since we have deterministic sancs enabled, LLMQ on, chainlocks On, instantsend On, there is a very high level of integrity to the chain itself (you cant add a block to the chain unless it passes the chainlocks rules for the whole supermajority).
However, an instantsend transaction that is 'questionable' can actually stop the chain on an individual sanc, if that sanc feels it is a dangerous lock.
I believe this is what happened because of Talismans transaction (certain sancs had it in memory) and when they get resyned with -erasechain=1, finally they are purged of the memory.
The other thing is with this Investor mode, we have a high level of sancs that are not keeping the quorums (because they are confused who to talk to), so without quorums, we have this propensity of not agreeing on IX tx's; when our quorums are "on", we all advance in lock step.
One other problem we have right now is since LLMQ quorums arent sticking, half of our investors are being paid full block rewards (in contrast to 50%).

Thankfully I believe there is a solution to all this without a mandatory upgrade.  Im releasing a mandatory upgrade for the sancs later today that should make the sancs lock in the LLMQs and place investors in their investor brackets more accurately; then we can reassess the situation.  The telltale sign is to check our LLMQ quorums over a couple weeks and see if we stay in quorum mode.

In the mean time, just relaunch any sanc that is out of sync with -erasechain=1. 
Both chainz and SX are not affected.

Thanks!

BiblePay 0.17.4.2
Mandatory Upgrade for Sanctuaries and Home Controller Wallets

- Add more robust POSE ban for investors (keep our quorums intact) and keep investor payments at 50%
- Add Investor pose status to masternodelist UI and masternodelist rpc command
- Added exec versionreport so we can see the % that have upgraded

** Note: On your self compiled sanctuary, you can simply type "git pull origin master && make -j4" to upgrade.  Then stop and restart all instances.

Blast from the Past


We used to have an RPC command back in mid 2019 called "Version Report".  It was useful in determining how much of the network upgraded from one version of BBP to another.

I decided to throw that back in the code as it is useful right now to see when we get a supermajority of the sancs mining on v0.17.4.2 (meaning that the Investor payments will be reduced to 50% when this number gets closer to 90%+).

This report was just released in the latest version (above), but for now, I ran it:


15:32:51
exec versionreport


15:32:51
{
  "Command": "versionreport",
  "version_report": {
    "Version": "Popularity,Percent %",
    "1737": "100; 48.78%",
    "1738": "40; 19.51%",
    "1741": "65; 31.71%"
  }
}


This means that a supermajority of the sancs are still mining on old versions.

PS We actually need 70%+ to upgrade to 0.17.4.2 for the investor pay distribution feature to work correctly.

The Linux binaries in biblepay-lin64.zip appear to only have  BiblePay.BMSD


 

BiblePay 0.17.4.2
Mandatory Upgrade for Sanctuaries and Home Controller Wallets

- Add more robust POSE ban for investors (keep our quorums intact) and keep investor payments at 50%
- Add Investor pose status to masternodelist UI and masternodelist rpc command
- Added exec versionreport so we can see the % that have upgraded

** Note: On your self compiled sanctuary, you can simply type "git pull origin master && make -j4" to upgrade.  Then stop and restart all instances.

BiblePay 0.17.4.2
Mandatory Upgrade for Sanctuaries and Home Controller Wallets

- Add more robust POSE ban for investors (keep our quorums intact) and keep investor payments at 50%
- Add Investor pose status to masternodelist UI and masternodelist rpc command
- Added exec versionreport so we can see the % that have upgraded

** Note: On your self compiled sanctuary, you can simply type "git pull origin master && make -j4" to upgrade.  Then stop and restart all instances.

Hi Rob,

Since version 0.17.4.1 my Sanctuary keeps going to invest , i revieveSanc then its fine for  a bit and then after a while it goes back to been Invest, any ideas?

Thanks

Hi Rob,

Since version 0.17.4.1 my Sanctuary keeps going to invest , i revieveSanc then its fine for  a bit and then after a while it goes back to been Invest, any ideas?

Thanks

Hi bro Budinga, hope you have been good!

Could you give me the sanc IP?
You can try this, from the home IP type:
telnet your_sanc_ip your_sanc_port
And see if it answers?

If other nodes can see your sanc it should stay in non investor state.

Oh btw, you have to allow whatever your port is through your VMS firewall too (as a sanc). 

The Linux binaries in biblepay-lin64.zip appear to only have  BiblePay.BMSD

Thanks bro Oncoapop, I am ready to release 0.17.4.3 now, and will check that and announce it shortly.

BiblePay - 0.17.4.4
Mandatory upgrade for Sanctuaries (Leisure Upgrade for Users)
- Ensure investor blocks are always 50% (+7 bbp)
- Help revive our LLMQ Quorums

Hi bro Budinga, hope you have been good!

Could you give me the sanc IP?
You can try this, from the home IP type:
telnet your_sanc_ip your_sanc_port
And see if it answers?

If other nodes can see your sanc it should stay in non investor state.

Oh btw, you have to allow whatever your port is through your VMS firewall too (as a sanc).

Hi Rob,

I am doing great i hope you are too my friend.

Maybe it was my norton, i had it on auto i just set BBP to allow all. and run the command will see how it goes.

Hi Rob,

I am doing great i hope you are too my friend.

Maybe it was my Norton, i had it on auto i just set BBP to allow all. and run the command will see how it goes.

It is definitely doing it again. firewall is completely open and i also forwarded the port to my pc just to check. But for some reason i cannot telnet in to it anymore, since the update.

It is definitely doing it again. firewall is completely open and i also forwarded the port to my pc just to check. But for some reason i cannot telnet in to it anymore, since the update.

Seems fine over here.  Just let me know if you need me to help.

Seems fine over here.  Just let me know if you need me to help.

Ok maybe its my Norton blocking the ports ill have a play with it next week as i don't have the time at the moment.
But 100% i cant telnet in to it.. very strange this just started happening.

Ok maybe its my Norton blocking the ports ill have a play with it next week as i don't have the time at the moment.
But 100% i cant telnet in to it.. very strange this just started happening.

Yeah, the prereq to have an active node is to definitely be able to telnet to it from the outside world.  Heres my config for one of my sancs on a nonstandard port that may help clear this up, in your biblepay.conf:
port=10002
rpcport=50002


This one listens on port 10002 instead of 40000, and the reason for setting rpcport is to simply make them distinct per sanc.
Then I also do the 'sudo ufw allow 10002/tcp' on my sanc and now I can telnet to it from my home machine when its running and it doesnt get pose banned.

** Update for Mining Sanctuaries that have Lagging Blocks **


So I've been pulling my hair out trying to find out why 'sometimes' (and this also was reported by Oncoapop recently) out of 10 nodes in a Temple, 1 or 2 will start lagging behind on the block count and eventually they need restarted with -erasechain=1 (that was the recommended current solution at least).

Today I finally got the raw logs from one sanc that was doing that and I found part of the root cause:
2023-09-15T23:47:39Z CChainLocksHandler::EnforceBestChainLock -- CLSIG (CChainLockSig(nHeight=447722, blockHash=b6202fdf5ffdf1b315577a2d4ab53f141212f2b8652b7aaff426d781c956bff9)) marked block 494b4e05928fd613d1d831c40fbeb3c2af0044333e793c5ae09e15fbfcddc91e as conflicting
2023-09-15T23:47:39Z ConflictingChainFound: conflicting block=494b4e05928fd613d1d831c40fbeb3c2af0044333e793c5ae09e15fbfcddc91e  height=447720  log2_work=60.28583982  date=2023-09-15T23:40:31Z
2023-09-15T23:47:39Z ConflictingChainFound:  current best=dd261ab9c7215d3e4acde9d8b4fa2e24c7dccd822b294734798b8f406220b81d  height=447717  log2_work=60.28583982  date=2023-09-15T23:25:08Z
2023-09-15T23:47:39Z CChainLocksHandler::EnforceBestChainLock -- CLSIG (CChainLockSig(nHeight=447722, blockHash=b6202fdf5ffdf1b315577a2d4ab53f141212f2b8652b7aaff426d781c956bff9)) marked block 494b4e05928fd613d1d831c40fbeb3c2af0044333e793c5ae09e15fbfcddc91e as conflicting


In llmq for chainlocks we go round robin around all the sancs and form quorums.  When the chainlocks is Active, blocks must be enforced by the whole quorum for the best chain.  Now we have a unique scenario where the actual sancs are mining also.  So if a sanc finds two blocks in a row, I believe quicker than those can be locked by the rest of the network, while another sanc solves a block at the same height that gets locked before the first sanc, it ends up in an internal conflicting state (its pindex status for the conflicting blocks keep getting stored in memory as a conflict and does not get updated).  The real question is why does it not reorganize when the main chain grows larger (the chainlocked chain).  That part is still is not solved.

But I have a partial solution for now:

If this happens to your node, you can simply stop it with './biblepay-cli -conf=/configs/yournodenumber.conf stop\r\n./biblepayd -conf=/configs/yournodenumber.conf &' to stop and start the node, and when it restarts, the conflicting block will be purged from ram and it will sync to the tip (which is a big relief because I did not want to recommend resyncing the chain in this case as that gets old pretty quick).

I will keep looking into the root cause to see if we can make the node recover when this happens.

Hey All,
Whoever wants to participate in our next testnet release, please check it out here:

https://forum.biblepay.org/index.php?topic=944.0