TestNet - PODC 2.0 (Proof of Distributed Computing)

I find it very odd that we have 21 testnet nodes, all on the newest version I see.

This last release does not feel right.  After deleting instantsend Im getting different heights now on my sancs.

I'm dissapointed in the latest stable branch.  Right now we are running what Dash prod is running.
Looks like we need to dig deeper into the problem.

After deleting instandsend.dat, and restarting I can only get to height 18380 even after reassesschains.

After deleting instandsend.dat, and restarting I can only get to height 18380 even after reassesschains.

Sorry i meant instantsend.

After deleting instandsend.dat, and restarting I can only get to height 18380 even after reassesschains.

I realized that I reported too soon... reassess got me to a higher height.

Code: [Select]

getblockcount
18380
cli getblockhash 18433
error code: -8
error message:
Block height out of range

cli exec reassesschains
{
 "Command": "reassesschains",
 "progress": 65
}
cli mnsync status
{
 "AssetID": 0,
 "AssetName": "MASTERNODE_SYNC_INITIAL",
 "AssetStartTime": 1574804204,
 "Attempt": 0,
 "IsBlockchainSynced": false,
 "IsSynced": false,
 "IsFailed": false
}

wait for sync to complete.

Code: [Select]

cli mnsync status
{
 "AssetID": 999,
 "AssetName": "MASTERNODE_SYNC_FINISHED",
 "AssetStartTime": 1574807022,
 "Attempt": 0,
 "IsBlockchainSynced": true,
 "IsSynced": true,
 "IsFailed": false
}
cli getblockhash 18433
5e76b6e223aafa7ceb64e7bf187454425ba7856828974f05d104ed41d07c4f9d
Getblockcount
18448


I realized that I reported too soon... reassess got me to a higher height.

Code: [Select]

getblockcount
18380
cli getblockhash 18433
error code: -8
error message:
Block height out of range

cli exec reassesschains
{
 "Command": "reassesschains",
 "progress": 65
}
cli mnsync status
{
 "AssetID": 0,
 "AssetName": "MASTERNODE_SYNC_INITIAL",
 "AssetStartTime": 1574804204,
 "Attempt": 0,
 "IsBlockchainSynced": false,
 "IsSynced": false,
 "IsFailed": false
}

wait for sync to complete.

Code: [Select]

cli mnsync status
{
 "AssetID": 999,
 "AssetName": "MASTERNODE_SYNC_FINISHED",
 "AssetStartTime": 1574807022,
 "Attempt": 0,
 "IsBlockchainSynced": true,
 "IsSynced": true,
 "IsFailed": false
}
cli getblockhash 18433
5e76b6e223aafa7ceb64e7bf187454425ba7856828974f05d104ed41d07c4f9d
Getblockcount
18448


Yeah, Im looking through the heights on my node that has 21 connections and everyone is at a different height, so its not the GSC block at all, it points to LLMQ and IX locks.

Unfortunately what it looks like is something like this:
Half of the sancs were forming their own version of the LLMQ quorum, so they have a different cache of IX locks.
The other half, has a different quorum.
The reason I can tell, is I found I can end up on a different height after erasing the chain twice, and, I can see what height some of the sancs are at by looking them up by IP.

This is interesting because it appears that the chain you end up on depends on which governance data you synced from.

So let me look deeper into the issue.  The answer is probably going to be marking a checkpoint block, and then ensuring our LLMQ all adheres to one chain after we upgrade.

This is much more complicated than plain old POW in bitcoin.

Yeah, Im looking through the heights on my node that has 21 connections and everyone is at a different height, so its not the GSC block at all, it points to LLMQ and IX locks.

Unfortunately what it looks like is something like this:
Half of the sancs were forming their own version of the LLMQ quorum, so they have a different cache of IX locks.
The other half, has a different quorum.
The reason I can tell, is I found I can end up on a different height after erasing the chain twice, and, I can see what height some of the sancs are at by looking them up by IP.

This is interesting because it appears that the chain you end up on depends on which governance data you synced from.

So let me look deeper into the issue.  The answer is probably going to be marking a checkpoint block, and then ensuring our LLMQ all adheres to one chain after we upgrade.

This is much more complicated than plain old POW in bitcoin.

Thank you, Rob.

Whatever you did appeared to bring some stability to the network as sampled by my VPSs on the testnet chain, after deletion of instantsend.dat, exec reassesschains and waiting for mnsync status to finish.

currently, all my clients and sancs appear to be on the same chain

https://oncoapop.sdf.org/biblepaytest/testnet_chainstate.shtml

Thank you, Rob.

Whatever you did appeared to bring some stability to the network as sampled by my VPSs on the testnet chain, after deletion of instantsend.dat, exec reassesschains and waiting for mnsync status to finish.

currently, all my clients and sancs appear to be on the same chain

https://oncoapop.sdf.org/biblepaytest/testnet_chainstate.shtml

I turned off the IX autolocks spork, then sent out a reconsiderlast 100 blocks spork, and now I see that your sanc page matches my hash also.

Lets let it run on this version overnight, until we surpass the highest bad block that these other straggling nodes have.

Ill keep looking into this issue deeper .

I kept trying to erasechain but kept getting to a lower block height than you guys
I had to totally nuke my testnet3 folder (delete everything except the biblepay.conf, deterministic.conf, masternode.conf and wallet.dat files)

So now my 2 nodes and my 1 masternode are all synced with oncoapop
https://oncoapop.sdf.org/biblepaytest/testnet_chainstate.shtml

I kept trying to erasechain but kept getting to a lower block height than you guys
I had to totally nuke my testnet3 folder (delete everything except the biblepay.conf, deterministic.conf, masternode.conf and wallet.dat files)

So now my 2 nodes and my 1 masternode are all synced with oncoapop
https://oncoapop.sdf.org/biblepaytest/testnet_chainstate.shtml

Yeah, the issue is -erasechain=1 does not erase instantsend.dat.
So I added that into the next version.

Cool that you are synced now.

I suppose we need to find reliability in testnet before we consider this version to be stable.

InstantSend (IX) used to say 5 out of 6 confirmations. Now it is just 1 out of 6, but with the key checkbox icon in the column? I notice the transaction line continues to be blue.

Yeah, the issue is -erasechain=1 does not erase instantsend.dat.
So I added that into the next version.

Cool that you are synced now.

I suppose we need to find reliability in testnet before we consider this version to be stable.

Hmm, blockhashes appear to diverge from block 18475 on testchain. No intervention on my part.

Hmm, blockhashes appear to diverge from block 18475 on testchain. No intervention on my part.

at Block 18514,
exec reassesschains
wait for mnsync status to finish

>cli getblockhash 18514
1f280bd0f1bb8796980493b54ed71c7fb5f35d16400ad0e3f0bca7d774f5e93d

So all wallets are back in sync again.

InstantSend (IX) used to say 5 out of 6 confirmations. Now it is just 1 out of 6, but with the key checkbox icon in the column? I notice the transaction line continues to be blue.

That should be OK, there is a separate path in the code for the base confirms - and we can fix the final value, either way the result is the same:
https://forum.biblepay.org/index.php?topic=465.msg6701#msg6701

at Block 18514,
exec reassesschains
wait for mnsync status to finish

>cli getblockhash 18514
1f280bd0f1bb8796980493b54ed71c7fb5f35d16400ad0e3f0bca7d774f5e93d

So all wallets are back in sync again.

I was not awake at that particular height, but looking at my 4 wallets (I have 3 sancs) they all agree with your getblockhash 18514.
So the height you mentioned - 18475 - thats right after the last GSC.

Was it just one of your 3 sancs that went out and required manual intervention, and after that, did all 3 stay in sync?

Yes, we really need to get to the root of this and prevent this entirely, it might require me adding a checkpoint and all of us doing some testing from a certain height forward.

I was not awake at that particular height, but looking at my 4 wallets (I have 3 sancs) they all agree with your getblockhash 18514.
So the height you mentioned - 18475 - thats right after the last GSC.

Was it just one of your 3 sancs that went out and required manual intervention, and after that, did all 3 stay in sync?

Yes, we really need to get to the root of this and prevent this entirely, it might require me adding a checkpoint and all of us doing some testing from a certain height forward.

2/3 sancs went on a parallel chain diverging at the block 18475, in fact both these appeared to be stuck on the previous block for some time after the two (1 sanc and 1 wallet) already accepted the block.

I left them till block 18514 to see if they will correct themselves but they did not and hence manually intervened to reassesschains.

All 3/3 sancs and the controller wallet have been in sync since block 18514.

Togo, the Kanye proposal looks like it will pay this time, please double check @ 18655.

The prior proposal just missed the automatic trigger, but I see the trigger was created this time.