BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

[Rob Andrews]

Rob,

i did it by your instrunctions,typed and nothing... and btw: i had in crontab correct    when you checked my screens what i posted here .. you must saw it
it looks,that all your setup is broken.Exists anybody whom working masternode except you? I dont know this person.By me nobody.It will be very hard start masternode when doesnt working still.





what happened when ill close controller-win-wallet and again run it? its OK?


thanks

That command you ran is good.  When it results in an empty result, Watchman is working.  I see your IP is working on the list also, so its obviously working.  The trick is to get the exact command that runs in the crontab working from the command line, then ensure its in the crontab, and then every 5 mins when it runs, your watchdog will not squeal out to the network that your node is down.

I dont think our guide is bad.  I think it needs a couple rough edges polished.  Ive already edited it based on Togo and Your issues over the last 2 days.

What we need is someone to come in here and start a new one and give us fresh advice.  Anyone who wants to start a testnet Sanctuary,
run the Wiki.biblepay.org/Create_Masternode Guide, and let us know if you think it is lacking.  We have to also take into account that it was written in English, and we have no Polish translation, so Im sure communication is part of the problem.  Also, keep in mind not everyone will use biblepayd to run the commands, for example, I set up my Cold node using the windows RPC, and used Samba and nano for all my settings changes.

[klondike]

now again working masternode= received rewards .... nice

for CZECH+SLOVAK languages i can help
user Slovakia can help with CZECH SLOVAKIA HUNGARY

[Rob Andrews]

Alright, should I mine at the same time or just be connected and synced to the network?

You can mine if you want, no problem, as long as you are synced first.

But now all of a sudden we have a fork.  I dont know if it has anything to do with your nodes coming on? How many did you bring into testnet?

We all need to be synced before I can make any sense out of the test harness forensic commands.

[Rob Andrews]

it looks,that we are stuck on 41100

Yeah I see half the network is on 41116 and the other half on 41293....

Looking...  Not sure why this started all of a sudden, we were going so well other than having a rag tag set of sanctuaries we were staying in sync for a long time.  This happened at 5AM this morning (4 hours ago).

[Rob Andrews]

If you see my post above, I hope it's not my node who broke the network!

Whats your IP, Id like to see if you are banned on any of my 3 nodes?

[Rob Andrews]

The mining node now on block 41309 is 94.130.49.121 or 2a01:4f8:10b:26e3::2

Ah, good news, we had Exactly 50-50 on both chains!  I just resynced one node, and the balance is shifting.  Im observing this very closely to see if the reset of the net syncs up to 41309+...  I just see one node is reorging by itself.
Unfortunately Ive got company stopping in now, I may be gone for a litle while. 

[jaapgvk]

I've been trying to get my masternode up and running again (making sure watchdog is functioning).

My sanctuary wallet was indeed stuck on a certain block and wouldn't sync. But I've reindexed both and it seems I'm on the right chain right now (both wallets are on the same block).

Controller wallet:

Code Select Expand



{
  "version": 1000506,
  "protocolversion": 70708,
  "walletversion": 61000,
  "wallet_fullversion": "1.0.5.6",
  "balance": 38514484.00745234,
  "privatesend_balance": 0.00000000,
  "blocks": 41330,
  "timeoffset": -2,
  "connections": 3,
  "proxy": "",
  "difficulty": 0.1980807012241197,
  "testnet": true,
  "keypoololdest": 1507285625,
  "keypoolsize": 1001,
  "paytxfee": 0.00000000,
  "relayfee": 0.00010000,
  "errors": ""
}


Sanctuary wallet:

Code Select Expand

{
  "version": 1000508,
  "protocolversion": 70708,
  "walletversion": 61000,
  "wallet_fullversion": "1.0.5.8",
  "balance": 0.00000000,
  "privatesend_balance": 0.00000000,
  "blocks": 41330,
  "timeoffset": 0,
  "connections": 8,
  "proxy": "",
  "difficulty": 0.1980807012241197,
  "testnet": true,
  "keypoololdest": 1508425662,
  "keypoolsize": 1001,
  "paytxfee": 0.00000000,
  "relayfee": 0.00010000,
  "errors": ""
}


The only thing now is that the 'masternodelist' command gets different results on my controller and sanctuary wallet.

Controller wallet:

Code Select Expand


17:21:14

{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "NEW_START_REQUIRED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "ENABLED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "WATCHDOG_EXPIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "NEW_START_REQUIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "ENABLED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "ENABLED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "ENABLED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "ENABLED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "ENABLED"
}


Sanctuary wallet:

Code Select Expand


{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "NEW_START_REQUIRED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "ENABLED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "NEW_START_REQUIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "NEW_START_REQUIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "NEW_START_REQUIRED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "NEW_START_REQUIRED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "NEW_START_REQUIRED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "ENABLED"
}


I am 'a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4' btw.

[Rob Andrews]

haha no worries. It's 3 am so I won't be up for long too.

Also,  I only brought the two nodes I talked about in my 2 previous posts above. 1 mining and 1 non-mining that's it.

This is what happened around the time of the "fork" so I'm wondering if it could have been me.

Cool, I took a look at that log, and I can see how that particular node would go off on its own and ban everyone.  In this case, it did not have the same mncache.dat view that everyone else had, so its mnpayment schedule was different, and since we enforce masternode recipients, it was not going to agree on the next block (because it couldnt fill out a masternode while the supermajority could). The question is how did it get all synced with a different view of mncache?  Probably because of the headers it rejected it was mining on a fork or something earlier.

The other thing is, if we have a supermajority that is synced with synced masternode data, a couple bad fraudulently compiled nodes with hosed up integrated business logic compiled in should not be able to take us down.  Of course with the fragility of testnet, having only 4 synced masternodes, its possible to hose us up if you bring just 2 bad nodes online. 

Either way lets move past this one and wait til you have the checkblock(1) error showing and then maybe that will shed light on why your headers were rejected. 

[Rob Andrews]

I've been trying to get my masternode up and running again (making sure watchdog is functioning).

My sanctuary wallet was indeed stuck on a certain block and wouldn't sync. But I've reindexed both and it seems I'm on the right chain right now (both wallets are on the same block).

Controller wallet:

Code Select Expand



{
  "version": 1000506,
  "protocolversion": 70708,
  "walletversion": 61000,
  "wallet_fullversion": "1.0.5.6",
  "balance": 38514484.00745234,
  "privatesend_balance": 0.00000000,
  "blocks": 41330,
  "timeoffset": -2,
  "connections": 3,
  "proxy": "",
  "difficulty": 0.1980807012241197,
  "testnet": true,
  "keypoololdest": 1507285625,
  "keypoolsize": 1001,
  "paytxfee": 0.00000000,
  "relayfee": 0.00010000,
  "errors": ""
}


Sanctuary wallet:

Code Select Expand

{
  "version": 1000508,
  "protocolversion": 70708,
  "walletversion": 61000,
  "wallet_fullversion": "1.0.5.8",
  "balance": 0.00000000,
  "privatesend_balance": 0.00000000,
  "blocks": 41330,
  "timeoffset": 0,
  "connections": 8,
  "proxy": "",
  "difficulty": 0.1980807012241197,
  "testnet": true,
  "keypoololdest": 1508425662,
  "keypoolsize": 1001,
  "paytxfee": 0.00000000,
  "relayfee": 0.00010000,
  "errors": ""
}


The only thing now is that the 'masternodelist' command gets different results on my controller and sanctuary wallet.

Controller wallet:

Code Select Expand


17:21:14

{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "NEW_START_REQUIRED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "ENABLED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "WATCHDOG_EXPIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "NEW_START_REQUIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "ENABLED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "ENABLED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "ENABLED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "ENABLED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "ENABLED"
}


Sanctuary wallet:

Code Select Expand


{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "NEW_START_REQUIRED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "ENABLED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "NEW_START_REQUIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "NEW_START_REQUIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "NEW_START_REQUIRED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "NEW_START_REQUIRED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "NEW_START_REQUIRED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "ENABLED"
}


I am 'a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4' btw.

Yeah, I have the same problem.  It looks like everyone who reindexed the chain today lost all their masternode data (stored in mncache.dat), all the governance objects are getting revoted on now etc.  So basically the masternode state is very bad in testnet.  We will have to wait a while to see if our 5 masternodes switch back to enabled. 

[klondike]

all 5 MNodes is ENABLED but we are STUCKED? 41124

[Rob Andrews]

all 5 MNodes is ENABLED but we are STUCKED? 41124

All my 3 nodes rolling at 41534 and synced; status = ENABLED, try on your bad node, try going to peers and ban anyone with a low height, and see if it syncs, if not -reindex.

[Rob Andrews]

I got some errors for you 

https://pastebin.com/NaHRQBi7

Also I'm not sure if you saw it but this is what I got on mainnet just before switching that box to testnet

Oh good, thanks .

Oh I see its in mainnet, so that should be easy to research, hopefully.

Will check it out right now.

[Rob Andrews]

I got some errors for you 

https://pastebin.com/NaHRQBi7

Also I'm not sure if you saw it but this is what I got on mainnet just before switching that box to testnet

Thanks, this is extremely helpful as we went out of sync in testnet a couple times, like I said I saw this error a few times in prod in random logs, but it was so elusive I couldnt reproduce it.  Luckily, it looks like our advanced forensic logging yielding some info (knock on wood).  Lets hope we can get to the bottom of this in only a few passes.  Id hate to see this take a month to fix.

Anyway heres what I am seeing.  On the extremely bright side, I am generating the exact same biblehashes as your node was at the time of the problem.  Thats good, because I didnt want to see some type of non-deterministic biblehash.  So to follow me on that part of it, the biblehash requires 4 inputs, for example your first checkblock error requires me to run this command:


exec biblehash 333e1c9e6fc52027012f734b2b701a3325591759ec6f29686879faae98a4b585 1510797674 1510797171 16911
yielding biblehash: 0000000000072c18e0c42605b5431bd2e2343ced7942038c6a39f76ffa077225

Which matches yours, so that is great.

Next I checked the bits in that block that yields the difficulty and therefore the targethash which matches your targethash in the checkblock message. 

Here is where it gets interesting.  If you convert the hex to decimal (if you need to unless you can do this in your head) the biblehash with a 7a7 prefix must be Lower than the uint256 target (947e).  The 7a7 is 1959 decimal and the 947e is 38014, meaning that it passes. 

So basically what I am saying here is your arithmetic library is failing to calculate the less than operator here (LOL).  I know it sounds strange, now Bitcoin is using something called UintToArith256  to convert the uint256 big number to an arithmetic vector, allowing the math problem to be performed.  So of course that leads me to wonder how can the node function at all if that less than is failing? LOL.  So I move on and I tested the next two examples.  Same issue:  677a (26490) is less than (43096) meaning that the test should have passed, and the block was good and the error was a bad call.
Next 72c1 < 7e28: same issue, block was good, bad call.

So what Im going to do now is Ill add some more code to store the biblehash before its compared, and then compare it, then store it again and log both the original and the compared values in the log, then we can see if somehow, your biblehash changes between the hash and the comparison.  Because obviosly you are doing math for all other blocks. 

Alright Ill be back in about a half hour with the new version.

Good Job.

[Rob Andrews]

Welp that is indeed really strange since it's not failing on every block! There must be something special about these numbers

It will be interesting to see what the issue is/was.

Yeah, it will have to reveal itself as either a vector problem in the biblehash accessor, or a problem in that arithmetic lib, or it could even be a BOOST lib issue, maybe one particular boost lib in your version being manipulated a certain way before the call, Oh who knows

Anyway, I just pushed out 1.0.6.0 if you want to run that for a while until we get more examples. 

[Rob Andrews]

Doing that now, would like me to run on testnet or mainnet?

Mainnet is probably better.