BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

[Rob Andrews]

If you see my post above, I hope it's not my node who broke the network!

Whats your IP, Id like to see if you are banned on any of my 3 nodes?

[Rob Andrews]

The mining node now on block 41309 is 94.130.49.121 or 2a01:4f8:10b:26e3::2

Ah, good news, we had Exactly 50-50 on both chains!  I just resynced one node, and the balance is shifting.  Im observing this very closely to see if the reset of the net syncs up to 41309+...  I just see one node is reorging by itself.
Unfortunately Ive got company stopping in now, I may be gone for a litle while. 

[jaapgvk]

I've been trying to get my masternode up and running again (making sure watchdog is functioning).

My sanctuary wallet was indeed stuck on a certain block and wouldn't sync. But I've reindexed both and it seems I'm on the right chain right now (both wallets are on the same block).

Controller wallet:

Code Select Expand



{
  "version": 1000506,
  "protocolversion": 70708,
  "walletversion": 61000,
  "wallet_fullversion": "1.0.5.6",
  "balance": 38514484.00745234,
  "privatesend_balance": 0.00000000,
  "blocks": 41330,
  "timeoffset": -2,
  "connections": 3,
  "proxy": "",
  "difficulty": 0.1980807012241197,
  "testnet": true,
  "keypoololdest": 1507285625,
  "keypoolsize": 1001,
  "paytxfee": 0.00000000,
  "relayfee": 0.00010000,
  "errors": ""
}


Sanctuary wallet:

Code Select Expand

{
  "version": 1000508,
  "protocolversion": 70708,
  "walletversion": 61000,
  "wallet_fullversion": "1.0.5.8",
  "balance": 0.00000000,
  "privatesend_balance": 0.00000000,
  "blocks": 41330,
  "timeoffset": 0,
  "connections": 8,
  "proxy": "",
  "difficulty": 0.1980807012241197,
  "testnet": true,
  "keypoololdest": 1508425662,
  "keypoolsize": 1001,
  "paytxfee": 0.00000000,
  "relayfee": 0.00010000,
  "errors": ""
}


The only thing now is that the 'masternodelist' command gets different results on my controller and sanctuary wallet.

Controller wallet:

Code Select Expand


17:21:14

{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "NEW_START_REQUIRED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "ENABLED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "WATCHDOG_EXPIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "NEW_START_REQUIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "ENABLED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "ENABLED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "ENABLED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "ENABLED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "ENABLED"
}


Sanctuary wallet:

Code Select Expand


{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "NEW_START_REQUIRED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "ENABLED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "NEW_START_REQUIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "NEW_START_REQUIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "NEW_START_REQUIRED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "NEW_START_REQUIRED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "NEW_START_REQUIRED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "ENABLED"
}


I am 'a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4' btw.

[Rob Andrews]

haha no worries. It's 3 am so I won't be up for long too.

Also,  I only brought the two nodes I talked about in my 2 previous posts above. 1 mining and 1 non-mining that's it.

This is what happened around the time of the "fork" so I'm wondering if it could have been me.

Cool, I took a look at that log, and I can see how that particular node would go off on its own and ban everyone.  In this case, it did not have the same mncache.dat view that everyone else had, so its mnpayment schedule was different, and since we enforce masternode recipients, it was not going to agree on the next block (because it couldnt fill out a masternode while the supermajority could). The question is how did it get all synced with a different view of mncache?  Probably because of the headers it rejected it was mining on a fork or something earlier.

The other thing is, if we have a supermajority that is synced with synced masternode data, a couple bad fraudulently compiled nodes with hosed up integrated business logic compiled in should not be able to take us down.  Of course with the fragility of testnet, having only 4 synced masternodes, its possible to hose us up if you bring just 2 bad nodes online. 

Either way lets move past this one and wait til you have the checkblock(1) error showing and then maybe that will shed light on why your headers were rejected. 

[Rob Andrews]

I've been trying to get my masternode up and running again (making sure watchdog is functioning).

My sanctuary wallet was indeed stuck on a certain block and wouldn't sync. But I've reindexed both and it seems I'm on the right chain right now (both wallets are on the same block).

Controller wallet:

Code Select Expand



{
  "version": 1000506,
  "protocolversion": 70708,
  "walletversion": 61000,
  "wallet_fullversion": "1.0.5.6",
  "balance": 38514484.00745234,
  "privatesend_balance": 0.00000000,
  "blocks": 41330,
  "timeoffset": -2,
  "connections": 3,
  "proxy": "",
  "difficulty": 0.1980807012241197,
  "testnet": true,
  "keypoololdest": 1507285625,
  "keypoolsize": 1001,
  "paytxfee": 0.00000000,
  "relayfee": 0.00010000,
  "errors": ""
}


Sanctuary wallet:

Code Select Expand

{
  "version": 1000508,
  "protocolversion": 70708,
  "walletversion": 61000,
  "wallet_fullversion": "1.0.5.8",
  "balance": 0.00000000,
  "privatesend_balance": 0.00000000,
  "blocks": 41330,
  "timeoffset": 0,
  "connections": 8,
  "proxy": "",
  "difficulty": 0.1980807012241197,
  "testnet": true,
  "keypoololdest": 1508425662,
  "keypoolsize": 1001,
  "paytxfee": 0.00000000,
  "relayfee": 0.00010000,
  "errors": ""
}


The only thing now is that the 'masternodelist' command gets different results on my controller and sanctuary wallet.

Controller wallet:

Code Select Expand


17:21:14

{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "NEW_START_REQUIRED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "ENABLED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "WATCHDOG_EXPIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "NEW_START_REQUIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "ENABLED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "ENABLED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "ENABLED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "ENABLED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "ENABLED"
}


Sanctuary wallet:

Code Select Expand


{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "NEW_START_REQUIRED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "ENABLED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "NEW_START_REQUIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "NEW_START_REQUIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "NEW_START_REQUIRED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "NEW_START_REQUIRED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "NEW_START_REQUIRED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "ENABLED"
}


I am 'a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4' btw.

Yeah, I have the same problem.  It looks like everyone who reindexed the chain today lost all their masternode data (stored in mncache.dat), all the governance objects are getting revoted on now etc.  So basically the masternode state is very bad in testnet.  We will have to wait a while to see if our 5 masternodes switch back to enabled. 

[klondike]

all 5 MNodes is ENABLED but we are STUCKED? 41124

[Rob Andrews]

all 5 MNodes is ENABLED but we are STUCKED? 41124

All my 3 nodes rolling at 41534 and synced; status = ENABLED, try on your bad node, try going to peers and ban anyone with a low height, and see if it syncs, if not -reindex.

[Rob Andrews]

I got some errors for you 

https://pastebin.com/NaHRQBi7

Also I'm not sure if you saw it but this is what I got on mainnet just before switching that box to testnet

Oh good, thanks .

Oh I see its in mainnet, so that should be easy to research, hopefully.

Will check it out right now.

[Rob Andrews]

I got some errors for you 

https://pastebin.com/NaHRQBi7

Also I'm not sure if you saw it but this is what I got on mainnet just before switching that box to testnet

Thanks, this is extremely helpful as we went out of sync in testnet a couple times, like I said I saw this error a few times in prod in random logs, but it was so elusive I couldnt reproduce it.  Luckily, it looks like our advanced forensic logging yielding some info (knock on wood).  Lets hope we can get to the bottom of this in only a few passes.  Id hate to see this take a month to fix.

Anyway heres what I am seeing.  On the extremely bright side, I am generating the exact same biblehashes as your node was at the time of the problem.  Thats good, because I didnt want to see some type of non-deterministic biblehash.  So to follow me on that part of it, the biblehash requires 4 inputs, for example your first checkblock error requires me to run this command:


exec biblehash 333e1c9e6fc52027012f734b2b701a3325591759ec6f29686879faae98a4b585 1510797674 1510797171 16911
yielding biblehash: 0000000000072c18e0c42605b5431bd2e2343ced7942038c6a39f76ffa077225

Which matches yours, so that is great.

Next I checked the bits in that block that yields the difficulty and therefore the targethash which matches your targethash in the checkblock message. 

Here is where it gets interesting.  If you convert the hex to decimal (if you need to unless you can do this in your head) the biblehash with a 7a7 prefix must be Lower than the uint256 target (947e).  The 7a7 is 1959 decimal and the 947e is 38014, meaning that it passes. 

So basically what I am saying here is your arithmetic library is failing to calculate the less than operator here (LOL).  I know it sounds strange, now Bitcoin is using something called UintToArith256  to convert the uint256 big number to an arithmetic vector, allowing the math problem to be performed.  So of course that leads me to wonder how can the node function at all if that less than is failing? LOL.  So I move on and I tested the next two examples.  Same issue:  677a (26490) is less than (43096) meaning that the test should have passed, and the block was good and the error was a bad call.
Next 72c1 < 7e28: same issue, block was good, bad call.

So what Im going to do now is Ill add some more code to store the biblehash before its compared, and then compare it, then store it again and log both the original and the compared values in the log, then we can see if somehow, your biblehash changes between the hash and the comparison.  Because obviosly you are doing math for all other blocks. 

Alright Ill be back in about a half hour with the new version.

Good Job.

[Rob Andrews]

Welp that is indeed really strange since it's not failing on every block! There must be something special about these numbers

It will be interesting to see what the issue is/was.

Yeah, it will have to reveal itself as either a vector problem in the biblehash accessor, or a problem in that arithmetic lib, or it could even be a BOOST lib issue, maybe one particular boost lib in your version being manipulated a certain way before the call, Oh who knows

Anyway, I just pushed out 1.0.6.0 if you want to run that for a while until we get more examples. 

[Rob Andrews]

Doing that now, would like me to run on testnet or mainnet?

Mainnet is probably better.

[Rob Andrews]

Not sure if you're still around Rob but I just got that

2017-11-17 02:43:04 ProcessNewBlock : ACCEPTED
2017-11-17 02:56:42 CheckProofOfWork(1.0): BlockHash d07f0bd5d5fbebd8317309465b14c5ab94684f41d68944c0aab938d306ab60fb, ProdChain TRUE, SSLVersion OpenSSL 1.0.1t  3 May 2016, BlockTime 1510887401.000000, PrevBlockTime 1510886577.000000, BibleHash1 0004010c0376b246463104ca838735effd6c8d9d8294084276cf15bb4fd602c7, BibleHash2 0004010c0376b246463104ca838735effd6c8d9d8294084276cf15bb4fd602c7, TargetHash 0000000000074772000000000000000000000000000000000000000000000000, Forensics exec biblehash d07f0bd5d5fbebd8317309465b14c5ab94684f41d68944c0aab938d306ab60fb 1510887401 1510886577 17065
2017-11-17 02:56:42 ERROR: CheckProofOfWork(1): BibleHash does not meet POW level, prevheight 17065.000000 pindexPrev 7ec3beeb8f2710d57961c2a75687578b19687f7d219d6a062037c764bcb1f712
2017-11-17 02:56:42 ERROR: CheckBlockHeader(): proof of work failed
2017-11-17 02:56:42 Misbehaving: 35.199.177.240:40000 (0 -> 5)
2017-11-17 02:56:42 ERROR: invalid header received d07f0bd5d5fbebd8317309465b14c5ab94684f41d68944c0aab938d306ab60fb
2017-11-17 02:56:42 UpdateTip: new best=d07f0bd5d5fbebd8317309465b14c5ab94684f41d68944c0aab938d306ab60fb  height=17066  log2_work=57.808768  tx=28757  date=2017-11-17 02:56:41 progress=1.000000  cache=0.0MiB(7tx)
--
2017-11-17 02:58:23 ProcessNewBlock : ACCEPTED
2017-11-17 03:03:54 CheckProofOfWork(1.0): BlockHash de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261, ProdChain TRUE, SSLVersion OpenSSL 1.0.1t  3 May 2016, BlockTime 1510887834.000000, PrevBlockTime 1510887502.000000, BibleHash1 0016bc69d09c727fedfac1fda865bf9666d94e4a20d02527c18efebeaaea19fb, BibleHash2 0016bc69d09c727fedfac1fda865bf9666d94e4a20d02527c18efebeaaea19fb, TargetHash 000000000008528c000000000000000000000000000000000000000000000000, Forensics exec biblehash de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261 1510887834 1510887502 17067
2017-11-17 03:03:54 ERROR: CheckProofOfWork(1): BibleHash does not meet POW level, prevheight 17067.000000 pindexPrev 8c7665df98db007329275eb36a43f6f4d735324af64728e787811dc7d0079578
2017-11-17 03:03:54 ERROR: CheckBlockHeader(): proof of work failed
2017-11-17 03:03:54 Misbehaving: 51.15.89.39:40000 (0 -> 5)
2017-11-17 03:03:54 ERROR: invalid header received de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261
2017-11-17 03:03:54 UpdateTip: new best=de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261  height=17068  log2_work=57.809114  tx=28759  date=2017-11-17 03:03:54 progress=1.000000  cache=0.0MiB(9tx)

Whats very strange is in these two examples, the exec biblehash command does not produce the same output as the error.  This is very strange, especially since each of your error messages we call it twice.

Anyway, I have to close out soon, in the mean time, what does your node report for this command in prod:

exec biblehash de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261 1510887834 1510887502 17067

Mine reports : 000000000005fc7360c14e1b6c3d3a7530d27e942b928bdb17c8801fdafc19ba
Which is a different hash than the error message reports.

[tiras]

all nodes on testnet are down except for 1 :
./biblepay-cli masternode list
{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "ENABLED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "WATCHDOG_EXPIRED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "WATCHDOG_EXPIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "WATCHDOG_EXPIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "WATCHDOG_EXPIRED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "WATCHDOG_EXPIRED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "WATCHDOG_EXPIRED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "WATCHDOG_EXPIRED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "WATCHDOG_EXPIRED"


I upgraded mine to 1.0.6    and after re-syncing it's showing "watchdog expired"   .  Watchdog is running with no errors. I didn't touch it.

  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "WATCHDOG_EXPIRED",

[tiras]

all nodes on testnet are down except for 1 :
./biblepay-cli masternode list
{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "ENABLED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "WATCHDOG_EXPIRED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "WATCHDOG_EXPIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "WATCHDOG_EXPIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "WATCHDOG_EXPIRED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "WATCHDOG_EXPIRED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "WATCHDOG_EXPIRED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "WATCHDOG_EXPIRED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "WATCHDOG_EXPIRED"


I upgraded mine to 1.0.6    and after re-syncing it's showing "watchdog expired"   .  Watchdog is running with no errors. I didn't touch it.

  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "WATCHDOG_EXPIRED",

looks ok now :

Code Select Expand

./biblepay-cli masternode list
{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "ENABLED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "ENABLED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "WATCHDOG_EXPIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "WATCHDOG_EXPIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "ENABLED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "ENABLED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "ENABLED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "EXPIRED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "EXPIRED"

[Rob Andrews]

all nodes on testnet are down except for 1 :
./biblepay-cli masternode list
{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "ENABLED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "WATCHDOG_EXPIRED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "WATCHDOG_EXPIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "WATCHDOG_EXPIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "WATCHDOG_EXPIRED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "WATCHDOG_EXPIRED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "WATCHDOG_EXPIRED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "WATCHDOG_EXPIRED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "WATCHDOG_EXPIRED"


I upgraded mine to 1.0.6    and after re-syncing it's showing "watchdog expired"   .  Watchdog is running with no errors. I didn't touch it.

  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "WATCHDOG_EXPIRED",

Well actually that masternode list in this case is just the view from your local node which is down, checking the masternode list, we have the same 5 ENABLED we had 20 hours ago (after the fork and after everyone restarted), so things are stll improving from that Point, but my guess is you havent actually recovered from that particular fork.

First you will have to sync and then work on the watchdog later.  mnsync status must show 999 before you can work on testing watchman.