Messages

3796

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 17, 2017, 06:34:29 AM »

all nodes on testnet are down except for 1 :
./biblepay-cli masternode list
{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "ENABLED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "WATCHDOG_EXPIRED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "WATCHDOG_EXPIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "WATCHDOG_EXPIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "WATCHDOG_EXPIRED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "WATCHDOG_EXPIRED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "WATCHDOG_EXPIRED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "WATCHDOG_EXPIRED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "WATCHDOG_EXPIRED"


I upgraded mine to 1.0.6    and after re-syncing it's showing "watchdog expired"   .  Watchdog is running with no errors. I didn't touch it.

  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "WATCHDOG_EXPIRED",

Well actually that masternode list in this case is just the view from your local node which is down, checking the masternode list, we have the same 5 ENABLED we had 20 hours ago (after the fork and after everyone restarted), so things are stll improving from that Point, but my guess is you havent actually recovered from that particular fork.

First you will have to sync and then work on the watchdog later.  mnsync status must show 999 before you can work on testing watchman.

3797

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 16, 2017, 09:53:41 PM »

Not sure if you're still around Rob but I just got that

2017-11-17 02:43:04 ProcessNewBlock : ACCEPTED
2017-11-17 02:56:42 CheckProofOfWork(1.0): BlockHash d07f0bd5d5fbebd8317309465b14c5ab94684f41d68944c0aab938d306ab60fb, ProdChain TRUE, SSLVersion OpenSSL 1.0.1t  3 May 2016, BlockTime 1510887401.000000, PrevBlockTime 1510886577.000000, BibleHash1 0004010c0376b246463104ca838735effd6c8d9d8294084276cf15bb4fd602c7, BibleHash2 0004010c0376b246463104ca838735effd6c8d9d8294084276cf15bb4fd602c7, TargetHash 0000000000074772000000000000000000000000000000000000000000000000, Forensics exec biblehash d07f0bd5d5fbebd8317309465b14c5ab94684f41d68944c0aab938d306ab60fb 1510887401 1510886577 17065
2017-11-17 02:56:42 ERROR: CheckProofOfWork(1): BibleHash does not meet POW level, prevheight 17065.000000 pindexPrev 7ec3beeb8f2710d57961c2a75687578b19687f7d219d6a062037c764bcb1f712
2017-11-17 02:56:42 ERROR: CheckBlockHeader(): proof of work failed
2017-11-17 02:56:42 Misbehaving: 35.199.177.240:40000 (0 -> 5)
2017-11-17 02:56:42 ERROR: invalid header received d07f0bd5d5fbebd8317309465b14c5ab94684f41d68944c0aab938d306ab60fb
2017-11-17 02:56:42 UpdateTip: new best=d07f0bd5d5fbebd8317309465b14c5ab94684f41d68944c0aab938d306ab60fb  height=17066  log2_work=57.808768  tx=28757  date=2017-11-17 02:56:41 progress=1.000000  cache=0.0MiB(7tx)
--
2017-11-17 02:58:23 ProcessNewBlock : ACCEPTED
2017-11-17 03:03:54 CheckProofOfWork(1.0): BlockHash de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261, ProdChain TRUE, SSLVersion OpenSSL 1.0.1t  3 May 2016, BlockTime 1510887834.000000, PrevBlockTime 1510887502.000000, BibleHash1 0016bc69d09c727fedfac1fda865bf9666d94e4a20d02527c18efebeaaea19fb, BibleHash2 0016bc69d09c727fedfac1fda865bf9666d94e4a20d02527c18efebeaaea19fb, TargetHash 000000000008528c000000000000000000000000000000000000000000000000, Forensics exec biblehash de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261 1510887834 1510887502 17067
2017-11-17 03:03:54 ERROR: CheckProofOfWork(1): BibleHash does not meet POW level, prevheight 17067.000000 pindexPrev 8c7665df98db007329275eb36a43f6f4d735324af64728e787811dc7d0079578
2017-11-17 03:03:54 ERROR: CheckBlockHeader(): proof of work failed
2017-11-17 03:03:54 Misbehaving: 51.15.89.39:40000 (0 -> 5)
2017-11-17 03:03:54 ERROR: invalid header received de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261
2017-11-17 03:03:54 UpdateTip: new best=de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261  height=17068  log2_work=57.809114  tx=28759  date=2017-11-17 03:03:54 progress=1.000000  cache=0.0MiB(9tx)

Whats very strange is in these two examples, the exec biblehash command does not produce the same output as the error.  This is very strange, especially since each of your error messages we call it twice.

Anyway, I have to close out soon, in the mean time, what does your node report for this command in prod:

exec biblehash de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261 1510887834 1510887502 17067

Mine reports : 000000000005fc7360c14e1b6c3d3a7530d27e942b928bdb17c8801fdafc19ba
Which is a different hash than the error message reports.

3798

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 16, 2017, 07:58:27 PM »

Doing that now, would like me to run on testnet or mainnet?

Mainnet is probably better.

3799

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 16, 2017, 07:46:30 PM »

Welp that is indeed really strange since it's not failing on every block! There must be something special about these numbers

It will be interesting to see what the issue is/was.

Yeah, it will have to reveal itself as either a vector problem in the biblehash accessor, or a problem in that arithmetic lib, or it could even be a BOOST lib issue, maybe one particular boost lib in your version being manipulated a certain way before the call, Oh who knows

Anyway, I just pushed out 1.0.6.0 if you want to run that for a while until we get more examples. 

3800

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 16, 2017, 07:28:35 PM »

I got some errors for you 

https://pastebin.com/NaHRQBi7

Also I'm not sure if you saw it but this is what I got on mainnet just before switching that box to testnet

Thanks, this is extremely helpful as we went out of sync in testnet a couple times, like I said I saw this error a few times in prod in random logs, but it was so elusive I couldnt reproduce it.  Luckily, it looks like our advanced forensic logging yielding some info (knock on wood).  Lets hope we can get to the bottom of this in only a few passes.  Id hate to see this take a month to fix.

Anyway heres what I am seeing.  On the extremely bright side, I am generating the exact same biblehashes as your node was at the time of the problem.  Thats good, because I didnt want to see some type of non-deterministic biblehash.  So to follow me on that part of it, the biblehash requires 4 inputs, for example your first checkblock error requires me to run this command:


exec biblehash 333e1c9e6fc52027012f734b2b701a3325591759ec6f29686879faae98a4b585 1510797674 1510797171 16911
yielding biblehash: 0000000000072c18e0c42605b5431bd2e2343ced7942038c6a39f76ffa077225

Which matches yours, so that is great.

Next I checked the bits in that block that yields the difficulty and therefore the targethash which matches your targethash in the checkblock message. 

Here is where it gets interesting.  If you convert the hex to decimal (if you need to unless you can do this in your head) the biblehash with a 7a7 prefix must be Lower than the uint256 target (947e).  The 7a7 is 1959 decimal and the 947e is 38014, meaning that it passes. 

So basically what I am saying here is your arithmetic library is failing to calculate the less than operator here (LOL).  I know it sounds strange, now Bitcoin is using something called UintToArith256  to convert the uint256 big number to an arithmetic vector, allowing the math problem to be performed.  So of course that leads me to wonder how can the node function at all if that less than is failing? LOL.  So I move on and I tested the next two examples.  Same issue:  677a (26490) is less than (43096) meaning that the test should have passed, and the block was good and the error was a bad call.
Next 72c1 < 7e28: same issue, block was good, bad call.

So what Im going to do now is Ill add some more code to store the biblehash before its compared, and then compare it, then store it again and log both the original and the compared values in the log, then we can see if somehow, your biblehash changes between the hash and the comparison.  Because obviosly you are doing math for all other blocks. 

Alright Ill be back in about a half hour with the new version.

Good Job.

3801

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 16, 2017, 06:55:04 PM »

I got some errors for you 

https://pastebin.com/NaHRQBi7

Also I'm not sure if you saw it but this is what I got on mainnet just before switching that box to testnet

Oh good, thanks .

Oh I see its in mainnet, so that should be easy to research, hopefully.

Will check it out right now.

3802

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 16, 2017, 03:42:57 PM »

all 5 MNodes is ENABLED but we are STUCKED? 41124

All my 3 nodes rolling at 41534 and synced; status = ENABLED, try on your bad node, try going to peers and ban anyone with a low height, and see if it syncs, if not -reindex.

3803

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 16, 2017, 01:21:31 PM »

I've been trying to get my masternode up and running again (making sure watchdog is functioning).

My sanctuary wallet was indeed stuck on a certain block and wouldn't sync. But I've reindexed both and it seems I'm on the right chain right now (both wallets are on the same block).

Controller wallet:

Code: [Select]


{
  "version": 1000506,
  "protocolversion": 70708,
  "walletversion": 61000,
  "wallet_fullversion": "1.0.5.6",
  "balance": 38514484.00745234,
  "privatesend_balance": 0.00000000,
  "blocks": 41330,
  "timeoffset": -2,
  "connections": 3,
  "proxy": "",
  "difficulty": 0.1980807012241197,
  "testnet": true,
  "keypoololdest": 1507285625,
  "keypoolsize": 1001,
  "paytxfee": 0.00000000,
  "relayfee": 0.00010000,
  "errors": ""
}

Sanctuary wallet:

Code: [Select]

{
  "version": 1000508,
  "protocolversion": 70708,
  "walletversion": 61000,
  "wallet_fullversion": "1.0.5.8",
  "balance": 0.00000000,
  "privatesend_balance": 0.00000000,
  "blocks": 41330,
  "timeoffset": 0,
  "connections": 8,
  "proxy": "",
  "difficulty": 0.1980807012241197,
  "testnet": true,
  "keypoololdest": 1508425662,
  "keypoolsize": 1001,
  "paytxfee": 0.00000000,
  "relayfee": 0.00010000,
  "errors": ""
}

The only thing now is that the 'masternodelist' command gets different results on my controller and sanctuary wallet.

Controller wallet:

Code: [Select]

17:21:14

{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "NEW_START_REQUIRED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "ENABLED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "WATCHDOG_EXPIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "NEW_START_REQUIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "ENABLED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "ENABLED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "ENABLED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "ENABLED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "ENABLED"
}

Sanctuary wallet:

Code: [Select]

{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "NEW_START_REQUIRED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "ENABLED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "NEW_START_REQUIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "NEW_START_REQUIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "NEW_START_REQUIRED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "NEW_START_REQUIRED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "NEW_START_REQUIRED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "ENABLED"
}

I am 'a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4' btw.

Yeah, I have the same problem.  It looks like everyone who reindexed the chain today lost all their masternode data (stored in mncache.dat), all the governance objects are getting revoted on now etc.  So basically the masternode state is very bad in testnet.  We will have to wait a while to see if our 5 masternodes switch back to enabled. 

3804

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 16, 2017, 01:19:57 PM »

haha no worries. It's 3 am so I won't be up for long too.

Also,  I only brought the two nodes I talked about in my 2 previous posts above. 1 mining and 1 non-mining that's it.

This is what happened around the time of the "fork" so I'm wondering if it could have been me.

Cool, I took a look at that log, and I can see how that particular node would go off on its own and ban everyone.  In this case, it did not have the same mncache.dat view that everyone else had, so its mnpayment schedule was different, and since we enforce masternode recipients, it was not going to agree on the next block (because it couldnt fill out a masternode while the supermajority could). The question is how did it get all synced with a different view of mncache?  Probably because of the headers it rejected it was mining on a fork or something earlier.

The other thing is, if we have a supermajority that is synced with synced masternode data, a couple bad fraudulently compiled nodes with hosed up integrated business logic compiled in should not be able to take us down.  Of course with the fragility of testnet, having only 4 synced masternodes, its possible to hose us up if you bring just 2 bad nodes online. 

Either way lets move past this one and wait til you have the checkblock(1) error showing and then maybe that will shed light on why your headers were rejected. 

3805

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 16, 2017, 10:08:23 AM »

The mining node now on block 41309 is 94.130.49.121 or 2a01:4f8:10b:26e3::2

Ah, good news, we had Exactly 50-50 on both chains!  I just resynced one node, and the balance is shifting.  Im observing this very closely to see if the reset of the net syncs up to 41309+...  I just see one node is reorging by itself.
Unfortunately Ive got company stopping in now, I may be gone for a litle while. 

3806

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 16, 2017, 09:59:51 AM »

If you see my post above, I hope it's not my node who broke the network!

Whats your IP, Id like to see if you are banned on any of my 3 nodes?

3807

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 16, 2017, 09:41:20 AM »

it looks,that we are stuck on 41100

Yeah I see half the network is on 41116 and the other half on 41293....

Looking...  Not sure why this started all of a sudden, we were going so well other than having a rag tag set of sanctuaries we were staying in sync for a long time.  This happened at 5AM this morning (4 hours ago).

3808

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 16, 2017, 09:40:05 AM »

Alright, should I mine at the same time or just be connected and synced to the network?

You can mine if you want, no problem, as long as you are synced first.

But now all of a sudden we have a fork.  I dont know if it has anything to do with your nodes coming on? How many did you bring into testnet?

We all need to be synced before I can make any sense out of the test harness forensic commands.

3809

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 16, 2017, 09:14:43 AM »

Rob,

i did it by your instrunctions,typed and nothing... and btw: i had in crontab correct    when you checked my screens what i posted here .. you must saw it
it looks,that all your setup is broken.Exists anybody whom working masternode except you? I dont know this person.By me nobody.It will be very hard start masternode when doesnt working still.





what happened when ill close controller-win-wallet and again run it? its OK?


thanks

That command you ran is good.  When it results in an empty result, Watchman is working.  I see your IP is working on the list also, so its obviously working.  The trick is to get the exact command that runs in the crontab working from the command line, then ensure its in the crontab, and then every 5 mins when it runs, your watchdog will not squeal out to the network that your node is down.

I dont think our guide is bad.  I think it needs a couple rough edges polished.  Ive already edited it based on Togo and Your issues over the last 2 days.

What we need is someone to come in here and start a new one and give us fresh advice.  Anyone who wants to start a testnet Sanctuary,
run the Wiki.biblepay.org/Create_Masternode Guide, and let us know if you think it is lacking.  We have to also take into account that it was written in English, and we have no Polish translation, so Im sure communication is part of the problem.  Also, keep in mind not everyone will use biblepayd to run the commands, for example, I set up my Cold node using the windows RPC, and used Samba and nano for all my settings changes.

3810

TestNet Discussion Archive / Re: BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

« on: November 15, 2017, 10:16:06 PM »

Hey Rob, I upgraded my test node and I'm fully synced on testnet. Also just to clarify, the issue also happens on my US machines with the CST timezone and not only the European ones. I'm ready to do any test you would like me to do when you're around.

Im going to be hitting the sack in a few mins, but through the night you can fully sync in testnet, run the code, then see if you receive a CheckProofOfWork(1) log then post it.