BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

[Rob Andrews]

Not sure if you're still around Rob but I just got that

2017-11-17 02:43:04 ProcessNewBlock : ACCEPTED
2017-11-17 02:56:42 CheckProofOfWork(1.0): BlockHash d07f0bd5d5fbebd8317309465b14c5ab94684f41d68944c0aab938d306ab60fb, ProdChain TRUE, SSLVersion OpenSSL 1.0.1t  3 May 2016, BlockTime 1510887401.000000, PrevBlockTime 1510886577.000000, BibleHash1 0004010c0376b246463104ca838735effd6c8d9d8294084276cf15bb4fd602c7, BibleHash2 0004010c0376b246463104ca838735effd6c8d9d8294084276cf15bb4fd602c7, TargetHash 0000000000074772000000000000000000000000000000000000000000000000, Forensics exec biblehash d07f0bd5d5fbebd8317309465b14c5ab94684f41d68944c0aab938d306ab60fb 1510887401 1510886577 17065
2017-11-17 02:56:42 ERROR: CheckProofOfWork(1): BibleHash does not meet POW level, prevheight 17065.000000 pindexPrev 7ec3beeb8f2710d57961c2a75687578b19687f7d219d6a062037c764bcb1f712
2017-11-17 02:56:42 ERROR: CheckBlockHeader(): proof of work failed
2017-11-17 02:56:42 Misbehaving: 35.199.177.240:40000 (0 -> 5)
2017-11-17 02:56:42 ERROR: invalid header received d07f0bd5d5fbebd8317309465b14c5ab94684f41d68944c0aab938d306ab60fb
2017-11-17 02:56:42 UpdateTip: new best=d07f0bd5d5fbebd8317309465b14c5ab94684f41d68944c0aab938d306ab60fb  height=17066  log2_work=57.808768  tx=28757  date=2017-11-17 02:56:41 progress=1.000000  cache=0.0MiB(7tx)
--
2017-11-17 02:58:23 ProcessNewBlock : ACCEPTED
2017-11-17 03:03:54 CheckProofOfWork(1.0): BlockHash de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261, ProdChain TRUE, SSLVersion OpenSSL 1.0.1t  3 May 2016, BlockTime 1510887834.000000, PrevBlockTime 1510887502.000000, BibleHash1 0016bc69d09c727fedfac1fda865bf9666d94e4a20d02527c18efebeaaea19fb, BibleHash2 0016bc69d09c727fedfac1fda865bf9666d94e4a20d02527c18efebeaaea19fb, TargetHash 000000000008528c000000000000000000000000000000000000000000000000, Forensics exec biblehash de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261 1510887834 1510887502 17067
2017-11-17 03:03:54 ERROR: CheckProofOfWork(1): BibleHash does not meet POW level, prevheight 17067.000000 pindexPrev 8c7665df98db007329275eb36a43f6f4d735324af64728e787811dc7d0079578
2017-11-17 03:03:54 ERROR: CheckBlockHeader(): proof of work failed
2017-11-17 03:03:54 Misbehaving: 51.15.89.39:40000 (0 -> 5)
2017-11-17 03:03:54 ERROR: invalid header received de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261
2017-11-17 03:03:54 UpdateTip: new best=de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261  height=17068  log2_work=57.809114  tx=28759  date=2017-11-17 03:03:54 progress=1.000000  cache=0.0MiB(9tx)

Whats very strange is in these two examples, the exec biblehash command does not produce the same output as the error.  This is very strange, especially since each of your error messages we call it twice.

Anyway, I have to close out soon, in the mean time, what does your node report for this command in prod:

exec biblehash de3fdeb9096b6eca09358af07e7f126a6064a00024ca86f361e0b23233d13261 1510887834 1510887502 17067

Mine reports : 000000000005fc7360c14e1b6c3d3a7530d27e942b928bdb17c8801fdafc19ba
Which is a different hash than the error message reports.

[tiras]

all nodes on testnet are down except for 1 :
./biblepay-cli masternode list
{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "ENABLED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "WATCHDOG_EXPIRED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "WATCHDOG_EXPIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "WATCHDOG_EXPIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "WATCHDOG_EXPIRED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "WATCHDOG_EXPIRED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "WATCHDOG_EXPIRED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "WATCHDOG_EXPIRED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "WATCHDOG_EXPIRED"


I upgraded mine to 1.0.6    and after re-syncing it's showing "watchdog expired"   .  Watchdog is running with no errors. I didn't touch it.

  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "WATCHDOG_EXPIRED",

[tiras]

all nodes on testnet are down except for 1 :
./biblepay-cli masternode list
{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "ENABLED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "WATCHDOG_EXPIRED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "WATCHDOG_EXPIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "WATCHDOG_EXPIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "WATCHDOG_EXPIRED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "WATCHDOG_EXPIRED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "WATCHDOG_EXPIRED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "WATCHDOG_EXPIRED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "WATCHDOG_EXPIRED"


I upgraded mine to 1.0.6    and after re-syncing it's showing "watchdog expired"   .  Watchdog is running with no errors. I didn't touch it.

  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "WATCHDOG_EXPIRED",

looks ok now :

Code Select Expand

./biblepay-cli masternode list
{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "ENABLED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "ENABLED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "WATCHDOG_EXPIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "WATCHDOG_EXPIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "ENABLED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "ENABLED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "ENABLED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "EXPIRED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "EXPIRED"

[Rob Andrews]

all nodes on testnet are down except for 1 :
./biblepay-cli masternode list
{
  "04ca4bd54153476e6d19b72dc26bc27b74cdbbd687aea07c6cc1cd2182beab60-1": "ENABLED",
  "a23321871670924fa669c26284aeb9b36987d6bb3d7a590eaf2b5406b85201f4-1": "WATCHDOG_EXPIRED",
  "a27ae7dcc93df0ff266d2f11cb906f4929b7ad7d6b46bc98c192ac7998aa1058-1": "WATCHDOG_EXPIRED",
  "8ff3d83216612e2c79a36c5f66f191b34a07eb1d01600b104a76b3151de9e7f6-1": "WATCHDOG_EXPIRED",
  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "WATCHDOG_EXPIRED",
  "7b3873de1969b087f63cd287c64694b14250aecf7779c67f68691533a6616f3c-1": "NEW_START_REQUIRED",
  "d183b913e7a4b95c0202bf15d296e00bbdf0bedce317e340985a2c78af06cba4-1": "WATCHDOG_EXPIRED",
  "404f50700e2ffc8c4adb76bea92fb572584b1f3e24ec736d6883a64cc88417fb-1": "WATCHDOG_EXPIRED",
  "be3eaf8322909bbb150d89058bf261f0a2bf96369c8928313a3d69e2d5087add-1": "WATCHDOG_EXPIRED",
  "e413e134f7ecb6388ddfe401870f1e0ba602479abb8de8a5bf78c59c4087ee28-1": "NEW_START_REQUIRED",
  "b0cfdfa2194556e211099bc620a7e27487958aa5774db0c78ded0f839e3105d2-1": "NEW_START_REQUIRED",
  "c5a8405cbc39dd97004a64c1db586313ed25f3c4553ca59a0bdad30dd0c551d5-1": "NEW_START_REQUIRED",
  "44d550fef16c8bc5e340599cddbd6d98736218e1900f92b4625768e5830abfd0-1": "NEW_START_REQUIRED",
  "847be5b647856e9b030d785d7bc82f146362371a28663348e7a3d28df3e38c55-1": "WATCHDOG_EXPIRED"


I upgraded mine to 1.0.6    and after re-syncing it's showing "watchdog expired"   .  Watchdog is running with no errors. I didn't touch it.

  "78a235e8d3427b7fec1a0f3635cf3aeb2dbf7d0d53c44578815060d80280a271-1": "WATCHDOG_EXPIRED",

Well actually that masternode list in this case is just the view from your local node which is down, checking the masternode list, we have the same 5 ENABLED we had 20 hours ago (after the fork and after everyone restarted), so things are stll improving from that Point, but my guess is you havent actually recovered from that particular fork.

First you will have to sync and then work on the watchdog later.  mnsync status must show 999 before you can work on testing watchman.

[Rob Andrews]

ok,in controller wallet i BAN bad nodes and synced= all rewards received .... but linux masternode is stuck ... its need to resync masternode too?

what is command for reindex wallet in testnet i tried

./biblepay-cli -testnet -daemon -reindex
./biblepay-cli -testnet -reindex

and w/result

This isnt really a support thread on how to use the wallet.  Just resync your node and then lets continue testing sanctuaries.

EDIT: It looks like 95% of the network is at block 42122.  I see 65.x is resyncing.  Just try to get up to 42122+.

[Rob Andrews]

Just started a VM to try it and I also managed to get different biblehash!

Note: The mn4 machine above is running OpenSSL 101t while this one is running OpenSSL 102g

}
root@snapshot-7903-unknown-4gb-fsn1-1 ~ # biblepay-cli exec biblehash e0118ac9c4b5f6ec508e4bfefc38599fd55e7f79d108a560aeff9ceb0debc347 1510892191 1510890756 17074revheight 17111.000000 pindexPrev 20e9f4416b8e469b23a6fde82f197342086622e9ac827e78875701d001ca3916
2017-11-17 10:05:14 ERROR: CheckBlockHeader(): proof of work failed
2017-11-17 10:05:14 ERROR: ProcessNewBlock: CheckBlock FAILED
2017-11-17 10:05:14 Misbehaving: 97.99.69.33:40000 (0 -> 5)
2017-11-17 10:05:14 UpdateTip: new best=541a323bbf91103d90556af0e5b9143673a2cd36680fffe21bc3b7a9cbc85356  height=17112  log2_work=57.818294  tx=28819  date=2017-11-17 10:05:12 progress=1.000000  cache=0.0MiB(112tx)

Nice tests, thanks, you really gave a lot of data to work with and pretty much every condition LOL.

So Im going to make this my top priority today. 

On your question about re-running the biblehash command with a prior blocknumber parameter (the ones you noted 17069, same as 17079) etc, that is fine because the block number parameter being sent into the biblehash is only used for it to determine the chain height, to turn on or off f7000 or f8000, and that is enforced on all nodes so that is secure, so changing that between f7000 start and f8000 start will just give you the same exact biblehash, so that is OK.  Changing the lastblocktime is a similar issue: that only drives when the block becomes "late".  When the block time exceeds 30 minutes then and only then the hash changes, so that is OK too.

On the question regarding the if statement, which hash its using, in this last version 1060, it is actually checking both, first it says if biblehash1 is too high, then if biblehash2 is too high, so your log only yields errors where they both were high hashes.  But thats OK too, because you still exposed enough information.

However I wasnt sure if you had to run the cli command 10,000 times programatically to change the hash in your example because I ran one of your examples 125 times from my RPC and received the same hash every time so so far I have been unable to change the hash like you have, but dont worry, I have enough info from your log for something to go on.

But focusing in on the one where you show that the hash1 and hash2 are actually completely different now that is the big dog right there, that is the smoking gun, I have never seen that.  Imo, that rules out the math operator and math class, that was generated on 101t, so it "points" toward not being openSSL versioning, it really points to either the MD5 function 'caching' the prior result (from the last blockindex check), or the AES512 vector 'caching' the last result.  So I think today, Im going to hone in on making an alternate (compatible of course) biblehash, using a different md5 supplier and refactoring the AES512 call, and we can run that side by side, and see if something changes.

[Rob Andrews]

  im deleted chain+blocks ans im back  ...

question: how long being when status WATCHDOG changed to ENABLED from WATCH_DOG EXPIRED?

thanks

There are multiple layers of issues when you resync. When you have been synced for 24+ hours, if you had a watchdog problem, watchman would update the database and the gui and the votes in 15 minutes.  When you re-sync, your mncache.dat file is hosed and it takes about 6 hours to fix itself.

Lets just ensure your watchman command returns empty, and then wait for something to change once you are synced.  If not fixed in 8 hours something is wrong.

[Rob Andrews]

Oh also and I may have misunderstood you. Since everything is the same but the prev block number, shouldn't these hashes be the same?

root@mn2 ~ # biblepay-cli exec biblehash e0118ac9c4b5f6ec508e4bfefc38599fd55e7f79d108a560aeff9ceb0debc347 1510892191 1510890756 17079
{
  "Command": "biblehash",
  "BibleHash": "0000000000002aaa46b9dc122688a0e203a7ddec60b104c95c0f1372eb43e0dd"
}
root@mn2 ~ # biblepay-cli exec biblehash e0118ac9c4b5f6ec508e4bfefc38599fd55e7f79d108a560aeff9ceb0debc347 1510892191 1510890756 17078
{
  "Command": "biblehash",
  "BibleHash": "0000000000011c6f2cd7107900e4308d6db47227d9f1753e65b9d6fe206f3072"

root@mn2 ~ # biblepay-cli exec biblehash e0118ac9c4b5f6ec508e4bfefc38599fd55e7f79d108a560aeff9ceb0debc347 1510892191 1510890756 17069
{
  "Command": "biblehash",
  "BibleHash": "0000000000002aaa46b9dc122688a0e203a7ddec60b104c95c0f1372eb43e0dd"

Yes, they should be the same - all 3 of them should be the same since only the last parameter is changing, the correct hash is 2aaa in prod - Im running it now in the rpc.  The 11c6 hash is a bad hash.   That looks like the problem.  Its interesting you can generate those hashes so easily.  Would you say that the type of node you are testing on in prod is a very, very slow node with very low ram, or is it a super machine running many threads with a lot of ram?

[klondike]

Rob



1.how long being when status WATCHDOG changed to ENABLED from WATCH_DOG EXPIRED when watchman run?
2.where i can see that watcham is running?

3.PRE-ENABLED : watchman doesnt running,right?

[Rob Andrews]

All the "mn" nodes are Ryzen 7 with 64gb of DDR4 and 2xSSDs in Raid0. So decent machines I think!

Thats very interesting.  Im honestly thinking, the Ryzen processor may be so fast that one mining thread is loading the AES512 encryption key partially in the core while the other receives a hash based on a faulty encryption key.  I did find a circumstance where I believe that is possible, so all this effort is definitely worth it, as this will "potentially" give us a stable biblehash, if this is the problem.

Ok, I tested the new version to ensure it yields the same hashes and mines in prod and testnet so I am checking it in.
Please grab 1061.

In this version, we use a different call to openssl's md5 (IE directly from the biblehash lib), we change our call to AES512 to be more efficient, and we set up a few miningparams once before we mine to not require those calls for every hash (so thats a slight improvement there too).  The biggest change, the one that might make or break this as the fix, is this:  I found the bibleminer was re-loading the encryption key for every hash, and that could be the problem.  I fixed this! So lets go ahead and try it again and see if it fixes.

[Rob Andrews]

Rob

1.how long being when status WATCHDOG changed to ENABLED from WATCH_DOG EXPIRED when watchman run?
2.where i can see that watcham is running?

3.RE-ENABLED : watchman doesnt running,right?

1) If you delete your Watchman.db database and re-sync masternode data, and clear all your sanctuary .dat files (like mncache), then 15 minues, otherwise 8 hours.
2) You run watchman from the command line and see if it returns empty string.
3) Depends, if you just synced, you have to do queries in the watchman database to find the answer.

[Rob Andrews]

im still PRE-ENABLED  ... shit


by me,this masternodes is PONZI SCHEME like BBP : and i screw on all masternodes..... still not exists guide why watchman doesant running stable,im do all by guide and ROB=ADMIN still wrote with puzzles

ASKING YOU : how long being when you start WATCHMAN and your STATUS in controller wallet changes from preenabled to enabled? immediately?


i hope that masternodes wont be exists on BBP,cos this setuping is massacer

Dude its obvious you have a bad attitude and I dont want it here, why dont you try dash out for a while and see how easy it is to set one up and then come back later and tell us how it went?

[Rob Andrews]

Hey Rob, I finished updating one node and it is now mining on mainnet. How should I test that new version? Wait until I get an error in the log or should I spam the forensics command and see if I get different hashes again?

You can do both, go ahead and try to break it.

Yeah, we wont know its broken til it spurts out an error though....

[Rob Andrews]

Well well, I've been spamming the forensics command and I can happily say that it is now giving me the same hash no matter what!

However, with the same parameters but the block height, am I still getting the wrong hash for block 17078 and 17079 for example?

biblepay-cli exec biblehash e0118ac9c4b5f6ec508e4bfefc38599fd55e7f79d108a560aeff9ceb0debc347 1510892191 1510890756 17078
{
  "Command": "biblehash",
  "BibleHash": "0000000000011c6f2cd7107900e4308d6db47227d9f1753e65b9d6fe206f3072"

biblepay-cli exec biblehash e0118ac9c4b5f6ec508e4bfefc38599fd55e7f79d108a560aeff9ceb0debc347 1510892191 1510890756 17079
{
  "Command": "biblehash",
  "BibleHash": "0000000000002aaa46b9dc122688a0e203a7ddec60b104c95c0f1372eb43e0dd"

Did you say 0011 was invalid?

Im not able to reproduce that on 1061, I am running 1061 in testnet and moving from 78 to 79 still yields the 011 (not seeing the 2aaa).
What network are you in and is this definitely 1061?   Yeah, no changes should occur until you hit something like block 27000 etc.

Great on the improvement so far!  I bet we fixed it!  I bet this anomaly you found has a reason, such as cli being connected to 1060

[Rob Andrews]

Oh this is mainnet not testnet sorry

Let me restart in mainnet and try that....