BIBLEPAY - TESTNET THREAD - TESTING SANCTUARIES FOR GO LIVE AT CHRISTMAS

[tiras]

No, I'm just doing the same West did, so we could compare. Maybe someone else can debug it or something.

I don't have enough Linux knowledge to really contribute here, sorry. But if I can do anything to help, just tell me what to do

I've got the same error output :

Code Select Expand

~/.biblepaycore/watchman# venv/bin/python bin/watchman.py
Traceback (most recent call last):
  File "bin/watchman.py", line 244, in <module>
    main()
  File "bin/watchman.py", line 191, in main
    perform_biblepayd_object_sync(biblepayd)
  File "bin/watchman.py", line 24, in perform_biblepayd_object_sync
    GovernanceObject.sync(biblepayd)
  File "lib/models.py", line 87, in sync
    (go, subobj) = self.import_gobject_from_biblepayd(biblepayd, item)
  File "lib/models.py", line 142, in import_gobject_from_biblepayd
    subobj, created = subclass.get_or_create(object_hash=object_hash, defaults=subdikt)
  File "/root/.biblepaycore/watchman/venv/local/lib/python2.7/site-packages/peewee.py", line 4800, in get_or_create
    return query.get(), False
  File "/root/.biblepaycore/watchman/venv/local/lib/python2.7/site-packages/peewee.py", line 3090, in get
    return next(clone.execute())
  File "/root/.biblepaycore/watchman/venv/local/lib/python2.7/site-packages/peewee.py", line 2267, in next
    obj = self.iterate()
  File "/root/.biblepaycore/watchman/venv/local/lib/python2.7/site-packages/peewee.py", line 2253, in iterate
    return self.process_row(row)
  File "/root/.biblepaycore/watchman/venv/local/lib/python2.7/site-packages/peewee.py", line 2331, in process_row
    setattr(instance, column, func(row[i]))
  File "/root/.biblepaycore/watchman/venv/local/lib/python2.7/site-packages/peewee.py", line 1072, in python_value
    return decimal.Decimal(str(value))
  File "/usr/lib/python2.7/decimal.py", line 547, in __new__
    "Invalid literal for Decimal: %r" % value)
  File "/usr/lib/python2.7/decimal.py", line 3873, in _raise_error
    raise error(explanation)
decimal.InvalidOperation: Invalid literal for Decimal: '203,096'

[tiras]

I don't think I did anything differently, but I can say that the first and second one are mine.  So it's possible I borked it but I cannot imagine how.

West , can you  compare getinfo with mine :

Code Select Expand

./biblepay-cli getinfo
{
  "version": 1000701,
  "protocolversion": 70712,
  "walletversion": 61000,
  "wallet_fullversion": "1.0.7.1",
  "balance": 0.00000000,
  "privatesend_balance": 0.00000000,
  "retirement_balance": 0,
  "blocks": 22145,
  "timeoffset": 0,
  "connections": 208,
  "proxy": "",
  "difficulty": 319.8535804163807,
  "testnet": false,
  "keypoololdest": 1513631658,
  "keypoolsize": 1001,
  "paytxfee": 0.00000000,
  "relayfee": 0.00010000,
  "errors": ""

[togoshigekata]

I have BiblePay installed on another PC, and it is showing all sanctuaries as ENABLED or PRE_ENABLED,

My controller wallet shows most sanctuaries as WATCHDOG_EXPIRED.

Theres a note at the top of "My Sanctuaries" that says:

"Note: Status of your sanctuaries in local wallet can potentially be slightly incorrect.
Always wait for wallet to sync additional data and then double check from another node
if your masternode should be running but you still do not see "ENABLED" in "Status" field."

I wonder if we are just seeing incorrect data / experiencing a wallet glitch/bug in our Controller Wallets?

[jaapgvk]

At the moment, in my 'normal' wallets, the masternode statuses are a colourful rainbow of:

'enabled'
'pre enabled'
'watchdog expired'
'new start required'

I'm going to get some sleep now. Maybe it will sort itself out in the meantime...

[tiras]

The differences on mine are pretty expected

Blocks
Difficulty
Keypoololdest
Keypool Size

Code Select Expand

  "version": 1000701,
  "protocolversion": 70712,
  "walletversion": 61000,
  "wallet_fullversion": "1.0.7.1",
  "balance": 0.00000000,
  "privatesend_balance": 0.00000000,
  "retirement_balance": 0,
  "blocks": 22156,
  "timeoffset": 0,
  "connections": 116,
  "proxy": "",
  "difficulty": 623.3720306964408,
  "testnet": false,
  "keypoololdest": 1513488324,
  "keypoolsize": 1008,
  "paytxfee": 0.00000000,
  "relayfee": 0.00010000,
  "errors": ""


I meant the first few lines:

version": 1000701,
  "protocolversion": 70712,
  "walletversion": 61000,
  "wallet_fullversion": "1.0.7.1",

[tiras]

At the moment, in my 'normal' wallets, the masternode statuses are a colourful rainbow of:

'enabled'
'pre enabled'
'watchdog expired'
'new start required'

I'm going to get some sleep now. Maybe it will sort itself out in the meantime...

I wish we had some feedback of updates from the project leads  if it's normal and we should just disregard .
We were told the "WAtchdog_Expired"  takes the MN off the queue and voting . 
I had a couple of wins today though.

[togoshigekata]

Okay the other guys were right, there is an error in watchman when getting an object, I bet someone typed in a comma into their proposal amount maybe? hehehe

"decimal.InvalidOperation: Invalid literal for Decimal: '203,096'"

https://github.com/biblepay/watchman/blob/master/bin/watchman.py#L24
https://github.com/biblepay/watchman/blob/master/lib/models.py#L87
https://github.com/biblepay/watchman/blob/master/lib/models.py#L142

"DataString": "[[\"proposal\",{\"end_epoch\":\"1513805540\",\"name\":\"Help Blue Get A Sanctuary\",\"payment_address\":\"BCrxzSM2kLy8ERww4bch1fesRVFCLv2Tce\",\"payment_amount\":\"203,096\",\"start_epoch\":\"1513805540\",\"type\":1,\"url\":\"http://forum.biblepay.org/index.php?topic=52.0\"}]]",
    "Hash": "14ec3b00c46a8f1090be5bc46100e5df547f5e9ed6c3cba8a75ca72b4a9ef572",

LOL

[tiras]

Okay the other guys were right, there is an error in watchman when getting an object, I bet someone typed in a comma into their proposal amount maybe? hehehe

"decimal.InvalidOperation: Invalid literal for Decimal: '203,096'"

https://github.com/biblepay/watchman/blob/master/bin/watchman.py#L24
https://github.com/biblepay/watchman/blob/master/lib/models.py#L87
https://github.com/biblepay/watchman/blob/master/lib/models.py#L142

"DataString": "[[\"proposal\",{\"end_epoch\":\"1513805540\",\"name\":\"Help Blue Get A Sanctuary\",\"payment_address\":\"BCrxzSM2kLy8ERww4bch1fesRVFCLv2Tce\",\"payment_amount\":\"203,096\",\"start_epoch\":\"1513805540\",\"type\":1,\"url\":\"http://forum.biblepay.org/index.php?topic=52.0\"}]]",
    "Hash": "14ec3b00c46a8f1090be5bc46100e5df547f5e9ed6c3cba8a75ca72b4a9ef572",

LOL

means we are having another update coming soon ,   


Togo ,  you definitely deserve compensation for your support

[jaapgvk]

Okay the other guys were right, there is an error in watchman when getting an object, I bet someone typed in a comma into their proposal amount maybe? hehehe

So, this 'blue' guy, with his stupid proposal, single handedly crashed watchman. But it's not a bad thing, right? He 'found' a bug and it will be squashed.

[jaapgvk]

Aren't the proposals stored in the blockchain? I thought they needed six confirmations.

Oh, and the proposal wasn't from the real inblue  Just some guy that - i think - tried impersonating him to get BBP.

[Rob Andrews]

I wish we had some feedback of updates from the project leads  if it's normal and we should just disregard .
We were told the "WAtchdog_Expired"  takes the MN off the queue and voting . 
I had a couple of wins today though.

Well there are different levels of "down" in the core.  We are not deliberately lying .

So there is Watchdog Expired, POSE ban, Expired with bbp_daemon not responding, watchdog_expired with bbp daemon up and watchman-down, and the mn_payment eviction, the Spork control for Watchdog enforcement, etc.

In our implementation currently, we only evict you from the payment queue if you do this:
- Leave BBP daemon off for 30 hours
- Change your IP
- Get yourself POSE banned by breaking sanc rules
- Spending the Sanc collateral

We do not ban you for:
- Leaving BBP Up while Watchdog Expires due to a watchman issue


So in a nutshell if you do pull your sanc down for more than 30 hours, rewards finally stop.

The queue is set up this way on purpose to keep people in the same positions in the queue.

[Rob Andrews]

Im going to go ahead and start a new testnet thread for POL, so we can start nice and fresh.

[Rob Andrews]

My post from BCT forum, in case it's missed:

I wonder if it's coincidence that a large proportion of the MN's have just just gone to WatchDog Expired states ??

Code Select Expand

ENABLED 36
NEW_START_REQUIRED 3
WATCHDOG_EXPIRED 83
PRE_ENABLED 0
UPDATE_REQUIRED 0
EXPIRED 0

https://biblepay.eu/


Those might be running 1.0.8.2.  If the node actually Stopped, their payments will stop.

Anyone who owns one of those 20 nodes, please come online and tell us what happened.

Either way you have to upgrade to 1.0.8.6 to stay up without a crash.

The problem was fixed in 1.0.8.6


On https://biblepay.eu/ we can see the version distribution which is actually protocol distribution, but as I see in GitHub commits, 70715 was introduced in 1.0.8.4? So they are probably not running 1.0.8.2 (right?), but also we don't know if they are running 1.0.8.4-5, or 1.0.8.6+. Maybe Virus can pull the version data to his site, if not, maybe protocol version can be increased with every version, if that makes sense? (I doubt it makes sense )

Either way, my MN went to EXPIRED state last night and it's on 1.0.8.7. When I checked it, the daemon had crashed. These are the last lines of debug.log before crashing:

Code Select Expand

2018-01-30 04:12:30 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=b2e987cffe06e8476276f7a88287362e18e60e74, pubkeyFromSig=0fb4c558fb869b2fb813246c419180a01d316684, strMessage=36472b9df495a7fabe447fa9ddafb5423af5e311166fc02b9a5a6b73e8fcfc39-1|ed9ec4a65237b7d54f1766dffe39f1fb18cbc99783c3495fb927610f3c3620a1|1|1|1514729555, vchSig=HKq4s+Yl2VQ42mCFLy84mHKXx0Mludl3BZ+kq2Pv9KUKF3qTUIWWjPA8Yb/8Fg96MzKUfl+HntxlAsW7Di6Eizs=
2018-01-30 04:12:31 Disconnecting unauthorized peer with Network Time off by 7201.000000 seconds!^M
2018-01-30 04:14:37 Disconnecting unauthorized peer with Network Time off by 32401.000000 seconds!^M
2018-01-30 04:19:38 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=0fb4c558fb869b2fb813246c419180a01d316684, pubkeyFromSig=b2e987cffe06e8476276f7a88287362e18e60e74, strMessage=82e9a4def54f95dbc71e1fc2ed452c87e92cc44d004026b7774379d758b9e356-0|df3815d09091445c1ce839e7cd29f995abf117837d74525da866b51bbdb7bc16|1|1|1514729478, vchSig=GxNu58oFKlDbVCwpq6cFO0iIAm0GUXP7qsPtVDh5oGY5ErU9GC1SmamKLoaNXr0tI1CJbqavmyU2WKGkTIz1f/I=
2018-01-30 04:19:38 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=b2e987cffe06e8476276f7a88287362e18e60e74, pubkeyFromSig=0fb4c558fb869b2fb813246c419180a01d316684, strMessage=36472b9df495a7fabe447fa9ddafb5423af5e311166fc02b9a5a6b73e8fcfc39-1|df3815d09091445c1ce839e7cd29f995abf117837d74525da866b51bbdb7bc16|1|1|1514729478, vchSig=G4TVJF+oPO5QE6+5xFPWn7w7P25i0zSCgoTSEdq/2u1ZZbwZQPjFfK+G8wz58ULVldQkUAZCUpUvqOqVEfAdrt4=
2018-01-30 04:20:40 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=0fb4c558fb869b2fb813246c419180a01d316684, pubkeyFromSig=b2e987cffe06e8476276f7a88287362e18e60e74, strMessage=82e9a4def54f95dbc71e1fc2ed452c87e92cc44d004026b7774379d758b9e356-0|df3815d09091445c1ce839e7cd29f995abf117837d74525da866b51bbdb7bc16|1|1|1514729478, vchSig=GxNu58oFKlDbVCwpq6cFO0iIAm0GUXP7qsPtVDh5oGY5ErU9GC1SmamKLoaNXr0tI1CJbqavmyU2WKGkTIz1f/I=
2018-01-30 04:20:40 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=b2e987cffe06e8476276f7a88287362e18e60e74, pubkeyFromSig=0fb4c558fb869b2fb813246c419180a01d316684, strMessage=36472b9df495a7fabe447fa9ddafb5423af5e311166fc02b9a5a6b73e8fcfc39-1|df3815d09091445c1ce839e7cd29f995abf117837d74525da866b51bbdb7bc16|1|1|1514729478, vchSig=G4TVJF+oPO5QE6+5xFPWn7w7P25i0zSCgoTSEdq/2u1ZZbwZQPjFfK+G8wz58ULVldQkUAZCUpUvqOqVEfAdrt4=
2018-01-30 04:21:50 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=0fb4c558fb869b2fb813246c419180a01d316684, pubkeyFromSig=b2e987cffe06e8476276f7a88287362e18e60e74, strMessage=82e9a4def54f95dbc71e1fc2ed452c87e92cc44d004026b7774379d758b9e356-0|ed9ec4a65237b7d54f1766dffe39f1fb18cbc99783c3495fb927610f3c3620a1|1|1|1514729555, vchSig=HELzseIXLZTQQaCa2MqjUP9VpuCOlQ6zrgBrrRlmq6avSg1waraLOfbXYkYtLVs/c/T2xY6kwhuYN51+OuYOjjo=
2018-01-30 04:21:50 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=b2e987cffe06e8476276f7a88287362e18e60e74, pubkeyFromSig=0fb4c558fb869b2fb813246c419180a01d316684, strMessage=36472b9df495a7fabe447fa9ddafb5423af5e311166fc02b9a5a6b73e8fcfc39-1|ed9ec4a65237b7d54f1766dffe39f1fb18cbc99783c3495fb927610f3c3620a1|1|1|1514729555, vchSig=HKq4s+Yl2VQ42mCFLy84mHKXx0Mludl3BZ+kq2Pv9KUKF3qTUIWWjPA8Yb/8Fg96MzKUfl+HntxlAsW7Di6Eizs=
2018-01-30 04:22:28 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=0fb4c558fb869b2fb813246c419180a01d316684, pubkeyFromSig=b2e987cffe06e8476276f7a88287362e18e60e74, strMessage=82e9a4def54f95dbc71e1fc2ed452c87e92cc44d004026b7774379d758b9e356-0|ed9ec4a65237b7d54f1766dffe39f1fb18cbc99783c3495fb927610f3c3620a1|1|1|1514729555, vchSig=HELzseIXLZTQQaCa2MqjUP9VpuCOlQ6zrgBrrRlmq6avSg1waraLOfbXYkYtLVs/c/T2xY6kwhuYN51+OuYOjjo=
2018-01-30 04:22:28 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=b2e987cffe06e8476276f7a88287362e18e60e74, pubkeyFromSig=0fb4c558fb869b2fb813246c419180a01d316684, strMessage=36472b9df495a7fabe447fa9ddafb5423af5e311166fc02b9a5a6b73e8fcfc39-1|ed9ec4a65237b7d54f1766dffe39f1fb18cbc99783c3495fb927610f3c3620a1|1|1|1514729555, vchSig=HKq4s+Yl2VQ42mCFLy84mHKXx0Mludl3BZ+kq2Pv9KUKF3qTUIWWjPA8Yb/8Fg96MzKUfl+HntxlAsW7Di6Eizs=
2018-01-30 04:25:19 DSEG -- Sent 121 Masternode invs to peer 47696
2018-01-30 04:25:35 Disconnecting unauthorized peer with Network Time off by 14570.000000 seconds!^M
2018-01-30 04:27:14 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=0fb4c558fb869b2fb813246c419180a01d316684, pubkeyFromSig=b2e987cffe06e8476276f7a88287362e18e60e74, strMessage=82e9a4def54f95dbc71e1fc2ed452c87e92cc44d004026b7774379d758b9e356-0|df3815d09091445c1ce839e7cd29f995abf117837d74525da866b51bbdb7bc16|1|1|1514729478, vchSig=GxNu58oFKlDbVCwpq6cFO0iIAm0GUXP7qsPtVDh5oGY5ErU9GC1SmamKLoaNXr0tI1CJbqavmyU2WKGkTIz1f/I=
2018-01-30 04:27:14 CGovernanceVote::IsValid -- VerifyMessage() failed, error: Keys don't match: pubkey=b2e987cffe06e8476276f7a88287362e18e60e74, pubkeyFromSig=0fb4c558fb869b2fb813246c419180a01d316684, strMessage=36472b9df495a7fabe447fa9ddafb5423af5e311166fc02b9a5a6b73e8fcfc39-1|df3815d09091445c1ce839e7cd29f995abf117837d74525da866b51bbdb7bc16|1|1|1514729478, vchSig=G4TVJF+oPO5QE6+5xFPWn7w7P25i0zSCgoTSEdq/2u1ZZbwZQPjFfK+G8wz58ULVldQkUAZCUpUvqOqVEfAdrt4=
2018-01-30 04:27:39 Disconnecting unauthorized peer with Network Time off by 3575.000000 seconds!^M
2018-01-30 04:28:01 Disconnecting unauthorized peer with Network Time off by 7200.000000 seconds!

There is a lot more of these errors repeated, this is just a snippet of the last ones. I hope this can help.

Who is on our network with these clocks set off by 14 minutes?  Cmon, we are hit with every attack on a Christian Community possible huh?
It even looks like there are various nodes with varying timestamps 3575, 7200, 10000+ etc.

Anyway I cant tell what caused it to crash, but most likely whatever I merged in to fix instantsend, and the invalid vote.  The thing that is challening is Im running some sancs myself and do not mind valgrinding this, but Im not down so its a kind of elusive problem.

Blue, do you have more than 4Gb ram on your sanc?  If you do can you do me a favor?

sudo apt-get install valgrind

cd biblepay/src/qt
valgrind biblepay-qt

Run your sanc in valgrind with the defaults of valgrind set?

If you can do that next time you crash, you can send me the valgrind report which will contain the actual line number of source code that crashed it.

Thanks!

PS If your node is only 2gb RAM, its not worth it.
Btw, Ill run valgrind on my big dog also.

Regarding the protocol version, we can only up that when a large change occurs, but getinfos "version" is a hybrid of the software version and the protocol version, so Virus may be able to use that.

[Rob Andrews]

My MN is actually on a 512 MB RAM instance. And I don't have biblepay-qt nor the dependencies for it, because I installed only CLI. Is there any other way to record crash logs?

Also, now that you've mentioned it, those clock delays in seconds are almost exactly: 1 hour, 2 hours, 4 hours and 9 hours. Hmm, what could be the meaning of that, or what are they trying to achieve with that?

No other way to achieve crash logs- even if you ran with all the switches (instantsend=true, debugmaster=true, masternode=true) etc, would not guarantee that line that caused the segfault would be in the log.  The only thing in the log is what we write to the log.  However valgrind intakes its pointer to the line of source before executing each line, so it is really the only effective way.   

The time attack could be either a) to try to connect and slip a proof-of-work block in that has been pre-hashed on a supercomputer, (because they have the luxury of doing it over time), or b) to try to offset the GetNetworkTime() function - to adulterate it, to bring us as a network forward or back so they can inject a sttring of blocks to make money on their own fork.  We have NTP protection for past & future, of up to 10 minutes, but it relies on the network average time being correct, thats why we hang up on nodes who are off by more than 300 seconds.

[MIP]

Hi, I set up a MN on a Vultr instance (all exactly the same as in the instructions in http://wiki.biblepay.org/Create_Sanctuary_2) and everything went apparently fine.

However

$ ./biblepay-cli masternode status
{
  "vin": "CTxIn(COutPoint(0000000000000000000000000000000000000000000000000000000000000000, 4294967295), coinbase )",
  "service": "[::]:0",
  "status": "Not capable masternode: Masternode must accept connections from outside. Make sure listen configuration option is not overwritten by some another parameter."
}

I set up the firewall with rules as in the instructions, and did not create another extra firewall group at Vultr dashboard.

Any ideas for troubleshooting? Thank you


EDIT: just in case:

$  ./biblepay-cli mnsync status
{
  "AssetID": 999,
  "AssetName": "MASTERNODE_SYNC_FINISHED",
  "Attempt": 0,
  "IsBlockchainSynced": true,
  "IsMasternodeListSynced": true,
  "IsWinnersListSynced": true,
  "IsSynced": true,
  "IsFailed": false,
  "MasternodesEnabled": true
}

EDIT 2: ok it was my fault, for some reason I had "listen=0" in conf.

Now it says "Not capable masternode: Masternode not in masternode list"

I understand that it's fine now and I have to wait a few hours to get it listed, don't I?

thanks again!

EDIT 3: now it says "Masternode successfully created", but controller wallet still says "PRE_ENABLED". I will wait few hours more and see...