TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)

[Rob Andrews]

Ok I went to healthy on all my nodes, lets hope you all see the same.

'exec health'

[uptimeminer]

Switched to healthy=true on my node..

"Command": "health",
  "govobjhash": "a2ce1dd9f8fcb55ed32d62b3a071338928dc85f812f207ae918e04c38d49b4e1",
  "Amounts": "594237.00|299260.00|359384.00",
  "Addresses": "yS9YaCFpQuE6xpBrTvUbnPyJgwaXZep8C8|yUNSEjjtC9pdeHp4spswdFWh1npfV5Jvqe|ygavD5YuAJXpHDMLRyiiJ199YM5y9fTWfB",
  "votes": 3,
  "required_votes": 3,
  "last_superblock": 9040,
  "next_superblock": 9245,
  "next_superblock_triggered": false,
  "Healthy": true,
  "GSC_Voted_In": true

[Rob Andrews]

Switched to healthy=true on my node..

Yes, thank God we finally made it through the entire night and all the superblocks passed all the rules this time (we had about 4 so far I think).


getblockhash 10120

e13331a79a04776297f1ef66634525bb3d16aca8c9ff8d61a5be21f2c5202caf

I can see we are all still in sync also.  I forgot to mention, you can see by the difficulty that the chain is solid now as it jumped up to 1.4 (from .10 when we had the issues).


Great, now I can take a look at the proposal rules for the monthlies and see if we can expose the height, and test watchman. 


I'm going to be explaining a (what I consider to be) novel use of our new campaigns later today.

We can discuss bearing fruit in Jesus Name with Healing campaigns and spiritual warfare in Evo. 

I'm thinking we can add a CPK startup wizard, with the ability to read an instruction PDF, join a campaign that requires directions (as spiritual warfare for instance is dangerous if a person does not know what they are doing), and once they join, we also need the ability for a user to type in a days activity back into BiblePay.    This would allow the campaign to keep a diary of what the user is doing for the points (for these specific campaigns).

[Rob Andrews]

I've got a fun project coming up for us. 

So basically, here are the requirements for Watchman On The Wall we need to fulfill for replacement of The Sentinel, and be the first coin with our own integrated Watchman (not requiring python):

- Nodes get banned and not paid if they are not running watchman (this prevents people from running sancs that do not help vote on the budget)
- Nodes do not get voted on for a sanc payment if they are down (IE watchdog expired)
- Nodes who do run watchman can create a budget, sort it by votes, exclude proposals that pay dont fit, and vote on the proposal automatically during the warning period

We've got the first two covered (Watchman will automatically run in our sancs in c++, sancs will only vote on sancs who are Up (as we do test for that already), and we already have a spork for payment enforcement), and by the end of our testing, we will ensure Sanc don't expire prematurely--  so in light of all this, all that is left (the elephant in the room), is to test Watchman Budgeting.

In light of that, I am now modifying Watchman to handle budgets.  Where you all will come in to help, is we need to create about 10 *new* proposals (as the old ones are bad) -- for the next superblock (see 'getgovernanceinfo' to see the next budget is at height 11480, with getsuperblockbudget 11480 = 4.1MM bbp).

What we need to do in the future is : Verify proposals that missed a prior superblock deadline are no longer in the proposals list (this is broken currently), that future unpaid proposals are in the proposals list, and that watchman picks up these for budgeting, and sorts them by votes descending, and also excludes items that dont fit the cap.

What I want to do is have us all vote on these, and ensure the ones with low votes are not included and those that are too high for the budget cap are excluded automatically.

The current version *is* capable of taking new proposal now - for block 11480, so please, start now by creating 5-7 new proposals (please only create One at a time and then ensure the block has passed before creading another one).  I think if each of you create 2-3 that is sufficient.

Ill create a few myself now.

After the next release, the bugs above will be fixed and then we can test a watchman dry run at a height of 10% before block 11480 (that is the warning period) (IE 115 blocks early).

[oncoapop]

** Update **

So for the last few re-releases the common theme has been:
- No signs of a problem with abn weight, or gsc client side transactions, or creation of the pog points
- No signs of problems with actual POBH heat mining (IE no forks resulting from POW)
- No signs of problems on the *monthly* superblock budget system or its creation of budget data

- Every re-release so far and enhancement resulted from a problem related to the Daily superblock contract
(not a problem with the actual contract, a problem with the sancs not *finding* the contract, some find it, some lose it)

So I've been trying to hone in on this for the last 2 days - because at first I believed running only 3 sancs meant that one sanc out of 3 was not syncing properly- so we added rules to strengthen the minimum requirements of the voting required for a superblock to pass.  Yesterday however, all 3 sancs could see the votes on the superblock (we had 4), but only 2 of them had the actual gobject in memory (that is obviously a problem).

Let me explain this a different way, so we can all understand how a soft consensus can still fork the chain in testnet.
Sanc #1 and Sanc #2 had the gobject in memory (this is the GSC contract with its payments and addresses).  Sanc 1,2,and 3 had the *vote* for the gobject in memory and they all agreed the superblock was good and should pass.

Here is what happened on block 6990:

Sanc #1 found both the gobject and the trigger votes, and it said:  Valid, accepted
Sanc #2 found the votes only, and said "valid superblock at this height" but actually rejected the block because:  IsBlockValueValid, too few superblock payments <-- this is because it was missing the gobject with the hex data

So lets see what happens; Sanc #2 goes on its own chain now, Sanc 1 and 3 continue on the chain with the higher work.  This actually splits the testnet network because now Sanctuary votes for 1 cause (mnpayments to decrease), IE do to our mnpayments relying on chainheights- this becomes a hard split - and in a small
testnet network we cant have this, so therefore we have to fix the root of the problem.

This ends up causing a hard fork, because #2 will never re-correct its view of this block until completely resynced as it has now marked the block as a fail.

First, lets stop testing again until I can release a plan to fix this.
(It does not appear to be watchman).
It appears to be more of a timing issue.  In Dash, the gobjects are synced over a longer time span (as we normally have a month to get all the data for the monthly votes), but in our case, we are trying to sync (with 2 minute blocks) all the superblock data within 190 blocks (IE < 3 hours)-- so I can see if one sanc misses a gobject, its going to make a bad call on the next superblock.

So, my plan is this next:
1. I will investigate all code that syncs gobjects, and see if we are lacking something (Ie timing issues) and beef this up
2. I will do something special to allow us to Monitor the next daily gsc superblock, like a command line that will alert us that a node is going to make a bad call on the next block
3. We will make the sanc go out of its way to Pull the data it needs to make a good call on the block manually

Then we regroup here and do an in-depth analysis on the actual superblock heights as they are about to tick by and post something on the forum to see what the state of each one is as they occur. 

Please hang on for the next version, it might take a day, it might take less depending on how complex the timing issue is.

I appear to be on the correct chain by as far as I can tell (checking manually), I have not seen my sanc (the 4th) vote for any superblock. Does this have to be done manually?

bbpd@sanc2:~/Sanctuary$ cli getblockhash 10120
e13331a79a04776297f1ef66634525bb3d16aca8c9ff8d61a5be21f2c5202caf
bbpd@sanc2:~/Sanctuary$ cli exec health
{
  "Command": "health",
  "govobjhash": "9ff38ea222d3bb56085b2c696a10ddf02d16548cdb23beec3db828ae5ed8b929",
  "Amounts": "736194.00|180067.00|173696.00|101790.00|23491.00|37641.00",
  "Addresses": "yR26BVwFeGTnkgdfw1RRNgVF2LyPnrZqaT|yS9YaCFpQuE6xpBrTvUbnPyJgwaXZep8C8|yUNSEjjtC9pdeHp4spswdFWh1npfV5Jvqe|yfqGyVvuyidYytq5o2QvN1VdVeXtH9Lrkt|ygavD5YuAJXpHDMLRyiiJ199YM5y9fTWfB|yiWrpMBA8YXfVmxntAgZZ1dh18ueaBerbW",
  "votes": 2,
  "required_votes": 3,
  "last_superblock": 10680,
  "next_superblock": 10885,
  "next_superblock_triggered": true,
  "Healthy": true,
  "GSC_Voted_In": false
}

[Rob Andrews]

I appear to be on the correct chain by as far as I can tell (checking manually), I have not seen my sanc (the 4th) vote for any superblock. Does this have to be done manually?

bbpd@sanc2:~/Sanctuary$ cli getblockhash 10120
e13331a79a04776297f1ef66634525bb3d16aca8c9ff8d61a5be21f2c5202caf
bbpd@sanc2:~/Sanctuary$ cli exec health
{
  "Command": "health",
  "govobjhash": "9ff38ea222d3bb56085b2c696a10ddf02d16548cdb23beec3db828ae5ed8b929",
  "Amounts": "736194.00|180067.00|173696.00|101790.00|23491.00|37641.00",
  "Addresses": "yR26BVwFeGTnkgdfw1RRNgVF2LyPnrZqaT|yS9YaCFpQuE6xpBrTvUbnPyJgwaXZep8C8|yUNSEjjtC9pdeHp4spswdFWh1npfV5Jvqe|yfqGyVvuyidYytq5o2QvN1VdVeXtH9Lrkt|ygavD5YuAJXpHDMLRyiiJ199YM5y9fTWfB|yiWrpMBA8YXfVmxntAgZZ1dh18ueaBerbW",
  "votes": 2,
  "required_votes": 3,
  "last_superblock": 10680,
  "next_superblock": 10885,
  "next_superblock_triggered": true,
  "Healthy": true,
  "GSC_Voted_In": false
}

I think things are OK on the back end, but we might need another field - but first whats your sanc Ip, I can check some things to make sure I see your vote?

Btw, no - you should not have to vote manually, this is all automatic.

[oncoapop]

I think things are OK on the back end, but we might need another field - but first whats your sanc Ip, I can check some things to make sure I see your vote?

Btw, no - you should not have to vote manually, this is all automatic.

Thanks.

bbpd@sanc2:~$ ifconfig
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 45.62.239.200  netmask 255.255.255.0  broadcast 45.62.239.255
        inet6 fe80::250:56ff:fe95:4682  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:95:46:82  txqueuelen 1000  (Ethernet)
        RX packets 600708  bytes 190734759 (190.7 MB)
        RX errors 0  dropped 276  overruns 0  frame 0
        TX packets 530771  bytes 225023604 (225.0 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[Rob Andrews]

Thanks.

bbpd@sanc2:~$ ifconfig
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 45.62.239.200  netmask 255.255.255.0  broadcast 45.62.239.255
        inet6 fe80::250:56ff:fe95:4682  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:95:46:82  txqueuelen 1000  (Ethernet)
        RX packets 600708  bytes 190734759 (190.7 MB)
        RX errors 0  dropped 276  overruns 0  frame 0
        TX packets 530771  bytes 225023604 (225.0 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Yes, I see you as the 4th enabled sanc, great.

So trying to track this down took me longer than when it actually passed, so I found that it would be useful for us to have a couple more features.
1- The ability to see that the exec testgscvote is actually picking the most popular contract; right now I see 2 in there and it appears to be voting on the right one, but we cant really see it from the rpc
2- A way to filter the gobject get with a wildcard. 

Let me enhance the next version then we can test another one.

[Rob Andrews]

1.4.1.0-Leisure Upgrade for TestNet

- Add cascading GSC superblock creation (this decreases the propensity
for duplicate GSCs)
- Add intelligent GSC voting system (this means we vote yes on the
lowest PAM hashes and vote no on the highest PAM hashes)
- Add Watchman on the Wall budgeting
- Add 'gobject listwild all type wildcard' - This allows us to search
objects or proposals or gscs or triggers by wild card.  You can also
search by gov plain text data or by hash.
- Add getgschashes - This gives us a list of current gsc hashes for the
next superblock and pertinent data for each hash

[vuli]

very time the new wallet version is out, my wallet forked. I figured out, if I all of the sudden have several millions of tbbp than new version is out. 

[Rob Andrews]

So I have all the logs and I can see exactly what happened.
So we made it further this time, we passed through a couple days worth of superblocks without problems.
The problem is still with the GSC superblock (again), this time at height 11,500 it was rejected - half of the network accepted it and the other half rejected it causing a split at 11,501 and this is the reason half of our sancs now show Expired (they are on the wrong chain).

On the bright side, watchman-on-the-wall actually worked already and paid the proper amounts in the last monthly superblock and Im very happy about that.  Even though no one entered the proposals that we needed to test the overages - I had enough in there to at least test one overage (I had the block overpaid by 50K and it did successfully pick the first 3 highest voted that fit).

So looking at the root cause as to why two of my nodes rejected the block, I can see clearly that a non-sanctuary thought it was completely synced with the Sanc network (this has nothing to do with the chain sync, just the gobject sync) but in reality it did Not have the information it needed to make a good call. 

In light of this, what I plan on doing next is tackling this another way.

Please halt testing until the next version.

[Rob Andrews]

1.4.1.2b-TestNet Mandatory Upgrade

- Refine GSC Voter

[uptimeminer]

Hi Rob...How can I verify what version I am on Win32 wallet (testnet)...Help->About says 1.4.0.9 last couple of updates...

[Rob Andrews]

Hi Rob...How can I verify what version I am on Win32 wallet (testnet)...Help->About says 1.4.0.9 last couple of updates...

Hmmm, it appears the link changed.

Please see the OP post now (edited).

Thanks!

[MIP]

1.4.1.2b-TestNet Mandatory Upgrade

- Refine GSC Voter

Ready for MacOS

Regarding Ubuntu PPAs, I still have to figure out how to compile Chia-BLS and Relic into PPAs themselves, then add them as an internal dependency... not an easy task at the moment, will report any progress.