Bible Pay

Read 217457 times

  • Rob Andrews
  • Administrator

    • 4097


    • 97
    • June 05, 2017, 08:09:04 PM
    • Patmos, Island Of
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #600 on: July 24, 2019, 10:56:53 PM »
cli -version
BiblePay Core RPC client version 1.4.6.0

cli getblockhash 168736
c317c3de030df7c215f39b8568e9065b71a9e8e16af8d1fff5a09809e1ad18cd

leaderboard
Code: [Select]
{
 "Prominence": "Details",
 "CAMEROON-ONE: yUNSEjjtC9pdeHp4spswdFWh1npfV5Jvqe [N/A], Pts: 0.00": "0.00%",
 "CAMEROON-ONE: yfqGyVvuyidYytq5o2QvN1VdVeXtH9Lrkt [oncoapop1], Pts: 2667.00": "0.67%",
 "HEALING: yUNSEjjtC9pdeHp4spswdFWh1npfV5Jvqe [N/A], Pts: 0.00": "0.00%",
 "HEALING: yfqGyVvuyidYytq5o2QvN1VdVeXtH9Lrkt [oncoapop1], Pts: 111670149.00": "0.00%",
 "POG: yUNSEjjtC9pdeHp4spswdFWh1npfV5Jvqe [N/A], Pts: 596289318.00": "65.00%",
 "POG: yfqGyVvuyidYytq5o2QvN1VdVeXtH9Lrkt [oncoapop1], Pts: 0.00": "0.00%",
 "Healing": "Diary Entries",
 "oncoapop1": "Prayed for the salvation of the saints.",
 "Prominence": "Totals",
 "ALL: yUNSEjjtC9pdeHp4spswdFWh1npfV5Jvqe [N/A], Pts: 596289318.00, Reward: 537521.670": "65.000%",
 "ALL: yfqGyVvuyidYytq5o2QvN1VdVeXtH9Lrkt [oncoapop1], Pts: 111672816.00, Reward: 5553.030": "0.670%"
}

(my VPS is too small for GUI, so have to reply on cli, sorry)
cli exec analyze 168736 oncoapop1
Code: [Select]
{
 "Command": "analyze",
 "Campaign": "Totals",
 "0": "CAMEROON-ONE|yfqGyVvuyidYytq5o2QvN1VdVeXtH9Lrkt|2667|0.00671502|oncoapop1|2668",
 "1": "HEALING|yfqGyVvuyidYytq5o2QvN1VdVeXtH9Lrkt|111670149|0.00000000|oncoapop1|111670150",
 "2": "POG|yfqGyVvuyidYytq5o2QvN1VdVeXtH9Lrkt|0|0.00000000|oncoapop1|596289319",
 "3": "",
 "Campaign": "Points",
 "0": "User: yfqGyVvuyidYytq5o2QvN1VdVeXtH9Lrkt, Diary: , Height: 168733.00, TXID: fba1f5a89c130c8a1b20b822261c3bca93c732ced1b96c1e2d692c9f99bdec09, NickName: oncoapop1, Points: 2666.67, Campaign: CAMEROON-ONE, CoinAge: 56159643.2778, Donation: 0.0000, UserTotal: 2666.67",
 "1": "User: yfqGyVvuyidYytq5o2QvN1VdVeXtH9Lrkt, Diary: Prayed for the salvation of the saints., Height: 168733.00, TXID: 82758735ba9c7f2b63a477f15f1bdb98b245ae1e61e20ca1fef8b982e572af91, NickName: oncoapop1, Points: 111670149.47, Campaign: HEALING, CoinAge: 111670149.4654, Donation: 0.0000, UserTotal: 111672816.13",
 "2": ""
}


Ok, I see you in the leaderboard now.  (See pic).  Both of us look OK.

Yes, I see your exec analyze 168736.  The issue earlier was my machine was stopped on a bad llmq block before your block so it was actually me that was out.  (I just checked in a new patch that allows us to sync from zero with llmq activated).

Looks like we still need another sanc, I think MIP had one, MIP is yours down?



  • Rob Andrews
  • Administrator

    • 4097


    • 97
    • June 05, 2017, 08:09:04 PM
    • Patmos, Island Of
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #601 on: July 25, 2019, 08:38:24 AM »
As a side tip for anyone who is upgrading, remember you can do the 'exec reassesschains' if you end up at a lower height than us (it worked for me) due to LLMQ errors.

In the latest version I increased the LLMQ start height to 170,000.

Starting up LLMQ is much trickier (and dangerous) than I expected.

Since LLMQ drives chainlocks, the wallet is going to throw a bad block error, mark the block as dirty and put the wallet in a non recoverable state if it finds any block greater than 170,000 that is not in a quorum.

What this means is either the network has a lot of sancs, and a healthy quorum environment, or absolutely fails with a nightmare scenario.

This is obviously a decision Dash made to ensure there are no exceptions to the quorums once they are up and running. 

So in the current state of testnet, we would need to try to bring one more sanc on before 170,000 and see if a quorum forms, otherwise we need to keep increasing the LLMQ height.

What frightens me is if we start the quorums and then take down 70% of the nodes.  I think that means, we will need to regroup and bring the sancs back online. 

But from what I see, if lets say we lose those sancs VMs, and recreate them all, then the sigs will be new and the old quorums will be invalid (and I think that means a chain rollback).  We will cross that bridge when we come to it as our prod environment should be OK, as its always going to have for the most part more than 100 reliable sancs, so theoretically the quorums will never fail.




  • oncoapop
  • Full Member

    • 171


    • 17
    • October 23, 2018, 12:31:17 PM
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #602 on: July 26, 2019, 04:16:34 AM »
As a side tip for anyone who is upgrading, remember you can do the 'exec reassesschains' if you end up at a lower height than us (it worked for me) due to LLMQ errors.

In the latest version I increased the LLMQ start height to 170,000.

Starting up LLMQ is much trickier (and dangerous) than I expected.

Since LLMQ drives chainlocks, the wallet is going to throw a bad block error, mark the block as dirty and put the wallet in a non recoverable state if it finds any block greater than 170,000 that is not in a quorum.

What this means is either the network has a lot of sancs, and a healthy quorum environment, or absolutely fails with a nightmare scenario.

This is obviously a decision Dash made to ensure there are no exceptions to the quorums once they are up and running. 

So in the current state of testnet, we would need to try to bring one more sanc on before 170,000 and see if a quorum forms, otherwise we need to keep increasing the LLMQ height.

What frightens me is if we start the quorums and then take down 70% of the nodes.  I think that means, we will need to regroup and bring the sancs back online. 

But from what I see, if lets say we lose those sancs VMs, and recreate them all, then the sigs will be new and the old quorums will be invalid (and I think that means a chain rollback).  We will cross that bridge when we come to it as our prod environment should be OK, as its always going to have for the most part more than 100 reliable sancs, so theoretically the quorums will never fail.

Dear Rob,

Given your stressing the importance in testnet of the sancs testing of LLMQ (kindly remind me what this is again?), I have (surprisingly) resurrected one of my initial DIP3 sancs (just using old biblepay.conf and masternode.conf files left on the server WITHOUT the controlling wallet which I may have destroyed....), so I am hoping that it will be okay.

Let's test this to ensure, as far as we are able with a fraction of the network in testnet, we roll out a stable product in mainnet. Could you kindly help us to assist you in this by providing a checklist  (in simple layman's terms, please. Thanks  :) )  so that we can test specific aspects and report on.

Blessings,
oncoapop


Code: [Select]
{
  "outpoint": "51a7d0cdb93fb3377d274f1b448af3c260618eb23c145a52c6c2fc081192b4dc-1",
  "service": "45.62.239.200:40001",
  "proTxHash": "3fdfa35533185427856e07d7a966714d118c7434994b5d6a349c11558e79e290",
  "collateralHash": "51a7d0cdb93fb3377d274f1b448af3c260618eb23c145a52c6c2fc081192b4dc",
  "collateralIndex": 1,
  "dmnState": {
    "service": "45.62.239.200:40001",
    "registeredHeight": 65387,
    "lastPaidHeight": 171297,
    "PoSePenalty": 0,
    "PoSeRevivedHeight": -1,
    "PoSeBanHeight": -1,
    "revocationReason": 0,
    "ownerAddress": "yhDJCUZq19CVLiuDa4wEoY4PxWg17bZuTx",
    "votingAddress": "yhDJCUZq19CVLiuDa4wEoY4PxWg17bZuTx",
    "payoutAddress": "yRKdex8fFcDyjqx618Cz4hsQcQFjp55jRg",
    "pubKeyOperator": "05f58c1b79e898cf31a0375ded7a8e5bb6b0e5aacb48dc79f7c87e38eb0533bdf85c610cee5cf94b79c425c20f423cc4"
  },
  "state": "READY",
  "status": "Ready"
}



  • oncoapop
  • Full Member

    • 171


    • 17
    • October 23, 2018, 12:31:17 PM
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #603 on: July 26, 2019, 04:29:24 AM »
Dear Rob,

Given your stressing the importance in testnet of the sancs testing of LLMQ (kindly remind me what this is again?), I have (surprisingly) resurrected one of my initial DIP3 sancs (just using old biblepay.conf and masternode.conf files left on the server WITHOUT the controlling wallet which I may have destroyed....), so I am hoping that it will be okay.

Let's test this to ensure, as far as we are able with a fraction of the network in testnet, we roll out a stable product in mainnet. Could you kindly help us to assist you in this by providing a checklist  (in simple layman's terms, please. Thanks  :) )  so that we can test specific aspects and report on.

Blessings,
oncoapop


Code: [Select]
{
  "outpoint": "51a7d0cdb93fb3377d274f1b448af3c260618eb23c145a52c6c2fc081192b4dc-1",
  "service": "45.62.239.200:40001",
  "proTxHash": "3fdfa35533185427856e07d7a966714d118c7434994b5d6a349c11558e79e290",
  "collateralHash": "51a7d0cdb93fb3377d274f1b448af3c260618eb23c145a52c6c2fc081192b4dc",
  "collateralIndex": 1,
  "dmnState": {
    "service": "45.62.239.200:40001",
    "registeredHeight": 65387,
    "lastPaidHeight": 171297,
    "PoSePenalty": 0,
    "PoSeRevivedHeight": -1,
    "PoSeBanHeight": -1,
    "revocationReason": 0,
    "ownerAddress": "yhDJCUZq19CVLiuDa4wEoY4PxWg17bZuTx",
    "votingAddress": "yhDJCUZq19CVLiuDa4wEoY4PxWg17bZuTx",
    "payoutAddress": "yRKdex8fFcDyjqx618Cz4hsQcQFjp55jRg",
    "pubKeyOperator": "05f58c1b79e898cf31a0375ded7a8e5bb6b0e5aacb48dc79f7c87e38eb0533bdf85c610cee5cf94b79c425c20f423cc4"
  },
  "state": "READY",
  "status": "Ready"
}

Again I notice that all the sancs appear to be "valid" although I know that only one of my sancs (45.62.239.200) is online and the other (64.180.194.238) is  not (see below). Would this affect the quorum if we have zombie sancs?

Code: [Select]
telnet 45.62.239.200 40001
Trying 45.62.239.200...
Connected to 45.62.239.200.

telnet 64.180.194.238 40001
Trying 64.180.194.238...
telnet: Unable to connect to remote host: Connection refused


  • Rob Andrews
  • Administrator

    • 4097


    • 97
    • June 05, 2017, 08:09:04 PM
    • Patmos, Island Of
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #604 on: July 27, 2019, 10:42:27 AM »
Again I notice that all the sancs appear to be "valid" although I know that only one of my sancs (45.62.239.200) is online and the other (64.180.194.238) is  not (see below). Would this affect the quorum if we have zombie sancs?

Code: [Select]
telnet 45.62.239.200 40001
Trying 45.62.239.200...
Connected to 45.62.239.200.

telnet 64.180.194.238 40001
Trying 64.180.194.238...
telnet: Unable to connect to remote host: Connection refused

Well actually POSE gets enabled in tandem with LLMQ (long living masternode quorums).  So technically, when the 4 good sancs start forming quorums, they will start d-dossing the other sancs.  This will cause the other sancs POSE scores to increase.  Once they move to 100%, they will be banned and wont get paid.

The reason you see 0 for every sancs pose score is the LLMQs havent been forming yet.
I did ensure both the Dip and the LLMQ height were activated in testnet; so next I need to do some deeper analysis; Ill look today.

Let me know if you need more info on LLMQ other than this; you can read this and this should give you most of the info:
https://blog.dash.org/mitigating-51-attacks-with-llmq-based-chainlocks-7266aa648ec9

This too:
https://github.com/dashpay/dips/blob/master/dip-0006.md
« Last Edit: July 27, 2019, 01:02:28 PM by Rob Andrews »


  • Rob Andrews
  • Administrator

    • 4097


    • 97
    • June 05, 2017, 08:09:04 PM
    • Patmos, Island Of
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #605 on: July 27, 2019, 10:46:48 AM »
Dear Rob,

Given your stressing the importance in testnet of the sancs testing of LLMQ (kindly remind me what this is again?), I have (surprisingly) resurrected one of my initial DIP3 sancs (just using old biblepay.conf and masternode.conf files left on the server WITHOUT the controlling wallet which I may have destroyed....), so I am hoping that it will be okay.

Let's test this to ensure, as far as we are able with a fraction of the network in testnet, we roll out a stable product in mainnet. Could you kindly help us to assist you in this by providing a checklist  (in simple layman's terms, please. Thanks  :) )  so that we can test specific aspects and report on.

Blessings,
oncoapop


Code: [Select]
{
  "outpoint": "51a7d0cdb93fb3377d274f1b448af3c260618eb23c145a52c6c2fc081192b4dc-1",
  "service": "45.62.239.200:40001",
  "proTxHash": "3fdfa35533185427856e07d7a966714d118c7434994b5d6a349c11558e79e290",
  "collateralHash": "51a7d0cdb93fb3377d274f1b448af3c260618eb23c145a52c6c2fc081192b4dc",
  "collateralIndex": 1,
  "dmnState": {
    "service": "45.62.239.200:40001",
    "registeredHeight": 65387,
    "lastPaidHeight": 171297,
    "PoSePenalty": 0,
    "PoSeRevivedHeight": -1,
    "PoSeBanHeight": -1,
    "revocationReason": 0,
    "ownerAddress": "yhDJCUZq19CVLiuDa4wEoY4PxWg17bZuTx",
    "votingAddress": "yhDJCUZq19CVLiuDa4wEoY4PxWg17bZuTx",
    "payoutAddress": "yRKdex8fFcDyjqx618Cz4hsQcQFjp55jRg",
    "pubKeyOperator": "05f58c1b79e898cf31a0375ded7a8e5bb6b0e5aacb48dc79f7c87e38eb0533bdf85c610cee5cf94b79c425c20f423cc4"
  },
  "state": "READY",
  "status": "Ready"
}

Great on adding another sanc!

Yes, I will make a list; but know that I have no intention of moving towards prod til after we fully believe testnet and chainlocks are tested in and out :).

There are so many things we need to test after Prod transitions to deterministic sancs in 0.14 against prod.  We definitely need a list, and it will make me feel better when we not only test chainlocks in testnet, but LLMQs against the deterministic sancs in prod.

We will see the GSC gov data and votes in testnet after prod upgrades to determistic.  LLMQ will also be enabled in .13 in prod, but won't be enforced like it is in testnet.



  • Rob Andrews
  • Administrator

    • 4097


    • 97
    • June 05, 2017, 08:09:04 PM
    • Patmos, Island Of
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #606 on: July 27, 2019, 02:45:50 PM »
Again I notice that all the sancs appear to be "valid" although I know that only one of my sancs (45.62.239.200) is online and the other (64.180.194.238) is  not (see below). Would this affect the quorum if we have zombie sancs?

Code: [Select]
telnet 45.62.239.200 40001
Trying 45.62.239.200...
Connected to 45.62.239.200.

telnet 64.180.194.238 40001
Trying 64.180.194.238...
telnet: Unable to connect to remote host: Connection refused

So I took at look at the "bottleneck" in TestNet (with POSE not working) - IE in laymans terms, Sancs are not banning each other.
Also we have an issue in TestNet where LLMQs are failing.

So, the problem is our LLMQ in TestNet is pointing to params that are for the wrong quorum.  (I did update them at one point, and triple checked them, but Dash's testnet is bigger than ours, and that led me to believe we were using RegTest in TestNet and they were using 5_60 in TestNet but in reality, Dash was requiring 7 masternodes in TestNet minimum, and our RegTest params were only being used in RegTest LOL).  So thats good this was discovered and makes sense, and explains why our LLMQ quorums have all failed so far.

So, we will need a mandatory in TestNet (I need to force a protocol version increase) in order to release this next test.

So MIP, lets build a new testnet release.

We will announce asap.

EDIT:  Please remember to erase your debug.log before starting the new version.

« Last Edit: July 27, 2019, 03:01:37 PM by Rob Andrews »


  • Rob Andrews
  • Administrator

    • 4097


    • 97
    • June 05, 2017, 08:09:04 PM
    • Patmos, Island Of
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #607 on: July 28, 2019, 02:26:45 PM »
BiblePay
1.4.6.1-Mandatory Upgrade for TestNet

- Bump protocol version up to 70740 for Sancs (to force upgrade)
- Modify minimum LLMQ for testnet to be a size of 3 with a threshhold of 2
- Use Testnet LLMQ for instantsend and for ChainLocks
- Increase LLMQ height to 175000 in testnet


  • Rob Andrews
  • Administrator

    • 4097


    • 97
    • June 05, 2017, 08:09:04 PM
    • Patmos, Island Of
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #608 on: July 28, 2019, 06:56:44 PM »
Great news!

It looks like the latest version is POSE banning people!

POSE, here we come!

EDIT:  Wait, one of my own sancs got POSE banned.
Guys, please ensure your Masternodeblsprivkey is set correctly btw.

« Last Edit: July 28, 2019, 06:59:33 PM by Rob Andrews »


  • Rob Andrews
  • Administrator

    • 4097


    • 97
    • June 05, 2017, 08:09:04 PM
    • Patmos, Island Of
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #609 on: July 28, 2019, 07:35:41 PM »
Great news!

It looks like the latest version is POSE banning people!

POSE, here we come!

EDIT:  Wait, one of my own sancs got POSE banned.
Guys, please ensure your Masternodeblsprivkey is set correctly btw.

In my case it was not the masternodeblsprivkey (but that is still the most common cause of POSE bans).
In my case I had the externalip= set (to the correct IP) but for some reason the other nodes didnt like that, so I recreated my sanc. (without an externalip).

Now Im back in the list with 0 pose ban.

Im interested in seeing who else gets POSE banned.

LLMQ is enabled now.

MIP do you have a sanc also?  I know Oncoapop just added one so we should have 5 total if MIP has one.



  • MIP
  • Sr. Member

    • 365


    • 47
    • February 13, 2018, 11:55:52 AM
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #610 on: July 29, 2019, 03:57:34 PM »
In my case it was not the masternodeblsprivkey (but that is still the most common cause of POSE bans).
In my case I had the externalip= set (to the correct IP) but for some reason the other nodes didnt like that, so I recreated my sanc. (without an externalip).

Now Im back in the list with 0 pose ban.

Im interested in seeing who else gets POSE banned.

LLMQ is enabled now.

MIP do you have a sanc also?  I know Oncoapop just added one so we should have 5 total if MIP has one.

I had one, I just updated but I think I may be in the wrong chain, I will resync.


  • Rob Andrews
  • Administrator

    • 4097


    • 97
    • June 05, 2017, 08:09:04 PM
    • Patmos, Island Of
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #611 on: July 29, 2019, 03:59:02 PM »
I had one, I just updated but I think I may be in the wrong chain, I will resync.

Im writing a script now to do a one-click sync, but it wont be ready til next version as I have to upload a file into github.



  • Rob Andrews
  • Administrator

    • 4097


    • 97
    • June 05, 2017, 08:09:04 PM
    • Patmos, Island Of
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #612 on: July 30, 2019, 02:06:51 PM »
** Low hassle Syncing from Zero **

As our blockchain gets bigger it will be useful for us to release utilities to allow one click syncing from zero.
This first version is only for testnet and linux.  Later, I will extend this to windows and prod.

Is anyone interested in testing?  I just tested this from my ubuntu remote box and I was able to sync in a couple minutes.

I will check this into github later, but for now, let me give manual instructions to use the script:

From your linux box:

cd ~/biblepay-evolution
(This is where your source starts, ie /src is one folder down)
wget biblepay.org/syncblocks_testnet.sh
chmod 777 syncblocks_testnet.sh

Now to get in sync for TestNet only, the script automatically deletes just the data files (dont worry, it wont delete anything else), then it pulls down the snapshot of the blocks gzipped and unzips them into the correct places (including the llmq and all necessary governance files), then its up to you re-launch the wallet.  The script does close biblepay as long as your machine supports pkill.

To sync from 0 type:
./syncblocks_testnet.sh

EDIT:

I just created a windows version and tested it on windows 7 and it works.  Will check-in during the next Develop release.

« Last Edit: July 30, 2019, 03:20:31 PM by Rob Andrews »


  • Rob Andrews
  • Administrator

    • 4097


    • 97
    • June 05, 2017, 08:09:04 PM
    • Patmos, Island Of
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #613 on: July 30, 2019, 05:07:37 PM »
So can we get a little update from everyone on testnet, do you all agree:


getblockhash 180650
28383ec96296dbee95f45a0240cae97ef8ef06ff8fd3fe4e90552eb2cbe72d6e

Id like to see all 5 of our sancs in agreement before we talk about testing chainlocks.
I see my 3 non-pose banned sancs are in agreement.

MIP, Oncoapop we could use feedback - and anyone else that wants to test chainlocks.




  • oncoapop
  • Full Member

    • 171


    • 17
    • October 23, 2018, 12:31:17 PM
    more
Re: TestNet - BiblePay-Evolution & GSCs (Generic Smart Contracts)
« Reply #614 on: July 30, 2019, 07:17:37 PM »
So can we get a little update from everyone on testnet, do you all agree:


getblockhash 180650
28383ec96296dbee95f45a0240cae97ef8ef06ff8fd3fe4e90552eb2cbe72d6e

Id like to see all 5 of our sancs in agreement before we talk about testing chainlocks.
I see my 3 non-pose banned sancs are in agreement.

MIP, Oncoapop we could use feedback - and anyone else that wants to test chainlocks.
Dear Rob,

I just have three VPS running in TESTNET. I did have one sanc but it is now banned, I have no idea how to get it unbanned. I restarted it to no avail. How does one get banned, other than not running bbp daemon?

Code: [Select]
{
 "outpoint": "0000000000000000000000000000000000000000000000000000000000000000-4294967295",
 "service": "45.62.239.200:40001",
 "state": "POSE_BANNED",
 "status": "Masternode was PoSe banned"
}

The VPS is up and running and port appears to be open
Code: [Select]
telnet 45.62.239.200 40001
Trying 45.62.239.200...
Connected to sanc2.myseqtools.com<http://sanc2.myseqtools.com>.

mnsync status
{
 "AssetID": 999,
 "AssetName": "MASTERNODE_SYNC_FINISHED",
 "AssetStartTime": 1564531134,
 "Attempt": 0,
 "IsBlockchainSynced": true,
 "IsSynced": true,
 "IsFailed": false
}

If you want to check my machines, you can see the automatically polled hashes of my three machines at any time on:
http://oncoapop.sdf.org/biblepaytest/testnet_chainstate.shtml

They are pretty puny VPS  so one or more may be offline syncing or updating…

Cheers,
Oncoapop