v1.4.2.8b, both Ubuntu, both command line daemon, one Ubuntu 16.04, other Ubuntu 18.04,
I was checking the log that didnt have debug=1 in it and couldnt see anything stick out,
by crash, I mean it started up and I typically keep entering "biblepay-cli getinfo" to check its progress, but it would give me some RPC error and biblepayd wouldnt be inside "ps -ef", and Id have to start it up again, and then it would crash again
I just downloaded the one that had debug=1 on, about to go through it, 80mb debug file
=
I believe this is the relevant section of the debug.log, hopefully it is and Im not wasting your time
(uploaded it to file hosting site, 4.7MB)
https://ufile.io/pqaao2vj
=
2019-05-08 04:50:28 ABN OK: 1.000000 LoadExternalBlockFile: Processing out of order child 7d37aaa250a86221b5ba434365aec3dc127183af7ccbabdec9367b7e9fa14719 of 4e5bcad04e7082d25315cfc964b463c70f8053de1e68013e532ce38f43d78d95
2019-05-08 04:50:28 ERROR: ReadBlockFromDisk: Deserialize or I/O error - ReadCompactSize(): size too large: iostream error at CBlockDiskPos(nFile=0, nPos=39353)
2019-05-08 04:50:28 *** Failed to read block
2019-05-08 04:50:28 Error: Error: A fatal internal error occurred, see debug.log for details
...
2019-05-08 04:50:28 Interrupting HTTP server
2019-05-08 04:50:28 Interrupting HTTP RPC server
2019-05-08 04:50:28 Interrupting RPC
2019-05-08 04:50:28 Exited http event loop
2019-05-08 04:50:28 addcon thread exit
2019-05-08 04:50:28 mncon thread exit
...
2019-05-08 04:50:28 PrepareShutdown: In progress...
2019-05-08 04:50:28 Stopping HTTP RPC server
2019-05-08 04:50:28 Unregistering HTTP handler for / (exactmatch 1)
2019-05-08 04:50:28 Stopping RPC
2019-05-08 04:50:28 RPC stopped.
2019-05-08 04:50:28 Stopping HTTP server
2019-05-08 04:50:28 Waiting for HTTP worker threads to exit
2019-05-08 04:50:28 Waiting for HTTP event thread to exit
2019-05-08 04:50:28 Stopped HTTP server
2019-05-08 04:50:28 CDBEnv::Flush: Flush(false)
2019-05-08 04:50:28 CDBEnv::Flush: wallet.dat checkpoint
2019-05-08 04:50:28 CDBEnv::Flush: wallet.dat detach
2019-05-08 04:50:28
BiblepayMiner -- terminated
0CDBEnv::Flush: wallet.dat closed
Is this the culprit?
2019-05-08 04:50:28 ERROR: ReadBlockFromDisk: Deserialize or I/O error - ReadCompactSize(): size too large: iostream error at CBlockDiskPos(nFile=0, nPos=39353)
2019-05-08 04:50:28 *** Failed to read block
2019-05-08 04:50:28 Error: Error: A fatal internal error occurred, see debug.log for details
=
From other logs it seems typically theres:
ThreadRPCServer method=stop
or with GUI
GUI: requestShutdown : Requesting shutdown
GUI: shutdown : Running Shutdown in thread
Thanks, got it (downloaded the file and analyzed).
At least I can tell you what exactly happened (that its not a systemic problem preventing release).
So in your case, your node ran and flushed the blocks db correctly on the prior run.
You ran it with reindex and it made it to height 21598 when it ran into a bad serialized block (IE the length pointer was longer than the maximum allowed data) meaning to biblepay, it believed the blocks database was corrupted. This obviously was probably not corrupted but something the reindex did to the file instead. For one it was reading the blocks out of order since height 1100, which is not too good for biblepay because of our contextual checkblock rules.
I personally have had a lot of problems with -reindex, so much that I stopped doing it (I always just sync from the network now).
My non-optimal solution is to create a blocks file download in prod weekly or monthly for our users who want high performance download of the initial blocks from biblepay.org/snapshot for example. We can make that a possibility when we hit around block 150K.
In this case, the program handled the exception but obviously exited and that left you with no choice but to resync the chain.
Note that -reindex does work when the blocks are read in sequentially, but they are not guaranteed to be sequential in berkeleydb.