Bible Pay

Read 7519 times

  • MIP
  • Sr. Member

    • 365


    • 47
    • February 13, 2018, 11:55:52 AM
    more
Iquidus Explorer: upgrade to support GETUTXO in API
« on: April 11, 2018, 02:04:23 PM »
We need Iquidus explorer to support GETUTXO in its API.

It's required to support the paper wallet sweep in mobile versions. I expected Breadwallet to support this via some SPV functionality but it was not the case.

SCOPE OF WORK

- Starting from current Iquidus Explorer master version, implement GETUTXO in explorer API

http://explorer.biblepay-central.org/ext/getutxos/{address}

Output format will be recognized by Biblepay mobile wallet.
(see sample in https://api.breadwallet.com/q/addr/3PauLgPVQu2r6QnvWvK1GDGXdeB3ouCdqw/utxo?currency=BTC)


TASK BREAKDOWN

- Fork Iquidus explorer project, install biblepaycore, install explorer on a dev ubuntu computer (2h)
- Change database load to store outpoint hash and index in transaction inputs (2h)
- Implement check_is_unspent as a query to know if an output has already been used (1h)
- Implement API route /ext/getutxos (2h)
- Explorer full load tests (2h)

Total Budget:
9h dev&test @40€/h = 360€
= 0,064285 BTC (@ 5600€) = 173745 BBP (@37 sat)

Work is very advanced, and it works pretty fast. It will be a nice addition that few coin explorers have.

All code will be open sourced.

I will need Lichtsucher's help to update biblepay-central explorer. I don't know if you have a test environment before we go live.
A full reindex of the database will be required so expect a few hours downtime.
« Last Edit: April 13, 2018, 02:16:57 AM by MIP »


  • jaapgvk
  • Hero Member

    • 558


    • 31
    • September 01, 2017, 08:02:57 PM
    • Netherlands
    more
Re: Iquidus Explorer: upgrade to support GETUTXO in API
« Reply #1 on: April 11, 2018, 04:51:23 PM »
You da man MIP 8)


  • MIP
  • Sr. Member

    • 365


    • 47
    • February 13, 2018, 11:55:52 AM
    more
Re: Iquidus Explorer: upgrade to support GETUTXO in API
« Reply #2 on: April 12, 2018, 10:07:59 AM »
Ok I tried to add outpoint info in inputs so with a simple query you could know if it's unspent or not.

However, there was a trap:  Iquidus explorer iterates through all transaction inputs, but doesn't record all of them, it creates only one per address and then accumulates the amounts over the same "virtual" input.  This is logical as it is what we expect to see in the user interface.

So my initial approach can't work.

My "plan B" approach is, when processing inputs, trying to recover back the original output  and "mark" it as spent, then save again.  It way harder to implement, but probably the only way to go without changing the database model drastically.

 


  • MIP
  • Sr. Member

    • 365


    • 47
    • February 13, 2018, 11:55:52 AM
    more
Re: Iquidus Explorer: upgrade to support GETUTXO in API
« Reply #3 on: April 12, 2018, 01:19:33 PM »
Plan B seemed to work. It's about maintaining a clear list of the address UTXOs.

It takes a bit more database space and longer to index (about twice the time) but the good point is that calling the getutxo API is 5 times faster than the other way. I don't expect the other queries to be affected.

As soon as I can contact Lichtsucher we will try to test it on a test environment.




  • MIP
  • Sr. Member

    • 365


    • 47
    • February 13, 2018, 11:55:52 AM
    more
Re: Iquidus Explorer: upgrade to support GETUTXO in API
« Reply #4 on: April 13, 2018, 02:17:04 AM »
Proposal created. Kindly consider to vote for this proposal, included in this month IT budget, if you find it interesting:

gobject vote-many 03aa25b80bf951bcaaed1589a52c8cd8cf58893912df176c2b4bafef47d90296 funding yes

Thenk you


  • znffal
  • Full Member

    • 129


    • 7
    • October 02, 2017, 04:01:47 PM
    more
Re: Iquidus Explorer: upgrade to support GETUTXO in API
« Reply #5 on: April 13, 2018, 04:54:05 AM »
Proposal created. Kindly consider to vote for this proposal, included in this month IT budget, if you find it interesting:

gobject vote-many 03aa25b80bf951bcaaed1589a52c8cd8cf58893912df176c2b4bafef47d90296 funding yes

Thenk you

Hi MIP,

I just wanted to say that you are doing a fantastic job. I will be voting for your proposals.
Also, I am not sure if others are like me, but I usually wait intil the last week before the superblock to vote, just so that I get a good idea of what is in the budget.

Thanks!


  • MIP
  • Sr. Member

    • 365


    • 47
    • February 13, 2018, 11:55:52 AM
    more
Re: Iquidus Explorer: upgrade to support GETUTXO in API
« Reply #6 on: April 13, 2018, 05:50:46 AM »
Hi MIP,

I just wanted to say that you are doing a fantastic job. I will be voting for your proposals.
Also, I am not sure if others are like me, but I usually wait intil the last week before the superblock to vote, just so that I get a good idea of what is in the budget.

Thanks!

I agree, voting the last week should be the right approach, as several proposals could be competing for the same resources, and votes should work as a way to prioritize the most critical projects.

In this particular month, I think the only IT proposals are mine but there could be a month where this is not the case.


  • Rob Andrews
  • Administrator

    • 4097


    • 97
    • June 05, 2017, 08:09:04 PM
    • Patmos, Island Of
    more
Re: Iquidus Explorer: upgrade to support GETUTXO in API
« Reply #7 on: April 14, 2018, 10:58:51 AM »
Great job!  Im glad you got the API path to expose the UTXOs, and figured out a way to persist the UTXO value in the database!

Im sure Lich will be happy to resync his chain!

Awesome!



  • MIP
  • Sr. Member

    • 365


    • 47
    • February 13, 2018, 11:55:52 AM
    more
Re: Iquidus Explorer: upgrade to support GETUTXO in API
« Reply #8 on: April 14, 2018, 02:42:52 PM »
Great job!  Im glad you got the API path to expose the UTXOs, and figured out a way to persist the UTXO value in the database!

Im sure Lich will be happy to resync his chain!

Awesome!

It's better to give it a good try first on a test environment, I want to be sure that this add-on is not impacting on the performance (it should not). However it will eat some storage, for example addresses collection takes about 180MB now and processing time for each transaction is slower than before.

I also found some glitches in PoDC staking addresses, as some UTXOs are not properly removed from the database array.
But we are quite close.
« Last Edit: April 14, 2018, 02:46:19 PM by MIP »


  • MIP
  • Sr. Member

    • 365


    • 47
    • February 13, 2018, 11:55:52 AM
    more
Re: Iquidus Explorer: upgrade to support GETUTXO in API
« Reply #9 on: April 17, 2018, 05:45:11 AM »
Ok, I think I finished polishing, development is finished.

I sent the update to Lichtsucher so he can do a final test of the last version on his environment.

As soon as this is published in biblepay-central explorer, I will release the Android mobile app in Google Play.


  • jaapgvk
  • Hero Member

    • 558


    • 31
    • September 01, 2017, 08:02:57 PM
    • Netherlands
    more
Re: Iquidus Explorer: upgrade to support GETUTXO in API
« Reply #10 on: April 17, 2018, 06:00:47 AM »
Ok, I think I finished polishing, development is finished.

I sent the update to Lichtsucher so he can do a final test of the last version on his environment.

As soon as this is published in biblepay-central explorer, I will release the Android mobile app in Google Play.

AWESOME!  8)


  • MIP
  • Sr. Member

    • 365


    • 47
    • February 13, 2018, 11:55:52 AM
    more
Re: Iquidus Explorer: upgrade to support GETUTXO in API
« Reply #11 on: April 25, 2018, 05:02:29 AM »
Tests are taking a while as Licht's test server is still indexing blocks... it's now on 39638. getutxos seem to be working as expected. As soon as it catches up I'll test with one paper wallet and if it works we will publish the android version.


  • MIP
  • Sr. Member

    • 365


    • 47
    • February 13, 2018, 11:55:52 AM
    more
Re: Iquidus Explorer: upgrade to support GETUTXO in API
« Reply #12 on: May 03, 2018, 02:54:46 AM »
It seems that explorer.biblepay-central.org/ is still catching up (now it's on 29th April). In my pitiful PC it took far less time, but it's true that if mongodb can use only 1 cpu at a time, it could be an explanation.

The queries run fast, at the cost of a slower indexing, as forecasted.

From my side, work on this proposal is over.


  • sunk818
  • Global Moderator

    • 521


    • 36
    • April 24, 2018, 02:02:20 PM
    more
Re: Iquidus Explorer: upgrade to support GETUTXO in API
« Reply #13 on: May 11, 2018, 10:22:14 PM »
I don't know how others feel, but I get confused by the amounts because net amounts are not shown. PODC updates especially showing spend very little, but sometimes take big starting amounts and end with big amounts. I think visually the QT does it right by showing the net amount. I think explorer should work the same way. Details should show gross amounts being moved around. I know this isn't exactly what the upgrade is to support, but if it is easy enough to add, I'm in support of it. It would add consistently to UX/UI between QT wallet and Explorer. btw, the green color for receiving and red for sending is a good visual hint that I enjoy.
BH6oxjLkyz3z8FYpvU3ZR7PTZ31Xt9DkXZ


  • MIP
  • Sr. Member

    • 365


    • 47
    • February 13, 2018, 11:55:52 AM
    more
Re: Iquidus Explorer: upgrade to support GETUTXO in API
« Reply #14 on: May 12, 2018, 01:43:58 AM »
I don't know how others feel, but I get confused by the amounts because net amounts are not shown. PODC updates especially showing spend very little, but sometimes take big starting amounts and end with big amounts. I think visually the QT does it right by showing the net amount. I think explorer should work the same way. Details should show gross amounts being moved around. I know this isn't exactly what the upgrade is to support, but if it is easy enough to add, I'm in support of it. It would add consistently to UX/UI between QT wallet and Explorer. btw, the green color for receiving and red for sending is a good visual hint that I enjoy.

QT wallet does it right because it knows that the change address is also yours. But Iquidus doesn't know. So that's basically why can't it make it work the same way.