A self-hosted API plugin for Bitcoin Core over Cyphernode

Batcher is Bitcoin transaction batching and automation plugin for cyphernode users (a cypherapp) designed for high-volume enterprise users performing multiple Bitcoin transactions per day.

Instead of making one Bitcoin transaction for each Bitcoin payment (e.g. a user withdrawing Bitcoin from an exchange) it allows you to schedule and manage batches of multiple Bitcoins payments sent to the Bitcoin Blockchain as a single transaction.

Batcher was developed by Francis Pouliot and Kexkey from Satoshi Portal Inc., the company behind bullbitcoin.com and bylls.com as part of the Cyphernode project, under MIT License.

Cypherapps can be conceived as cyphernode “plugins”. Running a Cyphernode instance is thus required to use the Batcher cypherapp. Instead of communicating directly with the Cyphernode API, users will connect to the Batcher API. Batcher will then manage how and when Cyphernode will be creating Bitcoin transactions.

The main features are:

• Self-hosted advanced API and batching automation configurations
• Add payments to a batch instead of sending to addresses
• Unique callback URLs for webhook notifications per payment
• Well kept logs, batch details and transactions details
• Aggregates amounts of multiple payments to same address
• Run multiple different batches with different configs simultaneously
• Set batch execution trigger (when all the payments are sent as unconfirmed on the network) based on Amount or Time:
  - Every (x) minutes
  AND/OR
  - Every time the batch hits (y) bitcoins
• Securely connected and controlling a Bitcoin Core wallet and node via the Cyphernode encrypted network.

Batcher is currently implemented in the Bull Bitcoin exchange.

Anybody can run his own self-hosted instance of Batcher: it is free and open-source… no third parties!

The software repo on github is here: https://github.com/SatoshiPortal/batcher

The technical details of the software for developers and product designers are at the end of this document.

Discussion: an on-chain solutions for Bitcoin scaling

The Bitcoin network’s transaction throughput is limited to 3.7 tx/s on average because of the maximum block weight limit. Over the past year, the network can fit ~360,000 transactions in the blockchain on its good days.

This limit keeps Bitcoin decentralized and preserves its core features: scarcity, censorship-resistance, auditability and sovereignty. Because of the growing demand for block space, and the fact that blocks are produced every 10 minutes, users that wish to have their transactions included have two options:

• Pay a higher transaction fee to get priority (faster confirmation)
• Wait a longer amount of time to get your transaction confirmed

There are generally four things we can do to alleviate this problem:

1. Transactions off-chain using second-layer networks (Lightning, Liquid): these technologies are great, but they are still experimental and have their own issues. We believe they are the right long-term approach for Bitcoin scaling, but will take time before they are mature enough for wider network effects to take hold.
2. Transactions off-chain using custodial platforms: his is completely against the point of Bitcoin. This is a lazy, bad solution.
3. Optimize the transactions to make their size smaller: this is what Bitcoin Core is working on, on many levels, Segwit was a big step in the right direction. More technologies will come over the next years.

4. Batch multiple Bitcoin payments are a single transaction: this is the lowest hanging fruit. It is an obvious and intuitive solution (e.g. fit more people in a bus on the highway instead of adding more lanes). However, it requires hard work to build software that allows this for non-custodial exchanges. Most exchange operators are custodial, and are not faced with the problems of non-custodial exchanges that have strict speed requirements for withdrawals. Others simply don’t care about the Bitcoin network outside of their immediate short-term interests, and they are subsidized by other means.

There was a market niche for a solution that automates all aspects of transaction batching, which is self-hosted and free, connected to Bitcoin Core (via Cyphernode). So we built it!

Before moving on to 2nd-layer networks, we think other exchanges should first optimize as much as possible their transaction batching. Batcher is free, open-source and self-hosted (your own keys, your own node). So just use it!

Why Batching? Do the math!

The concept of enterprise Bitcoin transaction batching is simple: instead of doing a single transaction each time you sent a payment to a recipient, you queue these payments and aggregate them as different outputs (recipients) of a single Bitcoin transaction. There are three main benefits: smaller overall transaction size (block space used), smaller amount of UTXOs created and avoiding the Unconfirmed Ancestor Count problems, explained below.

The math below is just an example of how much could be saved. It is unlikely that there are many services out there doing 2400 transactions to users per day that could reduce it to once per hour via batching. But, they do exist!

Premises:

• Your service does 100 transactions per hour.
• That’s 2400 transactions per day!
• You pay 50 sat/vbyte per transaction.
• Your average transaction size per payment is ~225 vbytes.
• Price of Bitcoin is $10,000.
• Each transaction costs you $1.13

Minimize your change UTXOs

Same math as above. You go from adding 2400 change utxos per day to 24 change utxos per day. This makes the hot wallet management so much easier and less prone to errors.

• Each change UTXO will cost about ~69 vbytes on average to spend.
• This adds up to 163944 vbytes of savings per day!
• And would have cost 8197200 satoshis! (0.081972 BTC per day)

Minimize transaction size

• You normally add 540,000 vbytes to the blockchain every day!
• Paying 27,000,000 sats per day! (0.27 BTC)
• But instead, now you batch them every hour.
• Now you do 24 transactions per day
• Each transaction of 100 outputs is ~4000 bytes
• Now you only add 100,000 vbytes to the blockchain each day!
• Which costs you only 5,000,000 sats per day (0.05 BTC)

Total savings

• Save 440,000 vbytes from transaction size batching
• Save 163,944 vbytes from change utxo
• Save 603,944 vbytes total per day
• Save 30197200 sats per day total
• Save an incredible 110.21978 BTC per years
• Transactions now cost around 0.0021 each ($20) instead of ($1.13)
• But you do 8,760 per year transactions instead of 867,240 per year
• Over 100 Bitcoin saved!

Minimize ancestor count

There is a very niche problem that mostly only high-volume Bitcoin services operators know about.

When you do a transaction, the change output goes back into your hot wallet. When creating the next transaction, your wallet may use that unconfirmed change output as an input. That second transaction will have an unconfirmed ancestor, as well as a new unconfirmed change output. If the third transaction uses the second unconfirmed change output as an input, that thirst transaction will have 2 unconfirmed ancestors.

Once you go over 25 unconfirmed transactions in a chain of transactions, other nodes will consider the transaction as invalid. You will have to wait for the previous transactions to be included in the blockchain.

There are a few methods to mitigate this problem or to avoid it entirely, but it is a problem that will occur if you are using the default Bitcoin Core transaction system without custom coin control.

Reducing the number of change outputs so drastically makes it extremely unlikely to encounter this problem.

Privacy drawbacks

People that are part of a batched transaction will know that the bitcoins sent to the other addresses in the transaction were sent by Bull Bitcoin too, and presumably they are customers of Bull Bitcoin (which they are not). We try to mitigate this by using Coinjoin at multiple steps of our processes. But if you receive a batched withdrawal, it is much more important for you to use Coinjoin. Use wasabi wallet as your receiving wallet, once you have over 0.1BTC mix the bitcoins with Coinjoin and then send them to your cold storage from there. This solves the problem of other Bull Bitcoin users tagging your coins as belonging to another Bull Bitcoin user.

Future research and development

The next step will be to create a system which fully utilizes the RBF functionality of Bitcoin Core and the Bitcoin Protocol. This would allow us to actually broadcast transactions and lowball the fees, adding more and more payments (outputs) to an already broadcast transaction, much like what is done in the OpenTimestamps calendar server. This would allow us to provide the end-users with unconfirmed bitcoin txids, which is much more desirable than just a message saying the paying has been batched. The drawback is that the txid would change every time and the previous ones would be invalid, which could be very confusing for the user.

In addition, we are keeping a close eye on the development of WabiSabi, a technique developed by Wasabi Wallet that will essentially merge the process of batching and coinjoining together. A potentially massive improvement.

Non-custodial?

Is it non-custodial? The short answer is no, the long answer is a little bit more complicated. When you set up a batching schedule, you are still in physical possession of the Bitcoin that a user has purchased from a non-custodial service like Bull Bitcoin. The time between which the Bitcoin is purchased and the Bitcoin is received technically could be considered custodial. But this is really more like a shipping service, which takes a bit of time. In the case of Bull Bitcoin, our batching will be every hour or every time the total withdrawals at that time exceed (x) Bitcoin, whichever comes first. The Canadian Securities Association has provisions which specifically talk about these kinds of delays in non-custodial services, recognizing that batching doesn’t make an otherwise non-custodial exchange custodial. Users can opt-out of the batch at any time, but they will have to pay the Bitcoin network fees. This allows them to obtain faster security of the funds if they want.

Technical documentation

Step 1: Creating a batching schedule

Create a batching schedule via the configuration file. You can opt for :

• Amount-based batch threshold (e.g. every time the batch reaches at least 0.5 Bitcoin)
• Time-based batch schedule (e.g. execute the current batches ever 4 hours).
• We recommend using both. For example, execute the batch every time the amount exceeds 1 Bitcoin or every hour, whichever comes first.

Edit the config here https://github.com/SatoshiPortal/batcher/blob/master/src/config/BatcherConfig.ts

BATCH_TIMEOUT_MINUTES: set this as the maximum frequency. If the threshold amount is not reached it will execute regardless at this frequency.

CHECK_THRESHOLD_MINUTES: frequency of checking the threshold.

BATCH_THRESHOLD_AMOUNT: the target batch threshold. When this amount is reached, the batch will be executed as a Bitcoin transaction. If it is not reached, the batch will be executed at according to the batch timeout setting.

BATCH_CONF_TARGET: when the batch is executed, this setting will determine which network fee level the Bitcoin Core wallet will use for the payments. You can for example have 2 batches, one with batch_conf_target of 6 for express withdrawals and one of batch_conf_target of 100 for non-urgent transactions.

API Workflow

• Add to batch: submit a Bitcoin address and amount of a payment to a batching queue via API.
• For each payment (output + amount) you should add a callback URL that will receive the webhook notification when the transaction is sent (0-conf) and/or confirmed (1-conf). This is useful for notifying users that their withdrawal has been processed. You will receive detailed transaction info.
• You can remove a payment from a batch at any time, for example if the user wants to have an instant withdrawal. We would suggest to then send the Bitcoin using the normal sendtoaddressAPI call. To make the end-user pay for the transaction fee instead of you (for example as a premium for opting out of transaction) you can subtract the fee from the amount.
• Specify which batching schedule you want that payment to be queued in when submitting Bitcoin payments to the Batcher API. You may want to have different batching schedules, some more frequent than others, and some with lower confirmation targets (lower fees) than others.

Adding a Bitcoin payment to a batch via API

curl -d '{"id":1,"method":"queueForNextBatch","params":{"address":"bcrt1q0jrfsg98jakmuz0xc0mmxp2ewmqpz0tuh273fy","amount":0.0001,"webhookUrl":"http://webhookserver:1111/indiv"}}' -H "Content-Type: application/json"  -k -u "<username>:<dd xxd output>" https://localhost/batcher/api | jq

API response after adding a payment to a batch

{
  result?: {
    batchRequestId: number;
    batchId: number;
    etaSeconds: number;
    cnResult: {
      batcherId?: number;
      batcherLabel?: string;
      outputId?: number;
      nbOutputs?: number;
      oldest?: Date;
      total?: number;
    }
  }
  error?: {
    code: number;
    message: string;
    data?: D;
  }
}

• In the API response above, you get the time of the next batch etaSeconds. You can notify the user that the batch will be executed at the latest at that time (and possibly earlier).
• If multiple payments are being made to a single Bitcoin address, batcher will aggregate the amounts and make a single payment to that Bitcoin address. This is not optional because Bitcoin Core would otherwise reject the transaction.
• Once the amount or expiry time thresholds have been reached, Batcher will dispatch a request to the Bitcoin Core instance running in cyphernode to create and broadcast the transaction using the “send multi” Bitcoin Core RPC call.
• In the API webhook notification, you will receive the information related to each Bitcoin payment as if it had been its own transaction. The transaction fees will have been adjusted “pro-rata” for each Bitcoin payment. This is meant to keep retro-compatibility with existing Cyphernode users which, when switching to batching, will not have to do anything else than call the new Batcher API instead of the “spend” API in Cyphernode.

Webhook notification sent to all the callback URLs submitted with payments to a batch

{
  "error": null,
  "result": {
    "batchRequestId": 48,
    "batchId": 8,
    "cnBatcherId": 1,
    "txid": "fc02518e32c22574158b96a513be92739ecb02d0caa463bb273e28d2efead8be",
    "hash": "fc02518e32c22574158b96a513be92739ecb02d0caa463bb273e28d2efead8be",
    "spentDetails": {
      "address": "2N8DcqzfkYi8CkYzvNNS5amoq3SbAcQNXKp",
      "amount": 0.0001,
      "firstseen": 1584568841,
      "size": 222,
      "vsize": 141,
      "replaceable": false,
      "fee": 0.00000141,
      "subtractfeefromamount": false
    }
  }
}

• Remove a payment from a batch using the api call below. This is useful for example of one of your users opted for the batch payment option and changes his mind after.

Request:dequeueFromNextBatch{
    batchRequestId: number;
}

• The information you get on a batch is

getBatchDetailsRequest:{
  batchRequestId?: number;
  batchId?: number;
}

Info on batch as API response

{
  result?: {
    batchId: number;
    cnBatcherId: number;
    txid?: string;
    spentDetails?: string;
    spentTimestamp?: Date;
    createdAt?: Date;
    updatedAt?: Date;
    batchRequests: [
      {
        batchRequestId: number;
        externalId?: number;
        description?: string;
        address: string;
        amount: number;
        cnBatcherId?: number;
        cnBatcherLabel?: string;
        webhookUrl?: string;
        calledback?: boolean;
        calledbackTimestamp?: Date;
        cnOutputId?: number;
        mergedOutput?: boolean;
        createdAt?: Date;
        updatedAt?: Date;
      },
      ...
    ]
  }
  error?: {
    code: number;
    message: string;
    data?: D;
  }
}

Conclusion

• You can use Batcher to build a complete hot wallet solution for your Bitcoin service. It is a self-hosted API pluging with a an automated transaction batching scheduling and reporting system.
• Batching is an effective way to scale your Bitcoin wallet operations without paying too much or having issues.
• You need to be using a cyphernode back-end.