**Updated on 3rd July 2020, with details of the completed patch.**
On Friday 26th June, a security issue on the Liquid Network was disclosed publicly by blockchain developer James Prestwich. Although this was a known issue, a fix for the problem had been delayed due to external challenges in coordinating the updates on the functionary servers that maintain the network.
The Bitcoin funds on Liquid continue to remain highly secure, but this is not up to our usual standard of trust-minimization and we have been working with the Liquid Federation to deploy a patch very soon.
Longer-term, the ongoing Dynamic Federations update is designed to fully resolve the issue while additionally providing a range of new features that provide further autonomy to the Liquid Federation.
Liquid’s Multisig Security Model
Bitcoins sent to the Liquid Network through peg-ins are secured in an 11-of-15 multisig wallet that is controlled by the Liquid Federation. To protect against network failures, this multisig wallet uses timelocks to enable a set of 2-of-3 emergency backup keys to recover the funds in the event that the network becomes inactive for an extended period.
The emergency keys are held by Blockstream in extreme cold storage distributed around the world.
A Description of the Issue
The current issue is caused by an inconsistency between the timelock parameters used by the functionary HSMs and the functionary servers. Due to this bug, some timelocks are occasionally being refreshed shortly after expiry, instead of before expiry as designed.
Normally the amount recoverable by the emergency backup keys as a result of this issue is small, but due to the recent rapid growth in BTC peg-ins (100BTC in December 2019 to 2,000BTC+ today), one timelock on a large UTXO (870 BTC) expired for 40 minutes.
All funds on the network remained secure — the backup recovery keys have never been accessed at any point during Liquid’s operation — and the timelock was refreshed by the network without any manual intervention.
HSM Updates Delayed
A complete fix for the issue has been pending a firmware update for the HSMs used by the Liquid functionaries. However, coordinating and deploying updates to the HSMs is typically a difficult process, requiring hands-on access to the devices (this is intentional for security).
Earlier in the year, we deprioritized the deployment of a patch to focus on other network improvements. This decision was made due to a complex set of factors, including:
- There was a relatively small amount of bitcoins on Liquid at the time.
- Some timelocks were being refreshed in advance of expiring through network activity.
- Only minor amounts of bitcoin had timelocks expiring for short periods of time.
- There is always an inherent risk of downtime when deploying patches to distributed networks.
- A complete fix was already on the way in the form of DynaFed (see below).
Later, as the Bitcoin peg-ins grew, deploying the patch was made almost impossible by the recent COVID-related lockdowns — operationally, the last few months have been a challenging time for exchanges and the Liquid Network in general.
Transparency is very important to us and we do more toward making our products auditable than most companies in any industry. In this instance, as per standard protocols for security-sensitive applications, we limited publicly disclosing the issue until it was resolved.
Patch on the Way
Some recent developments (e.g. ability to update functionaries individually and update procedure improvements) have made deploying updates to the functionaries easier and a preliminary workaround patch is now possible.
This patch will update the software running on the functionary servers rather than update the HSM firmware, ensuring that the timelocks on the bitcoins held on the network are always refreshed before expiry.
This is a temporary workaround with some limitations, the main limitation being that the bitcoins held on the network will start to become recoverable by the emergency keys 7 days after the network stops, instead of 14 days as designed. This will provide a shorter safety buffer in the event of a network outage.
The code for the patch has been submitted to Liquid’s Technology Board for review. Once approved, the patch will be rolled out on a portion of the functionaries (importantly, the planned patch only requires one or two functionaries to adopt the change for it to work). We’ll post a further update once the patch is live.
The DynaFed Update
Over the last few months, our engineering team has been working on deploying the Dynamic Federations (DynaFed) update in a phased roll-out.
DynaFed introduces a wide range of improvements to the Liquid Network, including a full correction of the timelock issue at the HSM level. Relevant to this issue, DynaFed also enables the federation to:
- Expand the number of functionaries (currently limited to 15)
- Reassign or disable the emergency keys
- Lengthen the timelock parameters to extend the safety buffer
We’ll also be publicly releasing the source code of the Liquid functionary block signer (already available to Liquid Federation members) to facilitate more third-party review.
Improving Our Communications
Finally, we recognize our initial public communication on this matter could have been improved. We’ll be working with the Liquid Federation to improve how important announcements about the network are made and shifting more responsibilities to the Liquid Federation members.
We’re also already in the final stages of launching a public Blockstream Help Center which will include clear, accessible information on how the Liquid Network works and its security model. That should be launching next week.
We would like to encourage responsible disclosure for any issues relating to Liquid, but we absolutely welcome scrutiny on Liquid or any of the other technologies we work on — Liquid (and the industry) will be stronger because of it.
Update on the Patch
3rd July 2020: The patch passed its review by the Liquid Technology Board and has been successfully deployed on four functionaries, providing a high degree of redundancy above the minimum required to workaround the issue.
With the exception of dust UTXOs (uneconomical to refresh), the Liquid Network will now attempt to refresh timelocks starting seven days in advance of their expiry.
Note that the speed at which a timelock refresh is completed will depend on the blockspace market at the time.
A fifth and final functionary should be patched by next week (five functionaries patched provides the maximum redundancy possible).
If you’ve discovered a security issue in the Liquid Network, please get in touch at security@blockstream.com (optionally using our PGP key). We’ll respond as quickly as possible to your report.
Note: This blog was originally posted at https://medium.com/blockstream/patching-the-liquid-timelock-issue-b4b2f5f9a973