October has been yet another big month for Notional. With Cosmoverse now a month behind us, the team has had time to sit down and begin carefully addressing the security issue that we alluded to in last month’s report. The issue is complex, and we are not yet prepared to release a full diagnostic, but we are pleased to report that we have commenced organizing mitigation strategies at multiple levels of the Cosmos software stack. We will cover a number of these strategies in this report, along with some routine maintenance that we continue to perform.
For those unfamiliar, CometBFT (the fork of the original Tendermint algorithm) serves as both the peer-to-peer layer for Cosmos blockchains and the consensus mechanism. The peer-to-peer component allows validators to communicate with each other regarding transactions that are being processed and blocks that are being proposed. The consensus component is what allows a collection of validators to agree (or come to consensus) on which blocks should be added to the blockchain. If you’d like a little more detail, you can check out our two-part series on how CometBFT works. This month, we have been particularly focused on upgrades that will boost the security of Cosmos networks. First, in PR 1518, we propose to reduce a parameter called MaxBytes down to 5mb. At the present time, the peer-to-peer software permits blocks up to 21mb, but these are incredibly rare on live networks and can have disruptive effects on validator connections. Second, in PR 1429 we propose pulling a feature from the Celestia network called the CAT mempool. This will add resilience to any blockchain using an up-to-date version of CometBFT.
IBC (or the interblockchain communication protocol) is the protocol that allows Cosmos blockchains to communicate with each other - send assets back and forth without insecure bridges, query each other for data, and so on. In our research, we have discovered some weaknesses of this protocol, and in PR 2983, we propose limitations on the size of the IBC receiver address. This string of characters can be interpreted by various Cosmos modules to trigger a variety of different functionalities, but there are no meaningful uses for extremely large addresses. In this PR, we suggest 2kb as a maximum. We have also created a reference Pull Request, PR 4917, which proposes the creation of a new testnet on which to stress-test a variety of changes to IBC. In particular, it will need a method of mitigating this issue, which was raised by the Notional Security Research Team, a bump to the latest version of the Cosmos SDK, and a few other upgrades.
Cosmos-SDK and Gaia
The issues we uncovered were especially complicated because they emerged from vulnerabilities at several layers of the Cosmos stack. In the Cosmos SDK, we discovered an opportunity to mediate some of these issues and we have proposed them in PR 18185. Here we suggest adjustments to some default parameters related to the storage of bytes. Our research indicates that increasing the default memo size to 1kb and increasing the gas cost of bytes by a factor of ~10 will fortify the SDK against problematic network conditions which can currently manifest when large transactions become more common. On Gaia, we are pushing to upgrade IBC from v4.4.2 to version 4.5.1 in PR 2773.
These changes have been proposed with substantial input from the community as well:
- In this forum post, we discuss removing the global fee module entirely.
- We propose increasing MaxBlockSize from 200k to 2mb here.
- Finally, the discussion regarding the minimum gas price for transactions is being debated in this forum post.
We are immensely grateful to a number of individuals and teams who have contributed to our efforts or supported our efforts to prioritize these issues. In no particular order:
- CryptoCrew validators for their incredible contributions to our testnets attacks and data collection.
- Zaki Manian for his wisdom (and pushing for the elimination of the mempool in this forum post!).
- Hypha Cosmos team for their assistance with the Cosmos Hub testnet.
- Sheldon Dearr for a wealth of wisdom and for helping us organize our reporting.
- Rarma for honest and diligent collaboration.
- The Celestia and Terra engineering teams for giving us access to their testnets.
Our discovery and investigation of the security issues currently contained within the Cosmos Hub (and by extension most other Cosmos blockchains) have left us frustrated. We believe that well-intentioned systems meant to deal with such circumstances have been ported from the Web2 world, but find that they are woefully inadequate to handle the needs and urgency of our decentralized environment. Contending with this frustration has led us to two distinct realizations that are shaping how we proceed in the Cosmos.
First, we are calling for a reexamination of how security issues are handled for the Cosmos Hub. In PR 2767, PR 19, and PR 18 we have begun to lay the groundwork for what more responsive and robust security practices could look like and highlight where we feel the deepest failings have occurred.
Second, we find that the current suite of testnets is an insufficient testing ground for many features and bugs that could be lurking in Cosmos. We are excited to unveil the Danger Zone - a live Cosmos network that is entirely designed to stress-test new Cosmos features in the wild. Our CEO Jacob revealed some details of the Danger Zone in this tweet, but here are a few summary points:
- Validators who wish to validate Danger will receive some tokens for staking, we are hoping for ~100 who can help contribute to security research.
- Incentivization: Attackers who successfully steal Danger tokens will be rewarded.
- Danger tokens are not intended to acquire any monetary value, but may do so regardless.
- Please do not treat Danger as an investment - it has been created for the sheer purpose of security research and being mercilessly rekt.
If you are a validator, we will hope you will join in our mission to make the Cosmos a safer more reliable place to build the decentralized economic systems of the future. As always, you can track our activity on the Cosmos Hub live here.