Notional Cosmos Hub Report - October 2023

October has been yet another big month for Notional. With Cosmoverse now a month behind us, the team has had time to sit down and begin carefully addressing the security issue that we alluded to in last month’s report. The issue is complex, and we are not yet prepared to release a full diagnostic, but we are pleased to report that we have commenced organizing mitigation strategies at multiple levels of the Cosmos software stack. We will cover a number of these strategies in this report, along with some routine maintenance that we continue to perform.

For those unfamiliar, CometBFT (the fork of the original Tendermint algorithm) serves as both the peer-to-peer layer for Cosmos blockchains and the consensus mechanism. The peer-to-peer component allows validators to communicate with each other regarding transactions that are being processed and blocks that are being proposed. The consensus component is what allows a collection of validators to agree (or come to consensus) on which blocks should be added to the blockchain. If you’d like a little more detail, you can check out our two-part series on how CometBFT works. This month, we have been particularly focused on upgrades that will boost the security of Cosmos networks. First, in PR 1518, we propose to reduce a parameter called MaxBytes down to 5mb. At the present time, the peer-to-peer software permits blocks up to 21mb, but these are incredibly rare on live networks and can have disruptive effects on validator connections. Second, in PR 1429 we propose pulling a feature from the Celestia network called the CAT mempool. This will add resilience to any blockchain using an up-to-date version of CometBFT.

IBC (or the interblockchain communication protocol) is the protocol that allows Cosmos blockchains to communicate with each other - send assets back and forth without insecure bridges, query each other for data, and so on. In our research, we have discovered some weaknesses of this protocol, and in PR 2983, we propose limitations on the size of the IBC receiver address. This string of characters can be interpreted by various Cosmos modules to trigger a variety of different functionalities, but there are no meaningful uses for extremely large addresses. In this PR, we suggest 2kb as a maximum. We have also created a reference Pull Request, PR 4917, which proposes the creation of a new testnet on which to stress-test a variety of changes to IBC. In particular, it will need a method of mitigating this issue, which was raised by the Notional Security Research Team, a bump to the latest version of the Cosmos SDK, and a few other upgrades.

The issues we uncovered were especially complicated because they emerged from vulnerabilities at several layers of the Cosmos stack. In the Cosmos SDK, we discovered an opportunity to mediate some of these issues and we have proposed them in PR 18185. Here we suggest adjustments to some default parameters related to the storage of bytes. Our research indicates that increasing the default memo size to 1kb and increasing the gas cost of bytes by a factor of ~10 will fortify the SDK against problematic network conditions which can currently manifest when large transactions become more common. On Gaia, we are pushing to upgrade IBC from v4.4.2 to version 4.5.1 in PR 2773.

These changes have been proposed with substantial input from the community as well:

We are immensely grateful to a number of individuals and teams who have contributed to our efforts or supported our efforts to prioritize these issues. In no particular order:

Our discovery and investigation of the security issues currently contained within the Cosmos Hub (and by extension most other Cosmos blockchains) have left us frustrated. We believe that well-intentioned systems meant to deal with such circumstances have been ported from the Web2 world, but find that they are woefully inadequate to handle the needs and urgency of our decentralized environment. Contending with this frustration has led us to two distinct realizations that are shaping how we proceed in the Cosmos.

First, we are calling for a reexamination of how security issues are handled for the Cosmos Hub. In PR 2767, PR 19, and PR 18 we have begun to lay the groundwork for what more responsive and robust security practices could look like and highlight where we feel the deepest failings have occurred.

Second, we find that the current suite of testnets is an insufficient testing ground for many features and bugs that could be lurking in Cosmos. We are excited to unveil the Danger Zone - a live Cosmos network that is entirely designed to stress-test new Cosmos features in the wild. Our CEO Jacob revealed some details of the Danger Zone in this tweet, but here are a few summary points:

If you are a validator, we will hope you will join in our mission to make the Cosmos a safer more reliable place to build the decentralized economic systems of the future. As always, you can track our activity on the Cosmos Hub live here.