This blog post discloses threats to the Ethereum network that existed from merges to Dencon hard forks.
background
Before the merge, different message size limits for RPC communications were set to protect clients from denial of service (DOS) attacks. These restrictions applied to messages received via HTTP endpoints have been taken over to the Engine API, which plays a key role in connecting execution and consensus layer clients during block production. Because the engine API is involved in block production, it could have generated blocks that exceeded the RPC size limits for some clients, but remained within the acceptable range for others.
If an attacker writes a message that exceeds the client's size limit while adhering to gas limit requirements and waits for a block to be generated, it can lead to a situation in which the block is considered valid, other clients reject the HTTP error code and consider it 413: the content to be too large.
Impact
Attackers who can create these messages can force most of the nodes to reject blocks that minorities accept. These blocks are kicked out and the proposer misses the reward.
Initially, I thought it was only possible to create these blocks using either a builder or modified version of the client. Geth has a 128kb limit for transactions. In other words, large transactions like those under discussion do not become a transaction pool for Geth nodes. However, we were able to cause the limit by suggesting blocks to clients with high limits and requesting that the CL validate this proposed larger block.
I proposed a solution to temporarily lower the RPC limit for all clients to a low (5MB). This disables blocking and attackers are very restricted with the chaos that can cause in the network, as most of the nodes refuse to block.
However, on February 7th, it was discovered that it was possible to create blocks that would fall below the 128kb limit and reach the 5MB limit for many transactions that would not exceed 30 million gas.
This is a big problem as we realised that attackers could create and send a large number of highly paid transactions to the network. He overtakes everyone else in Mempool, so every node (even Geth Nodes) contains attack transactions within the block, creating unacceptable blocks for most of the network, with many forks (all considered valid by a few nodes) and chains being repeated again.
Later on February 7, we concluded that everyone raising the RPC limit would be a safer alternative.
Timeline
- 2024-02-06 13:00: Toni (EF), PARI (EF), Justin (BESU) attempts to submit a specific grey ind transaction to the network. Transactions contribute up to 2.7 MB blocks when the snappy is compressed.
- 2024-02-06 13:25: PARI receives an error from his local Geth node, but the transaction is valid.
- 2024-02-06 15:14: Justin managed to block the transaction and submit it through a BESU client.
- 2024-02-06 20:46: Sam (EF) Alert Paris (Special thanks) Mysticryuujin x), Toni and Alex about the struggles of certain sepolianodes.
- 2024-02-06 21:05: Double checking and checking bugs from Geth's team with Marius.
- 2024-02-06 21:10: Gangs gather and debug
- 2024-02-07 23:40: All clients have decided to limit RPC requests to 5MB
- 2024-02-07 6:40: We discovered that there may be a bigger problem and that we can perform the attack on transactions below the size of 128kb.
- 2024-02-07 10:00: All clients have decided to increase RPC request limit.
- 2024-02-07 21:00: The correction was merged on Geth.
- 2024-02-09: Geth released
Geth was the only client affected by this bug, but other clients updated their defaults to be safe for this attack, even if gas limits increased. The client team showed that there are secure RPC restrictions for the next update.
Guess: v1.13.12
nethermind: v1.25.4
BESU: 24.1.2
Eligon: V2.58.0
Reply: v0.1.0-alpha.18
