Include receipt store earliest height in watermark bounds to prevent silent undercount on pruned blocks#3216
Conversation
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3216 +/- ##
==========================================
- Coverage 59.00% 58.96% -0.04%
==========================================
Files 2065 2062 -3
Lines 169422 168986 -436
==========================================
- Hits 99959 99644 -315
+ Misses 60700 60627 -73
+ Partials 8763 8715 -48
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
evmrpc/watermark_manager.go
Outdated
| if earliest <= 0 { | ||
| return nil | ||
| } |
There was a problem hiding this comment.
Trying to wrap my head around this. Is there something special about this? Is it not generalized in the if-statement below it?
There was a problem hiding this comment.
Thanks for feedback. That is a good point as the if statement below will not cause error since height is non-negative. I will remove it.
…height-for-watermark-bounds
masih
left a comment
There was a problem hiding this comment.
-
We also need to update GetBlockByNumber/Hash otherwise the behaviour between GetBlockTransactionCountByNumber/Hash and GetBlockByNumber/Hash would be inconsistent.
-
Left a comment on earliest version in parquet store; looks like it is broken?
Thanks for picking this work up @amir-deris 🍻
|
|
||
| // EarliestVersion returns the earliest version retained in the store. | ||
| func (s *Store) EarliestVersion() int64 { | ||
| return s.earliestVersion.Load() |
There was a problem hiding this comment.
Looks like earliestVersion is never changed in parquet implementation even after pruning?
CC @jewei1997 for context.
There was a problem hiding this comment.
@masih That is a nice observation. The earliest version in parquet store remains zero even after pruning, unlike other store implementations like pebbledb/mvcc/db.go. Should I fix that for Parquet store in another PR?
Problem
There was a fix implemented in PR 3125 in order to make
eth_getBlockTransactionCountByNumber == len(eth_getBlockByNumber.transactions)by using same filtering mechanism for evm transactions to consider availability of receipts. That fix raised the following concern:This change makes GetBlockTransactionCountByNumber and GetBlockTransactionCountByHash results dependent on the receipt store due to the receipt query leading to inconsistencies in historical results.
Those functions do ensure that the block is available based on watermarks. However, the receipt store does not contribute to the earliest height in the watermark manager, only the latest. So even if EnsureBlockHeightAvailable returns that the block is available that does not guarantee all receipts for that height are available. This will occur in the cases when ReceiptStore's KeepRecent is less than the StateStore's KeepRecent and MinRetainBlocks. In such cases the returned transaction count may diverge from the true count due to pruned receipts.
Analysis
eth_getBlockTransactionCountByNumberandeth_getBlockTransactionCountByHashsilently return an incorrect (lower) transaction count for historical blocks when the receipt store has been pruned further back than the block or state stores.The watermark check in
EnsureBlockHeightAvailableusesblockEarliest(derived from Tendermint) to gate requests, but the receipt store's earliest version never contributed to that bound. So a block could pass the availability check while its receipts were already pruned — causingcountBlockTxsLikeEncodeTmBlockto silently skip EVM transactions whose receipts are missing rather than returning an error.This scenario occurs when
ReceiptStoreConfig.KeepRecentis smaller thanStateStoreConfig.KeepRecent/MinRetainBlocks.Fix
EarliestVersion() int64to theReceiptStoreinterface and implemented it across all backends (receiptStore/pebble,parquetReceiptStore,cachedReceiptStore) and the underlyingparquet.Store.WatermarkManager.EnsureReceiptHeightAvailable— a targeted check that returns an explicit "receipts have been pruned" error when the requested height is below the receipt store's earliest retained version. Returnsnilwhen no receipt store is configured or no pruning has occurred.EnsureReceiptHeightAvailablein bothGetBlockTransactionCountByNumberandGetBlockTransactionCountByHashafter block resolution, so callers receive a clear error instead of a silently wrong count.Result
eth_getBlockByNumber)Fixes: PLT-256