Description
When attempting to synchronize a new Hydration node from scratch using the default (or explicitly set) --sync warp mode on the parachain, the process consistently downloads ~11.9M headers and the ~473MB state snapshot, but abruptly aborts with Calculated state root does not match right at completion.
Crucially, after analyzing the deep trace logs with -l sync=trace,state-db=trace,trie=trace, there are absolutely ZERO trie errors (no Incomplete database, Missing trie node warnings). This suggests that the 473MB state chunk is structurally complete and forms a valid localized Trie, but the final computed root_hash severely diverges from the expected state_root in the header (potentially due to a StateVersion encoding mismatch from the peer generating the snapshot, or a widespread polluted snapshot in the P2P network).
Furthermore, the fallback mechanism catastrophically deadlocks the node. Substrate throws away the state and attempts to fall back to block-by-block full execution (Falling back to full sync), but since the database completely lacks the genesis state, it fails on block 1 with Current state of blockchain has invalid authorities set, leaving the node permanently idling (best: 11.9M, finalized: 0) with 0 bps.
Expected Behavior
The node should either successfully calculate the identical 32-byte state root, or the network's warp state generation/distribution should systematically reject incompatible snapshots. Failing that, the node should not permanently "zombify" when falling back to a full sync from an empty state.
Actual Behavior
Possible Fix
Steps to Reproduce
1.Use a highly provisioned ubuntu machine and the official v45.0.0 release.
2.Completely purge the data directory: rm -rf ./data/*
3.Run the node via docker-compose with standard arguments, e.g.:
--name "Hydra-Docker"
--chain hydradx
--base-path /hydra/.local/share
--database paritydb
--sync warp
--state-pruning 256
--blocks-pruning 256
--in-peers 50
--out-peers 50
--no-telemetry
--
--chain polkadot
--sync warp
--no-telemetry
4.Wait for the state chunk to download (Importing state, 84%, 473.74 Mib...).
5.Observe the immediate failure and node deadlock.
[Parachain] Error importing state: State root mismatch.
[Parachain] Block import error: Calculated state root does not match.
[Parachain] Failed to import target block with state: Other(ClientImport("Calculated state root does not match.")).
[Parachain] State sync failed. Falling back to full sync.
[Parachain] ⚙️ Syncing 0.0 bps, target=#11935840 (29 peers), best: #0 (0xafdc…366d), finalized #0 (0xafdc…366d), ⬇ 792.9kiB/s ⬆ 3.8kiB/s
[Parachain] 💔 Verification failed for block 0x6386666a4d9059f2b766282409c29291398e87cabc01edc3463333f22238ee2f received from (12D3KooWQWkKkLH8KDdBunY4ttySKnjsVdGkYNruoDVcGRzhSsHW): "Could not fetch authorities at 0x4764be7684ad55ab619a1e6c83ea1b595681ee3d3393f37930464c98cbcddaa7: Current state of blockchain has invalid authorities set"
[Parachain] ⚙️ Syncing 0.2 bps, target=#11935841 (28 peers), best: #1 (0x4764…daa7), finalized #0 (0xafdc…366d), ⬇ 703.8kiB/s ⬆ 1.7kiB/s
Context
Your Environment
Operating System: Linux Ubuntu 22.04.5 LTS
CPU Architecture: x86_64 (8 cores, Intel Xeon)
Memory: 62GB RAM
Node Version: v45.0.0 (also tested with similar behavior on related traces)
Database: ParityDB (default)
Description
When attempting to synchronize a new Hydration node from scratch using the default (or explicitly set) --sync warp mode on the parachain, the process consistently downloads ~11.9M headers and the ~473MB state snapshot, but abruptly aborts with Calculated state root does not match right at completion.
Crucially, after analyzing the deep trace logs with -l sync=trace,state-db=trace,trie=trace, there are absolutely ZERO trie errors (no Incomplete database, Missing trie node warnings). This suggests that the 473MB state chunk is structurally complete and forms a valid localized Trie, but the final computed root_hash severely diverges from the expected state_root in the header (potentially due to a StateVersion encoding mismatch from the peer generating the snapshot, or a widespread polluted snapshot in the P2P network).
Furthermore, the fallback mechanism catastrophically deadlocks the node. Substrate throws away the state and attempts to fall back to block-by-block full execution (Falling back to full sync), but since the database completely lacks the genesis state, it fails on block 1 with Current state of blockchain has invalid authorities set, leaving the node permanently idling (best: 11.9M, finalized: 0) with 0 bps.
Expected Behavior
The node should either successfully calculate the identical 32-byte state root, or the network's warp state generation/distribution should systematically reject incompatible snapshots. Failing that, the node should not permanently "zombify" when falling back to a full sync from an empty state.
Actual Behavior
Possible Fix
Steps to Reproduce
1.Use a highly provisioned ubuntu machine and the official v45.0.0 release.
2.Completely purge the data directory: rm -rf ./data/*
3.Run the node via docker-compose with standard arguments, e.g.:
--name "Hydra-Docker"
--chain hydradx
--base-path /hydra/.local/share
--database paritydb
--sync warp
--state-pruning 256
--blocks-pruning 256
--in-peers 50
--out-peers 50
--no-telemetry
--
--chain polkadot
--sync warp
--no-telemetry
4.Wait for the state chunk to download (Importing state, 84%, 473.74 Mib...).
5.Observe the immediate failure and node deadlock.
[Parachain] Error importing state: State root mismatch.
[Parachain] Block import error: Calculated state root does not match.
[Parachain] Failed to import target block with state: Other(ClientImport("Calculated state root does not match.")).
[Parachain] State sync failed. Falling back to full sync.
[Parachain] ⚙️ Syncing 0.0 bps, target=#11935840 (29 peers), best: #0 (0xafdc…366d), finalized #0 (0xafdc…366d), ⬇ 792.9kiB/s ⬆ 3.8kiB/s
[Parachain] 💔 Verification failed for block 0x6386666a4d9059f2b766282409c29291398e87cabc01edc3463333f22238ee2f received from (12D3KooWQWkKkLH8KDdBunY4ttySKnjsVdGkYNruoDVcGRzhSsHW): "Could not fetch authorities at 0x4764be7684ad55ab619a1e6c83ea1b595681ee3d3393f37930464c98cbcddaa7: Current state of blockchain has invalid authorities set"
[Parachain] ⚙️ Syncing 0.2 bps, target=#11935841 (28 peers), best: #1 (0x4764…daa7), finalized #0 (0xafdc…366d), ⬇ 703.8kiB/s ⬆ 1.7kiB/s
Context
Your Environment
Operating System: Linux Ubuntu 22.04.5 LTS
CPU Architecture: x86_64 (8 cores, Intel Xeon)
Memory: 62GB RAM
Node Version: v45.0.0 (also tested with similar behavior on related traces)
Database: ParityDB (default)