Description
Problem
On April 23, nodes running against master were observed to have diverged from mainnet. Nodes with various commits off of master forked off at slot 190061698, and this hit this panic.
thread 'solReplayStage' panicked at 'We have tried to repair duplicate slot: 190061698 more than 10 times
and are unable to freeze a block with bankhash J5GobCrZZLvvj8EW72BV1Q7973TefL7f3qtqGinebCWF,
instead we have a block with bankhash Some(FqJHBuSpEXHGM5Rt5pk1hWuKQh2Wg3iyQnyiQ4kwT8NW).
This is most likely a bug in the runtime. At this point manual intervention is needed to make progress. Exiting',
The incorrect bank hash and associated fields that make up the bank hash look like:
bank frozen: 190061698
hash: FqJHBuSpEXHGM5Rt5pk1hWuKQh2Wg3iyQnyiQ4kwT8NW
accounts_delta: nSAB5dSBaN94HF7UNT4ZwKna4rYfZKMUPLoA9ZRAHTE
signature_count: 1640
last_blockhash: 4Nez6awxz3mqyEPPtvmBErhaUVFgCd1Us1GVsr9kFEU4
capitalization: 545940604106710686,
stats: BankHashStats {
num_updated_accounts: 4976,
num_removed_accounts: 21,
num_lamports_stored: 2524779513915279,
total_data_len: 20916547,
num_executable_accounts: 0
}
whereas the correct hash produced by v1.13/v1.14 looks like:
bank frozen: 190061698
hash: J5GobCrZZLvvj8EW72BV1Q7973TefL7f3qtqGinebCWF
accounts_delta: qnhRzBRp4Uapif5QewsCkLaEVL2zgLAiBvMacbchn45
signature_count: 1640
last_blockhash: 4Nez6awxz3mqyEPPtvmBErhaUVFgCd1Us1GVsr9kFEU4
capitalization: 545940604106710686
accounts hash slot: 190061698
stats: BankHashStats {
num_updated_accounts: 4984,
num_removed_accounts: 22,
num_lamports_stored: 2524854860183319,
total_data_len: 23534954,
num_executable_accounts: 1
}
At first glance, there is an obvious difference in some of these numbers which would point to a transaction had a different execution result between versions.
Adding some extra logging, I determined that the transaction (explorer link here) that executed differently between versions is:
6ujGsjkwWJtE3VsraD97PBrhWpJaKGVfbxQAyV5thV32NTR9HWa6PjJgqGComn8y9oXBJMc1iT1NwZRLWLm4aM8
Looking at the basic logs in explorer, the above transaction fully succeeded in v1.13 / v1.14. However, looking at the debug logs from executing with master via solana-ledger-tool, I see the following for this tx:
[...] Program ComputeBudget111111111111111111111111111111 invoke [1]
[...] Program ComputeBudget111111111111111111111111111111 success
[...] Program SMPLecH534NA9acpos4G6x7uf3LWbCAwZQE9e8ZekMu invoke [1]
[...] Program log: Instruction: ExecuteTransaction
[...] Program hvsrNC3NKbcryqDs2DocYHZ9yPKEVzdSjQG6RVtK1s8 invoke [2]
[...] Program has been closed
[...] Program hvsrNC3NKbcryqDs2DocYHZ9yPKEVzdSjQG6RVtK1s8 failed: invalid account data for instruction
[...] Program SMPLecH534NA9acpos4G6x7uf3LWbCAwZQE9e8ZekMu consumed 17573 of 1400000 compute units
[...] Program SMPLecH534NA9acpos4G6x7uf3LWbCAwZQE9e8ZekMu failed: invalid account data for instruction
...
[...] tx error: InstructionError(1, InvalidAccountData) SanitizedTransaction { ... }