Closed
Description
After deploying the beta candidate bc79f9e to adriatic, two nodes crashed with consistency failures:
E170413 05:20:27.443006 35242 storage/replica_command.go:2142 [n4,s4,r2391/2:/System/{""-tsd},@c420180300] replica {2 2 3} is inconsistent: expected checksum e5c8d79134775742
2465630bc063cfa42037630e39aa0f94b97d4caf850fa9bf27000981bbcad2005496bb5f4d518893e73cde21045fb7e480202830d4b3c6f5, got 7dec4c498df48feead465d2c808d0699c1544324c651d9c2a7f3cbf81
bb875e9ecc8e6ff53f200fcb96f4fe68b32a6d54f92b7917278c641bc1e500cb5b16ec6
--- leaseholder
+++ follower
-0.000000000,0 /Local/RangeID/2391/r/RangeFrozenStatus
- ts:<zero>
- value:^R^D^H^@^P^@^X^@ ^@(^@2^F^@^@^@^@^A^@
- raw_key:/Local/RangeID/2391/r/RangeFrozenStatus raw_value:1204080010001800200028003206000000000100
-0.000000000,0 /Local/RangeID/2391/r/RangeStats
- ts:<zero>
- value:^R^D^H^@^P^@^X^@ ^@(^@2|<83>Ӭ<99>^C B<B3><BB><A4>iݴ^T^Q^@^@^@^@^@^@^@^@^Y<8B>LVM<84>^@^@^@!^_C^A^@^@^@^@^@)^S^@^@^@^@^@^@^@1^A8&^@^@^@^@^@9^T^@^@^@^@^@^@^@A^K^?J^@
^@^@^@^@I;/^C^@^@^@^@^@Q^@^@^@^@^@^@^@^@Y^@^@^@^@^@^@^@^@aV^C^@^@^@^@^@^@i^L^@^@^@^@^@^@^@p^@
- raw_key:/Local/RangeID/2391/r/RangeStats raw_value:120408001000180020002800327c83d3ac99030942b3bba469ddb414110000000000000000198b4c564d84000000211f4301000000000029130000000
0000000310138260000000000391400000000000000410b7f4a0000000000493b2f030000000000510000000000000000590000000000000000615603000000000000690c000000000000007000
+0.000000000,0 /Local/RangeID/2391/r/RangeStats
+ ts:<zero>
+ value:^R^D^H^@^P^@^X^@ ^@(^@2|<DD>a<AD><84>^C B<B3><BB><A4>iݴ^T^Q^@^@^@^@^@^@^@^@^Y<8B>LVM<84>^@^@^@!^_C^A^@^@^@^@^@)^S^@^@^@^@^@^@^@1^A8&^@^@^@^@^@9^T^@^@^@^@^@^@^@
A^K^?J^@^@^@^@^@I;/^C^@^@^@^@^@Q^@^@^@^@^@^@^@^@Y^@^@^@^@^@^@^@^@a7^C^@^@^@^@^@^@i^K^@^@^@^@^@^@^@p^@
+ raw_key:/Local/RangeID/2391/r/RangeStats raw_value:120408001000180020002800327cdd61ad84030942b3bba469ddb414110000000000000000198b4c564d84000000211f4301000000000029130000000
0000000310138260000000000391400000000000000410b7f4a0000000000493b2f030000000000510000000000000000590000000000000000613703000000000000690b000000000000007000
F170413 05:20:27.443248 33652 storage/replica_command.go:2158 [n4,s4,r2391/2:/System/{""-tsd},@c420180300] consistency check failed with 1 inconsistent replicas
We stopped writing the "frozen" field in #14779, but a split that straddled the upgrade boundary on some replicas would have it written inconsistently. I'll make a change to keep writing the field as a constant false.