Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: https://cockroach-labs.sentry.io/issues/6678613034/?referrer=webhooks_plugin
Panic Message:
catch.go:24: opening object for writing: failed to delete from the payload table while preparing for overwrite: delete-payload-table: runtime error: index out of range [6] with length 6
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cloud.WriteFile
| pkg/cloud/cloud_io.go:429
| github.com/cockroachdb/cockroach/pkg/backup/backupinfo.WriteBackupLock
| pkg/backup/backupinfo/manifest_handling.go:558
| github.com/cockroachdb/cockroach/pkg/backup.maybeWriteBackupLock
| pkg/backup/backup_job.go:2059
| github.com/cockroachdb/cockroach/pkg/backup.(*backupResumer).Resume
| pkg/backup/backup_job.go:594
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
| pkg/jobs/registry.go:1720
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
| pkg/jobs/registry.go:1721
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
| pkg/jobs/adopt.go:410
| github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2
| pkg/jobs/jobs.go:799
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:500
Wraps: (2) opening object for writing
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cloud/userfile/filetable.(*FileToTableSystem).deleteFileWithoutTxn
| pkg/cloud/userfile/filetable/file_table_read_writer.go:426
| github.com/cockroachdb/cockroach/pkg/cloud/userfile/filetable.(*FileToTableSystem).NewFileWriter
| pkg/cloud/userfile/filetable/file_table_read_writer.go:947
| github.com/cockroachdb/cockroach/pkg/cloud/userfile.(*fileTableStorage).Writer
| pkg/cloud/userfile/file_table_storage.go:240
| github.com/cockroachdb/cockroach/pkg/cloud.(*esWrapper).Writer
| pkg/cloud/impl_registry.go:428
| github.com/cockroachdb/cockroach/pkg/cloud.WriteFile
| pkg/cloud/cloud_io.go:427
| github.com/cockroachdb/cockroach/pkg/backup/backupinfo.WriteBackupLock
| pkg/backup/backupinfo/manifest_handling.go:558
| github.com/cockroachdb/cockroach/pkg/backup.maybeWriteBackupLock
| pkg/backup/backup_job.go:2059
| github.com/cockroachdb/cockroach/pkg/backup.(*backupResumer).Resume
| pkg/backup/backup_job.go:594
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
| pkg/jobs/registry.go:1720
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
| pkg/jobs/registry.go:1721
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
| pkg/jobs/adopt.go:410
| github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2
| pkg/jobs/jobs.go:799
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:500
Wraps: (4) failed to delete from the payload table while preparing for overwrite
Wraps: (5) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
| pkg/sql/internal.go:1260
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Close
| pkg/sql/internal.go:608
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| pkg/sql/internal.go:584
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execIEStmt
| pkg/sql/internal.go:872
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).ExecEx
| pkg/sql/internal.go:820
| github.com/cockroachdb/cockroach/pkg/cloud/userfile/filetable.(*FileToTableSystem).deleteFileWithoutTxn
| pkg/cloud/userfile/filetable/file_table_read_writer.go:423
| github.com/cockroachdb/cockroach/pkg/cloud/userfile/filetable.(*FileToTableSystem).NewFileWriter
| pkg/cloud/userfile/filetable/file_table_read_writer.go:947
| github.com/cockroachdb/cockroach/pkg/cloud/userfile.(*fileTableStorage).Writer
| pkg/cloud/userfile/file_table_storage.go:240
| github.com/cockroachdb/cockroach/pkg/cloud.(*esWrapper).Writer
| pkg/cloud/impl_registry.go:428
| github.com/cockroachdb/cockroach/pkg/cloud.WriteFile
| pkg/cloud/cloud_io.go:427
| github.com/cockroachdb/cockroach/pkg/backup/backupinfo.WriteBackupLock
| pkg/backup/backupinfo/manifest_handling.go:558
| github.com/cockroachdb/cockroach/pkg/backup.maybeWriteBackupLock
| pkg/backup/backup_job.go:2059
| github.com/cockroachdb/cockroach/pkg/backup.(*backupResumer).Resume
| pkg/backup/backup_job.go:594
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
| pkg/jobs/registry.go:1720
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
| pkg/jobs/registry.go:1721
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
| pkg/jobs/adopt.go:410
| github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2
| pkg/jobs/jobs.go:799
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:500
Wraps: (6) delete-payload-table
Wraps: (7) assertion failure
Wraps: (8) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/util/errorutil.ShouldCatch
| pkg/util/errorutil/catch.go:24
| github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder.(*Builder).build.func1
| pkg/sql/opt/exec/execbuilder/builder.go:356
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| runtime.goPanicIndex
| GOROOT/src/runtime/panic.go:115
| github.com/cockroachdb/cockroach/pkg/sql.(*optTable).Statistic
| pkg/sql/opt_catalog.go:1441
| github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder.(*Builder).maybeAnnotateWithEstimates
| pkg/sql/opt/exec/execbuilder/relational.go:418
| github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder.(*Builder).buildRelational
| pkg/sql/opt/exec/execbuilder/relational.go:383
| github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder.(*Builder).buildLookupJoin
| pkg/sql/opt/exec/execbuilder/relational.go:2733
| github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder.(*Builder).buildRelational
| pkg/sql/opt/exec/execbuilder/relational.go:228
| github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder.(*Builder).buildProject
| pkg/sql/opt/exec/execbuilder/relational.go:1152
| github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder.(*Builder).buildRelational
| pkg/sql/opt/exec/execbuilder/relational.go:203
| github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder.(*Builder).buildProject
| pkg/sql/opt/exec/execbuilder/relational.go:1152
| github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder.(*Builder).buildRelational
| pkg/sql/opt/exec/execbuilder/relational.go:203
| github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder.(*Builder).buildMutationInput
| pkg/sql/opt/exec/execbuilder/mutation.go:60
| github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder.(*Builder).buildDelete
| pkg/sql/opt/exec/execbuilder/mutation.go:607
| github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder.(*Builder).buildRelational
| pkg/sql/opt/exec/execbuilder/relational.go:267
| github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder.(*Builder).build
| pkg/sql/opt/exec/execbuilder/builder.go:380
| github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder.(*Builder).Build
| pkg/sql/opt/exec/execbuilder/builder.go:298
| github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).runExecBuilder
| pkg/sql/plan_opt.go:935
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).runExecBuild
| pkg/sql/plan_opt.go:344
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan
| pkg/sql/plan_opt.go:267
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan
| pkg/sql/conn_executor_exec.go:3320
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
| pkg/sql/conn_executor_exec.go:2867
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| pkg/sql/conn_executor_exec.go:1085
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func2
| ...
Stacktrace (expand for inline code snippets):
src/runtime/asm_arm64.s#L1222-L1224
pkg/util/stop/stopper.go#L499-L501
pkg/jobs/jobs.go#L798-L800
pkg/jobs/adopt.go#L409-L411
pkg/jobs/registry.go#L1720-L1722
pkg/jobs/registry.go#L1719-L1721
pkg/backup/backup_job.go#L593-L595
pkg/backup/backup_job.go#L2058-L2060
pkg/backup/backupinfo/manifest_handling.go#L557-L559
pkg/cloud/cloud_io.go#L428-L430
src/runtime/asm_arm64.s#L1222-L1224
pkg/util/stop/stopper.go#L499-L501
pkg/jobs/jobs.go#L798-L800
pkg/jobs/adopt.go#L409-L411
pkg/jobs/registry.go#L1720-L1722
pkg/jobs/registry.go#L1719-L1721
pkg/backup/backup_job.go#L593-L595
pkg/backup/backup_job.go#L2058-L2060
pkg/backup/backupinfo/manifest_handling.go#L557-L559
pkg/cloud/cloud_io.go#L426-L428
pkg/cloud/impl_registry.go#L427-L429
pkg/cloud/userfile/file_table_storage.go#L239-L241
pkg/cloud/userfile/filetable/file_table_read_writer.go#L946-L948
pkg/cloud/userfile/filetable/file_table_read_writer.go#L425-L427
src/runtime/asm_arm64.s#L1222-L1224
pkg/util/stop/stopper.go#L499-L501
pkg/jobs/jobs.go#L798-L800
pkg/jobs/adopt.go#L409-L411
pkg/jobs/registry.go#L1720-L1722
pkg/jobs/registry.go#L1719-L1721
pkg/backup/backup_job.go#L593-L595
pkg/backup/backup_job.go#L2058-L2060
pkg/backup/backupinfo/manifest_handling.go#L557-L559
pkg/cloud/cloud_io.go#L426-L428
pkg/cloud/impl_registry.go#L427-L429
pkg/cloud/userfile/file_table_storage.go#L239-L241
pkg/cloud/userfile/filetable/file_table_read_writer.go#L946-L948
pkg/cloud/userfile/filetable/file_table_read_writer.go#L422-L424
pkg/sql/internal.go#L819-L821
pkg/sql/internal.go#L871-L873
pkg/sql/internal.go#L583-L585
pkg/sql/internal.go#L607-L609
pkg/sql/internal.go#L1259-L1261
pkg/sql/internal.go#L252-L254
pkg/sql/conn_executor.go#L2365-L2367
pkg/sql/conn_executor.go#L2557-L2559
pkg/sql/conn_executor.go#L2555-L2557
pkg/sql/conn_executor_exec.go#L319-L321
pkg/sql/conn_executor_exec.go#L168-L170
pkg/sql/conn_executor_exec.go#L4475-L4477
pkg/sql/conn_executor_exec.go#L169-L171
pkg/sql/conn_executor_exec.go#L1084-L1086
pkg/sql/conn_executor_exec.go#L2866-L2868
pkg/sql/conn_executor_exec.go#L3319-L3321
pkg/sql/plan_opt.go#L266-L268
pkg/sql/plan_opt.go#L343-L345
pkg/sql/plan_opt.go#L934-L936
pkg/sql/opt/exec/execbuilder/builder.go#L297-L299
pkg/sql/opt/exec/execbuilder/builder.go#L379-L381
pkg/sql/opt/exec/execbuilder/relational.go#L266-L268
pkg/sql/opt/exec/execbuilder/mutation.go#L606-L608
pkg/sql/opt/exec/execbuilder/mutation.go#L59-L61
pkg/sql/opt/exec/execbuilder/relational.go#L202-L204
pkg/sql/opt/exec/execbuilder/relational.go#L1151-L1153
pkg/sql/opt/exec/execbuilder/relational.go#L202-L204
pkg/sql/opt/exec/execbuilder/relational.go#L1151-L1153
pkg/sql/opt/exec/execbuilder/relational.go#L227-L229
pkg/sql/opt/exec/execbuilder/relational.go#L2732-L2734
pkg/sql/opt/exec/execbuilder/relational.go#L382-L384
pkg/sql/opt/exec/execbuilder/relational.go#L417-L419
pkg/sql/opt_catalog.go#L1440-L1442
GOROOT/src/runtime/panic.go#L114-L116
GOROOT/src/runtime/panic.go#L790-L792
pkg/sql/opt/exec/execbuilder/builder.go#L355-L357
pkg/util/errorutil/catch.go#L23-L25
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 500
pkg/jobs/jobs.go in pkg/jobs.(*StartableJob).Start.func2 at line 799
pkg/jobs/adopt.go in pkg/jobs.(*Registry).runJob at line 410
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine at line 1721
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine.func2 at line 1720
pkg/backup/backup_job.go in pkg/backup.(*backupResumer).Resume at line 594
pkg/backup/backup_job.go in pkg/backup.maybeWriteBackupLock at line 2059
pkg/backup/backupinfo/manifest_handling.go in pkg/backup/backupinfo.WriteBackupLock at line 558
pkg/cloud/cloud_io.go in pkg/cloud.WriteFile at line 429
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 500
pkg/jobs/jobs.go in pkg/jobs.(*StartableJob).Start.func2 at line 799
pkg/jobs/adopt.go in pkg/jobs.(*Registry).runJob at line 410
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine at line 1721
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine.func2 at line 1720
pkg/backup/backup_job.go in pkg/backup.(*backupResumer).Resume at line 594
pkg/backup/backup_job.go in pkg/backup.maybeWriteBackupLock at line 2059
pkg/backup/backupinfo/manifest_handling.go in pkg/backup/backupinfo.WriteBackupLock at line 558
pkg/cloud/cloud_io.go in pkg/cloud.WriteFile at line 427
pkg/cloud/impl_registry.go in pkg/cloud.(*esWrapper).Writer at line 428
pkg/cloud/userfile/file_table_storage.go in pkg/cloud/userfile.(*fileTableStorage).Writer at line 240
pkg/cloud/userfile/filetable/file_table_read_writer.go in pkg/cloud/userfile/filetable.(*FileToTableSystem).NewFileWriter at line 947
pkg/cloud/userfile/filetable/file_table_read_writer.go in pkg/cloud/userfile/filetable.(*FileToTableSystem).deleteFileWithoutTxn at line 426
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 500
pkg/jobs/jobs.go in pkg/jobs.(*StartableJob).Start.func2 at line 799
pkg/jobs/adopt.go in pkg/jobs.(*Registry).runJob at line 410
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine at line 1721
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine.func2 at line 1720
pkg/backup/backup_job.go in pkg/backup.(*backupResumer).Resume at line 594
pkg/backup/backup_job.go in pkg/backup.maybeWriteBackupLock at line 2059
pkg/backup/backupinfo/manifest_handling.go in pkg/backup/backupinfo.WriteBackupLock at line 558
pkg/cloud/cloud_io.go in pkg/cloud.WriteFile at line 427
pkg/cloud/impl_registry.go in pkg/cloud.(*esWrapper).Writer at line 428
pkg/cloud/userfile/file_table_storage.go in pkg/cloud/userfile.(*fileTableStorage).Writer at line 240
pkg/cloud/userfile/filetable/file_table_read_writer.go in pkg/cloud/userfile/filetable.(*FileToTableSystem).NewFileWriter at line 947
pkg/cloud/userfile/filetable/file_table_read_writer.go in pkg/cloud/userfile/filetable.(*FileToTableSystem).deleteFileWithoutTxn at line 423
pkg/sql/internal.go in pkg/sql.(*InternalExecutor).ExecEx at line 820
pkg/sql/internal.go in pkg/sql.(*InternalExecutor).execIEStmt at line 872
pkg/sql/internal.go in pkg/sql.(*rowsIterator).Next at line 584
pkg/sql/internal.go in pkg/sql.(*rowsIterator).Close at line 608
pkg/sql/internal.go in pkg/sql.(*InternalExecutor).execInternal.func1.1 at line 1260
pkg/sql/internal.go in pkg/sql.(*InternalExecutor).runWithEx.func2 at line 253
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 2366
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2558
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func2 at line 2556
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execPortal at line 320
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 169
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 4476
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func2 at line 170
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 1085
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 2867
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).makeExecPlan at line 3320
pkg/sql/plan_opt.go in pkg/sql.(*planner).makeOptimizerPlan at line 267
pkg/sql/plan_opt.go in pkg/sql.(*planner).runExecBuild at line 344
pkg/sql/plan_opt.go in pkg/sql.(*optPlanningCtx).runExecBuilder at line 935
pkg/sql/opt/exec/execbuilder/builder.go in pkg/sql/opt/exec/execbuilder.(*Builder).Build at line 298
pkg/sql/opt/exec/execbuilder/builder.go in pkg/sql/opt/exec/execbuilder.(*Builder).build at line 380
pkg/sql/opt/exec/execbuilder/relational.go in pkg/sql/opt/exec/execbuilder.(*Builder).buildRelational at line 267
pkg/sql/opt/exec/execbuilder/mutation.go in pkg/sql/opt/exec/execbuilder.(*Builder).buildDelete at line 607
pkg/sql/opt/exec/execbuilder/mutation.go in pkg/sql/opt/exec/execbuilder.(*Builder).buildMutationInput at line 60
pkg/sql/opt/exec/execbuilder/relational.go in pkg/sql/opt/exec/execbuilder.(*Builder).buildRelational at line 203
pkg/sql/opt/exec/execbuilder/relational.go in pkg/sql/opt/exec/execbuilder.(*Builder).buildProject at line 1152
pkg/sql/opt/exec/execbuilder/relational.go in pkg/sql/opt/exec/execbuilder.(*Builder).buildRelational at line 203
pkg/sql/opt/exec/execbuilder/relational.go in pkg/sql/opt/exec/execbuilder.(*Builder).buildProject at line 1152
pkg/sql/opt/exec/execbuilder/relational.go in pkg/sql/opt/exec/execbuilder.(*Builder).buildRelational at line 228
pkg/sql/opt/exec/execbuilder/relational.go in pkg/sql/opt/exec/execbuilder.(*Builder).buildLookupJoin at line 2733
pkg/sql/opt/exec/execbuilder/relational.go in pkg/sql/opt/exec/execbuilder.(*Builder).buildRelational at line 383
pkg/sql/opt/exec/execbuilder/relational.go in pkg/sql/opt/exec/execbuilder.(*Builder).maybeAnnotateWithEstimates at line 418
pkg/sql/opt_catalog.go in pkg/sql.(*optTable).Statistic at line 1441
GOROOT/src/runtime/panic.go in runtime.goPanicIndex at line 115
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
pkg/sql/opt/exec/execbuilder/builder.go in pkg/sql/opt/exec/execbuilder.(*Builder).build.func1 at line 356
pkg/util/errorutil/catch.go in pkg/util/errorutil.ShouldCatch at line 24
Tags
Tag | Value |
---|---|
Command | start-single-node |
Environment | v25.2.1 |
Go Version | go1.23.7 X:nocoverageredesign |
Platform | linux arm64 |
Distribution | CCL |
Cockroach Release | v25.2.1 |
Cockroach SHA | f9f183d |
# of CPUs | 6 |
# of Goroutines | 424 |
Jira issue: CRDB-51540
Metadata
Metadata
Assignees
Labels
Type
Projects
Status