Tags: andypeng2015/wal-g
Tags
Fix Po 8000 stgres 12+/Cloudberry concurrent backups. (wal-g#1813) ### PostgreSQL / Cloudberry PostgreSQL 12+ may run exclusive backup and several non-exclusive backups at the same time. We use non-exclusive backups. At the end of backup we call `pg_stop_backup` and save it in fictional `backup_label` file that we store in on of the `part_xxx.tar` files. When `WALG_DOWNLOAD_CONCURRENCY` != 1, wal-g do the best to extract files as fast as possible. It is likely, that tiny `part_xxx.tar` with our `backup_label` will be fetched before huge main backup `part_000.tar`. IFF main backup has `backup_label` - we will end up with wrong `backup_label` and restore will end with `requested recovery stop point is before consistent recovery point` ### Please provide steps to reproduce (if it's a bug) ``` # imitate failed backup (label='abc', fast=true, exclusive=true) echo "SELECT pg_start_backup('abc', true, true);" | psql wal-g backup-push ${PGDATA} ``` ### Please add config and wal-g stdout/stderr logs for debug purpose <details><summary>logs</summary> ``` 2024-10-15 15:06:44.801294 UTC,,,p4390,th-1770865664,,,,0,,,seg-1,,,,,"LOG","00000","starting PostgreSQL 14.4 (Cloudberry Database 1.6.0 build dev) on aarch64-unknown-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0, 64-bit",,,,,,,0,,"postmaster.c",1370, 2024-10-15 15:06:44.801672 UTC,,,p4390,th-1770865664,,,,0,,,seg-1,,,,,"LOG","00000","listening on IPv4 address ""0.0.0.0"", port 7000",,,,,,,0,,"pqcomm.c",623, 2024-10-15 15:06:44.801683 UTC,,,p4390,th-1770865664,,,,0,,,seg-1,,,,,"LOG","00000","listening on IPv6 address ""::"", port 7000",,,,,,,0,,"pqcomm.c",623, 2024-10-15 15:06:44.804695 UTC,,,p4390,th-1770865664,,,,0,,,seg-1,,,,,"LOG","00000","listening on Unix socket ""/tmp/.s.PGSQL.7000""",,,,,,,0,,"pqcomm.c",618, 2024-10-15 15:06:44.810222 UTC,,,p4392,th-1770865664,,,,0,,,seg-1,,,,,"LOG","00000","database system was interrupted; last known up at 2024-10-15 15:06:10 UTC",,,,,,,0,,"xlog.c",6817, 2024-10-15 15:06:44.810255 UTC,,,p4392,th-1770865664,,,,0,,,seg-1,,,,,"LOG","00000","creating missing WAL directory ""pg_wal/archive_status""",,,,,,,0,,"xlog.c",4382, 2024-10-15 15:06:44.925788 UTC,,,p4392,th-1770865664,,,,0,,,seg-1,,,,,"LOG","00000","starting point-in-time recovery to ""backup_20241015T150608Z""",,,,,,,0,,"xlog.c",6916, 2024-10-15 15:06:45.876270 UTC,,,p4392,th-1770865664,,,,0,,,seg-1,,,,,"LOG","00000","restored log file ""000000010000000000000003"" from archive",,,,,,,0,,"xlogarchive.c",219, 2024-10-15 15:06:45.980452 UTC,,,p4392,th-1770865664,,,,0,,,seg-1,,,,,"LOG","00000","redo starts at 0/C000028",,,,,,,0,,"xlog.c",7675, 2024-10-15 15:06:46.868777 UTC,,,p4392,th-1770865664,,,,0,,,seg-1,,,,,"LOG","00000","restored log file ""000000010000000000000004"" from archive",,,,,,,0,,"xlogarchive.c",219, 2024-10-15 15:06:47.052610 UTC,,,p4392,th-1770865664,,,,0,,,seg-1,,,,,"LOG","00000","restored log file ""000000010000000000000005"" from archive",,,,,,,0,,"xlogarchive.c",219, 2024-10-15 15:06:47.152175 UTC,,,p4392,th-1770865664,,,,0,,,seg-1,,,,,"LOG","00000","recovery stopping at restore point ""backup_20241015T150608Z"", time 2024-10-15 15:06:28.640107+00",,,,,,,0,,"xlog.c",6053, 2024-10-15 15:06:47.162836 UTC,,,p4392,th-1770865664,,,,0,,,seg-1,,,,,"FATAL","XX000","requested recovery stop point is before consistent recovery point",,,,,,,0,,"xlog.c",7925,"Stack trace: 1 0xffff9aafd8c8 libpostgres.so errstart + 0x260 2 0xffff9a5a963c libpostgres.so StartupXLOG + 0x4f64 3 0xffff9a8d3c88 libpostgres.so StartupProcessMain + 0x108 4 0xffff9a5e87fc libpostgres.so AuxiliaryProcessMain + 0x6d4 5 0xffff9a8d33e8 libpostgres.so PostmasterMain + 0x11f0 6 0xaaaaddba1a08 postgres main + 0x4f8 7 0xffff99ed0e10 libc.so.6 __libc_start_main + 0xe8 8 0xaaaaddba1bb8 postgres <symbol not found> + 0xddba1bb8 ``` </details>
Merge pull request wal-g#1808 from NotAmigo/whitelist-for-mongodb-dat… …adir-clean add whitelist for mongodb datadir deletion protection
fix error when using WALG_S3_CA_CERT_FILE (wal-g#1764) Passes the custom CA bundle to the AWS sdk the first time when creating the session to avoid error: "HTTPClient's transport unsupported type caused by: unsupported transport, *s3.loggingTransport"
Merge pull request wal-g#1746 from wal-g/mongodb_manual_timeouts [mongodb] add opportunity set timeout for recovery oplog
Enable using aws iam roles (wal-g#1377) * enable using aws iam roles * added new variables to docs, fixed link to AWS config docs (seems the old one had been moved)
Add pgbackrest wal-fetch and wal-show commands (wal-g#1202) * Write pgbackrest wal-fetch and wal-show commands * Test pgbakcrest wal-fetch functionality * Fix lint * Download file without extension as last resort * Fix pgbench command * Fix build * Fix pgbackrest test * Update PostgreSQL.md * Update PostgreSQL.md * Add minor adjustments to DownloadAndDecompressStorageFile Co-authored-by: usernamedt <usernamedt@yandex-team.com>
Add --restore-point flag to Greenplum backup-fetch (wal-g#1221) * Add --restore-point flag to the greenplum backup-fetch * Add tests * Fix problems
PreviousNext