8000 Problem: `pgcopydb copy extension` fails when sequences exists part of ext configuration by arajkumar · Pull Request #621 · dimitri/pgcopydb · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Problem: pgcopydb copy extensi 8000 on fails when sequences exists part of ext configuration #621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jan 5, 2024
Merged

Problem: pgcopydb copy extension fails when sequences exists part of ext configuration #621

merged 1 commit into from
Jan 5, 2024

Conversation

arajkumar
Copy link
Contributor
@arajkumar arajkumar commented Jan 5, 2024 •
edited
Loading

Solution: Populate extension configuration objects along with their type values. Based on the type value, perform either table copy or sequence copy.

At the moment we support only copying extension configuration tables & sequences.

I didn't add test for this PR, because I've to hunt the extension which has sequence as part of the config table. I've manually tested with timescaledb which happens to have few sequences part of the extension config table.

pgcopydb list extensions
16:26:50.957 49804 INFO   Running pgcopydb version 0.14.1.58.g1f03537 from "/Users/arajkumar/works/dimitri/pgcopydb/src/bin/pgcopydb/pgcopydb"
16:26:50.969 49804 INFO   Using work dir "/tmp/pgcopydb"
16:26:50.973 49804 INFO   Re-using catalog caches
       OID |                      Name |               Schema |      Count | Config
-----------+---------------------------+----------------------+------------+-----------
     14568 |                   plpgsql |           pg_catalog |          0 |
     16420 |               timescaledb |               public |         34 | "_timescaledb_catalog"."hypertable_id_seq","_timescaledb_catalog"."hypertable","_timescaledb_catalog"."hypertable_data_node","_timescaledb_catalog"."tablespace","_timescaledb_catalog"."dimension_id_seq","_timescaledb_catalog"."dimension","_timescaledb_catalog"."dimension_partition","_timescaledb_catalog"."dimension_slice_id_seq","_timescaledb_catalog"."dimension_slice","_timescaledb_catalog"."chunk_id_seq","_timescaledb_catalog"."chunk","_timescaledb_catalog"."chunk_constraint","_timescaledb_catalog"."chunk_constraint_name","_timescaledb_catalog"."chunk_index","_timescaledb_catalog"."chunk_data_node","_timescaledb_config"."bgw_job_id_seq","_timescaledb_config"."bgw_job","_timescaledb_catalog"."metadata","_timescaledb_catalog"."continuous_agg","_timescaledb_catalog"."continuous_aggs_bucket_function","_timescaledb_catalog"."continuous_aggs_invalidation_threshold","_timescaledb_catalog"."continuous_aggs_watermark","_timescaledb_catalog"."continuous_aggs_hypertable_invalidation_log","_timescaledb_catalog"."continuous_aggs_materialization_invalidation_log","_timescaledb_catalog"."hypertable_compression","_timescaledb_catalog"."compression_chunk_size","_timescaledb_catalog"."remote_txn","_timescaledb_catalog"."continuous_agg_migrate_plan","_timescaledb_catalog"."continuous_agg_migrate_plan_step_step_id_seq","_timescaledb_catalog"."continuous_agg_migrate_plan_step","_timescaledb_internal"."job_errors","_timescaledb_cache"."cache_inval_hypertable","_timescaledb_cache"."cache_inval_bgw_job","_timescaledb_cache"."cache_inval_extension"

With the fix, I could copy the timescale extension from source to target,

pgcopydb copy extensions --restart
11:00:32.933 2483944 INFO   Running pgcopydb version 0.14.1.27.g7b14081 from "/home/ubuntu/dimitri/pgcopydb/src/bin/pgcopydb/pgcopydb"
11:00:32.933 2483944 INFO   [SOURCE] Copying database from "postgres://xxxxo:xxx/defaultdb?sslmode=require&keepalives=1&keepalives_idle=10&keepalives_interval=10&keepalives_count=60"
11:00:32.933 2483944 INFO   [TARGET] Copying database into "postgres://xxx:xx/tsdb?sslmode=require&keepalives=1&keepalives_idle=10&keepalives_interval=10&keepalives_count=60"
11:00:32.966 2483944 INFO   Using work dir "/tmp/pgcopydb"
11:00:33.131 2483944 INFO   Exported snapshot "00000032-0000C683-1" from the source database
11:00:33.899 2483944 INFO   Fetched information for 2 extensions
11:00:34.174 2483944 INFO   Found 8 indexes (supporting 2 constraints) in the target database
11:00:34.193 2483944 INFO   Creating extension "plpgsql"
11:00:34.211 2483944 WARN   NOTICE:  extension "plpgsql" already exists, skipping
11:00:34.212 2483944 INFO   Creating extension "timescaledb"
11:00:34.246 2483944 WARN   NOTICE:  extension "timescaledb" already exists, skipping
11:00:34.260 2483944 INFO   COPY extension "timescaledb" configuration sequence _timescaledb_catalog.hypertable_id_seq
11:00:34.298 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.hypertable
11:00:34.338 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.hypertable_data_node
11:00:34.376 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.tablespace
11:00:34.413 2483944 INFO   COPY extension "timescaledb" configuration sequence _timescaledb_catalog.dimension_id_seq
11:00:34.449 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.dimension
11:00:34.488 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.dimension_partition
11:00:34.528 2483944 INFO   COPY extension "timescaledb" configuration sequence _timescaledb_catalog.dimension_slice_id_seq
11:00:34.564 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.dimension_slice
11:00:34.605 2483944 INFO   COPY extension "timescaledb" configuration sequence _timescaledb_catalog.chunk_id_seq
11:00:34.641 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.chunk
11:00:34.679 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.chunk_constraint
11:00:34.718 2483944 INFO   COPY extension "timescaledb" configuration sequence _timescaledb_catalog.chunk_constraint_name
11:00:34.756 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.chunk_index
11:00:34.801 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.chunk_data_node
11:00:34.840 2483944 INFO   COPY extension "timescaledb" configuration sequence _timescaledb_config.bgw_job_id_seq
11:00:34.876 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_config.bgw_job
11:00:34.914 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.metadata
11:00:34.952 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.continuous_agg
11:00:34.990 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.continuous_aggs_bucket_function
11:00:35.039 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.continuous_aggs_invalidation_threshold
11:00:35.078 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.continuous_aggs_watermark
11:00:35.115 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.continuous_aggs_hypertable_invalidation_log
11:00:35.152 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.continuous_aggs_materialization_invalidation_log
11:00:35.192 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.hypertable_compression
11:00:35.232 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.compression_chunk_size
11:00:35.271 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.remote_txn
11:00:35.309 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.continuous_agg_migrate_plan
11:00:35.348 2483944 INFO   COPY extension "timescaledb" configuration sequence _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq
11:00:35.384 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_catalog.continuous_agg_migrate_plan_step
11:00:35.421 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_internal.job_errors
11:00:35.459 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_cache.cache_inval_hypertable
11:00:35.496 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_cache.cache_inval_bgw_job
11:00:35.533 2483944 INFO   COPY extension "timescaledb" configuration table _timescaledb_cache.cache_inval_extension

@arajkumar arajkumar changed the title Problem: Extension configuration sequences are not copied Problem: pgcopydb copy extension fails when sequence exists part of ext configuration Jan 5, 2024
@arajkumar arajkumar changed the title Problem: pgcopydb copy extension fails when sequence exists part of ext configuration Problem: pgcopydb copy extension fails when sequences exists part of ext configuration Jan 5, 2024
@arajkumar arajkumar force-pushed the ext-config-seq branch 2 times, most recently from 012b391 to f4879cb Compare January 5, 2024 10:54
@arajkumar
Copy link
Contributor Author

This will be useful to revive #312.

dimitri
dimitri requested changes Jan 5, 2024
View reviewed changes
Copy link
Owner
@dimitri dimitri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First round of review is in, thanks!

@arajkumar arajkumar marked this pull request as draft January 5, 2024 11:58
@arajkumar arajkumar force-pushed the ext-config-seq branch 2 times, most recently from 916ef30 to a6adc0c Compare January 5, 2024 12:17
@arajkumar
Problem: pgcopydb copy extension fails when sequences exists part o…
797522b 
…f ext configuration

Solution: Populate extension configuration objects along with their type
values. Based on the type value, perform either table copy or sequence copy.

At the moment we support only copying extension configuration tables & sequences.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
@arajkumar arajkumar marked this pull request as ready for review January 5, 2024 12:17
@arajkumar arajkumar requested a review from dimitri January 5, 2024 12:22
@arajkumar
Copy link
Contributor Author

Thanks @dimitri , addressed your review comments. PTAL.

@dimitri dimitri merged commit c343e20 into dimitri:main Jan 5, 2024
@dimitri dimitri added the bug Something isn't working label Jan 5, 2024
@dimitri dimitri added this to the v0.15 milestone Jan 5, 2024
arajkumar added a commit to arajkumar/pgcopydb that referenced this pull request Jan 17, 2024
@arajkumar
Problem: pgcopydb copy extension fails when sequences exists part o…
75d2770 
…f ext configuration (dimitri#621) (dimitri#25)

Solution: Populate extension configuration objects along with their type
values. Based on the type value, perform either table copy or sequence copy.

At the moment we support only copying extension configuration tables & sequences.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
arajkumar added a commit to arajkumar/pgcopydb that referenced this pull request Jan 18, 2024
@arajkumar
Problem: pgcopydb copy extension fails when sequences exists part o…
a67d101 
…f ext configuration (dimitri#621)

Solution: Populate extension configuration objects along with their type
values. Based on the type value, perform either table copy or sequence copy.

At the moment we support only copying extension configuration tables & sequences.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
arajkumar added a commit to arajkumar/pgcopydb that referenced this pull request Jan 18, 2024
@arajkumar
Problem: pgcopydb copy extension fails when sequences exists part o…
64a48e2 
…f ext configuration (dimitri#621)

Solution: Populate extension configuration objects along with their type
values. Based on the type value, perform either table copy or sequence copy.

At the moment we support only copying extension configuration tables & sequences.

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dimitri dimitri dimitri approved these changes

Assignees

@arajkumar arajkumar

Labels
bug Something isn't working
Projects
None yet
Milestone
v0.15
Development

Successfully merging this pull request may close these issues.
2 participants
@arajkumar @dimitri
0