8000 clickhouse operator doesn't create schema automatically when new shards are added in the cluster · Issue #1703 · Altinity/clickhouse-operator · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

clickhouse operator doesn't create schema automatically when new shards are added in the cluster #1703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
kushal1794 opened this issue May 8, 2025 · 1 comment
Closed

clickhouse operator doesn't create schema automatically when new shards are added in the cluster #1703

kushal1794 opened this issue May 8, 2025 · 1 comment

Comments

@kushal1794
Copy link
kushal1794 commented May 8, 2025 •
edited
Loading

Hi All,

We are having currently 3-shards and 3replica cluster in clickhouse, now we scaled up the cluster with 5shards-3replicas. After scaledup we can see that cluster got scaled but schemas are not created automatically on the newly created shards.

So request you to please help for this issue.

Clickhouse operator version - 0.18.0
clickhouse server version - clickhouse-server:24.8-alpine

#############. Operator logs post scaling #################

I0507 07:15:46.567638 1 announcer.go:218] reconcileHost():cmass-1c/cliff:Reconcile Host replica0-shard3 started
I0507 07:15:46.773072 1 announcer.go:218] getPodTemplate():cmass-1c/cliff:statefulSet chi-cliff-cliffcluster-replica0-shard3 use custom template pod-template-with-volumes-shard
I0507 07:15:46.918052 1 announcer.go:218] createConfigMap():cmass-1c/cliff:Create ConfigMap cmass-1c/chi-cliff-deploy-confd-cliffcluster-replica0-shard3
I0507 07:15:47.163871 1 announcer.go:218] getPodTemplate():cmass-1c/cliff:statefulSet chi-cliff-cliffcluster-replica0-shard3 use custom template pod-template-with-volumes-shard
I0507 07:15:47.166196 1 announcer.go:218] createStatefulSet():cmass-1c/cliff:Create StatefulSet cmass-1c/chi-cliff-cliffcluster-replica0-shard3 - started
I0507 07:15:47.373145 1 announcer.go:220] createStatefulSet():cmass-1c/cliff
I0507 07:15:47.373172 1 announcer.go:218] Create StatefulSet cmass-1c/chi-cliff-cliffcluster-replica0-shard3
I0507 07:15:47.515288 1 announcer.go:218] deleteLabelReady():cmass-1c/cliff:FAIL get pod for host cmass-1c/replica0-shard3 err:pods "chi-cliff-cliffcluster-replica0-shard3-0" not found
I0507 07:15:48.522255 1 announcer.go:218] deleteLabelReady():cmass-1c/cliff:FAIL get pod for host cmass-1c/replica0-shard3 err:pods "chi-cliff-cliffcluster-replica0-shard3-0" not found
I0507 07:15:49.527753 1 announcer.go:218] deleteLabelReady():cmass-1c/cliff:FAIL get pod for host cmass-1c/replica0-shard3 err:pods "chi-cliff-cliffcluster-replica0-shard3-0" not found
I0507 07:15:52.558468 1 announcer.go:218] pollStatefulSet():cmass-1c/chi-cliff-cliffcluster-replica0-shard3:OK :ObservedGeneration:1 Replicas:1 ReadyReplicas:0 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-cliff-cliffcluster-replica0-shard3-849cf965d8 UpdateRevision:chi-cliff-cliffcluster-replica0-shard3-849cf965d8
I0507 07:16:53.168240 1 announcer.go:218] pollStatefulSet():cmass-1c/chi-cliff-cliffcluster-replica0-shard3:WAIT:ObservedGeneration:1 Replicas:1 ReadyReplicas:0 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-cliff-cliffcluster-replica0-shard3-849cf965d8 UpdateRevision:chi-cliff-cliffcluster-replica0-shard3-849cf965d8
I0507 07:16:58.209359 1 announcer.go:218] pollStatefulSet():cmass-1c/chi-cliff-cliffcluster-replica0-shard3:OK :ObservedGeneration:1 Replicas:1 ReadyReplicas:1 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-cliff-cliffcluster-replica0-shard3-849cf965d8 UpdateRevision:chi-cliff-cliffcluster-replica0-shard3-849cf965d8
I0507 07:16:58.278282 1 announcer.go:218] createStatefulSet():cmass-1c/cliff:Create StatefulSet cmass-1c/chi-cliff-cliffcluster-replica0-shard3 - completed
I0507 07:17:01.192156 1 announcer.go:218] CreateServiceHost():cmass-1c/cliff:cmass-1c/chi-cliff-cliffcluster-replica0-shard3 for Set chi-cliff-cliffcluster-replica0-shard3
I0507 07:17:01.592918 1 announcer.go:218] createService():cmass-1c/cliff:Create Service cmass-1c/chi-cliff-cliffcluster-replica0-shard3
I0507 07:17:01.842342 1 announcer.go:218] migrateTables():cmass-1c/cliff:Adding tables on shard/host:3/0 cluster:cliffcluster
I0507 07:17:02.037658 1 announcer.go:218] HostCreateTables():cmass-1c/cliff:Migrating schema objects to host replica0-shard3
I0507 07:17:02.037809 1 announcer.go:218] getCreateReplicaObjects():cmass-1c/cliff:Extracting replicated table definitions from [chi-cliff-cliffcluster-replica0-shard3.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard3.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard3.cmass-1c.svc.cluster.local]
I0507 07:17:02.038163 1 announcer.go:218] Run query on: chi-cliff-cliffcluster-replica0-shard0.cmass-1c.svc.cluster.local of [chi-cliff-cliffcluster-replica0-shard0.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard0.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard0.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica0-shard1.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard1.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard1.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica0-shard2.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard2.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard2.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica0-shard3.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard3.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard3.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica0-shard4.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard4.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard4.cmass-1c.svc.cluster.local]
E0507 07:17:02.045903 1 announcer.go:265] connection.go:83:connect():FAILED Ping(http://:@chi-cliff-cliffcluster-replica0-shard0.cmass-1c.svc.cluster.local:8123/). Err: dial tcp: lookup chi-cliff-cliffcluster-replica0-shard0.cmass-1c.svc.cluster.local on [240b:c0e0:104:544d:b464:2:0:800a]:53: no such host
E0507 07:17:02.045965 1 announcer.go:267] connection.go:120:QueryContext():FAILED connect(http://:@chi-cliff-cliffcluster-replica0-shard0.cmass-1c.svc.cluster.local:8123/) for SQL: SELECT
DISTINCT name,
'CREATE DATABASE IF NOT EXISTS "' || name || '" Engine = ' || engine AS create_db_query
FROM clusterAllReplicas('cliffcluster', system.databases) databases
WHERE name != 'system'
SETTINGS skip_unavailable_shards = 1
W0507 07:17:02.045991 1 announcer.go:245] cluster.go:91:QueryAny():FAILED to run query on: chi-cliff-cliffcluster-replica0-shard0.cmass-1c.svc.cluster.local of [chi-cliff-cliffcluster-replica0-shard0.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard0.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard0.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica0-shard1.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard1.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard1.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica0-shard2.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard2.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard2.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica0-shard3.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard3.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard3.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica0-shard4.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard4.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard4.cmass-1c.svc.cluster.local] skip to next. err: FAILED connect(http://:@chi-cliff-cliffcluster-replica0-shard0.cmass-1c.svc.cluster.local:8123/) for SQL: SELECT
DISTINCT name,
'CREATE DATABASE IF NOT EXISTS "' || name || '" Engine = ' || engine AS create_db_query
FROM clusterAllReplicas('cliffcluster', system.databases) databases
WHERE name != 'system'
SETTINGS skip_unavailable_shards = 1
I0507 07:17:02.046022 1 announcer.go:218] Run query on: chi-cliff-cliffcluster-replica1-shard0.cmass-1c.svc.cluster.local of [chi-cliff-cliffcluster-replica0-shard0.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard0.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard0.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica0-shard1.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard1.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard1.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica0-shard2.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard2.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard2.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica0-shard3.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard3.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard3.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica0-shard4.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica1-shard4.cmass-1c.svc.cluster.local chi-cliff-cliffcluster-replica2-shard4.cmass-1c.svc.cluster.local]
E0507 07:17:02.053838 1 announcer.go:265] connection.go:83:connect():FAILED Ping(http://:@chi-cliff-cliffcluster-replica1-shard0.cmass-1c.svc.cluster.local:8123/). Err: dial tcp: lookup chi-cliff-cliffcluster-replica1-shard0.cmass-1c.svc.cluster.local on [240b:c0e0:104:544d:b464:2:0:800a]:53: no such host
E0507 07:17:02.053896 1 announcer.go:267] connection.go:120:QueryContext():FAILED connect(http://:@chi-cliff-cliffcluster-replica1-shard0.cmass-1c.svc.cluster.local:8123/) for SQL: SELECT
DISTINCT name,
'CREATE DATABASE IF NOT EXISTS "' || name || '" Engine = ' || engine AS create_db_query
@Slach
Copy link
Collaborator
Slach commented May 8, 2025

Clickhouse operator version - 0.18.0

this is 3 years outdated
upgrade to 0.24.5

@Slach Slach closed this as completed May 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Slach @kushal1794
0