10000 s3 tiered storage policies - default disk tried to restore into object disk · Issue #1121 · Altinity/clickhouse-backup · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
s3 tiered storage policies - default disk tried to restore into object disk #1121
Open
@Slach

Description

@Slach
 <storage_configuration>
        <disks>
            <miniocached>
                <type>s3</type>
                <endpoint>http://minio:8000/data/s3</endpoint>
                <use_environment_credentials>0</use_environment_credentials>
                <access_key_id>minioadmin</access_key_id>
                <secret_access_key>minioadmin</secret_access_key>
            </miniocached>
            <cache>
                <type>cache</type>
                <disk>miniocached</disk>
                <path>./disks/minio_cache/</path>
                <max_size>50Gi</max_size> <!-- 50GB local cache to cache remote data -->
            </cache>
        </disks>
        <policies>
            <miniotiered>
                <volumes>
                    <default>
                        <disk>default</disk>
                        <max_data_part_size_bytes>50000000000</max_data_part_size_bytes> <!-- only for parts less than 50GB after they moved to minio during merges -->
                    </default>
                    <minio_cache>
                        <disk>cache</disk> <!-- sandwich cache plus miniocached -->
                        <!-- prefer_not_to_merge>true</prefer_not_to_merge>
                        <perform_ttl_move_on_insert>false</perform_ttl_move_on_insert-->
                    </minio_cache>
                </volumes>
            </miniotiered>
        </policies>
    </storage_configuration>
clickhouse-backup create my_backup
clickhouse-backup restore my_backup

during restore wrong destination disk choosen

 > Link /Users/MManj3/backup/all_partition_backup/shadow/default/test_s3/default/all_1_1_0/checksums.txt -> disks/miniocached/store/a3d/a3da913a-700c-4ba0-b3d0-a01fd00d51ff/detached/all_1_1_0/checksums.txt
> Link /Users/MManj3/backup/all_partition_backup/shadow/default/test_s3/default/all_1_1_0/columns.txt -> disks/miniocached/store/a3d/a3da913a-700c-4ba0-b3d0-a01fd00d51ff/detached/all_1_1_0/columns.txt

and

2025-03-24 15:56:49.135 INF pkg/clickhouse/clickhouse.go:1122 > ALTER TABLE `default`.`test_s3` ATTACH PART 'all_1_1_0'
2025-03-24 15:56:49.269 INF pkg/clickhouse/clickhouse.go:335 > clickhouse connection closed
2025-03-24 15:56:49.269 FTL cmd/clickhouse-backup/main.go:742 > error="one of restoreDataRegular go-routine return error: can't attach data parts for table 'default.test_s3': code: 27, message: Cannot parse input: expected '\\n' before: 'gu\x13Ź\\f\\\\\x03dv\x1cO�\x02��\"\\0\\0\\0\x10\\0\\0\\0(�/� \x10�\\0\\0>\v\\0\\0\\0\\0\\0\\0>\v\\0\\0\\0\\0\\0\\0'"
➜  ~ 
SELECT d.path AS path, any(d.name) AS name,
any(lower(if(d.type='ObjectStorage',d.object_storage_type,d.type))) AS type,
min(d.free_space) AS free_space, 
groupUniqArray(s.policy_name) AS storage_policies 
FROM system.disks AS d  LEFT JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s 
ON s.disk = d.name 
GROUP BY d.path

return following results

Image

< 5146 div height="sm" width="random" class="Box-sc-g0xbh4-0 LoadingSkeleton-sc-695d630a-0 hpcToU izpEpw">

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions

    0