Description
Expected behavior
Running mc mirror myminio/bucket1 myminio/bucket2 where bucket1 uses tiered storage should create a full replica to bucket2
Actual behavior
Not all data will be replicated between the buckets. In my case, bucket1 has 3493 objects (69.4GiB). where 2372 objects (56.7GiB) have been transitioned to the tiered storage.
When running mc mirror --debug myminio/bucket1 myminio/bucket2, it appears to stall after approximately 2000 objects. During this period, network activity still suggests uploading from the cold tier to the hot tier. Eventually the mc mirror commands finishes with an exit code of 1 with an error to the following effect:
Error
{
"status": "error",
"error": {
"message": "Failed to copy `http://10.200.0.8:9000/gitea-packages/packages/00/3f/003f1109a21287fa17dc866e87e8c6685113960cbb0379fee8f42b83de63c647`.",
"cause": {
"message": "Put \"http://10.200.0.8:9000/gitea-packages-mc-mirror-disable-multipart/packages/00/3f/003f1109a21287fa17dc866e87e8c6685113960cbb0379fee8f42b83de63c647\": read tcp 172.17.245.99:48308-\u003e10.200.0.8:9000: i/o timeout",
"error": {
"Op": "Put",
"URL": "http://10.200.0.8:9000/gitea-packages-mc-mirror-disable-multipart/packages/00/3f/003f1109a21287fa17dc866e87e8c6685113960cbb0379fee8f42b83de63c647",
"Err": {
"Op": "read",
"Net": "tcp",
"Source": {
"IP": "172.17.245.99",
"Port": 48308,
"Zone": ""
},
"Addr": {
"IP": "10.200.0.8",
"Port": 9000,
"Zone": ""
},
"Err": {}
}
}
},
"type": "error",
"trace": [
{
"line": 1063,
"file": "client-s3.go",
"func": "cmd.(*S3Client).Copy"
},
{
"line": 246,
"file": "common-methods.go",
"func": "cmd.copySourceToTargetURL",
"env": {
"Tags": [
"myminio",
"http://10.200.0.8:9000/gitea-packages-mc-mirror-disable-multipart/packages/00/3f/003f1109a21287fa17dc866e87e8c6685113960cbb0379fee8f42b83de63c647"
]
}
},
{
"line": 509,
"file": "common-methods.go",
"func": "cmd.uploadSourceToTargetURL",
"env": {
"Tags": [
"http://10.200.0.8:9000/gitea-packages/packages/00/3f/003f1109a21287fa17dc866e87e8c6685113960cbb0379fee8f42b83de63c647"
]
}
},
{
"line": 558,
"file": "mirror-main.go",
"func": "cmd.(*mirrorJob).monitorMirrorStatus",
"env": {
"Tags": [
"http://10.200.0.8:9000/gitea-packages/packages/00/3f/003f1109a21287fa17dc866e87e8c6685113960cbb0379fee8f42b83de63c647"
]
}
}
],
"sysinfo": {
"host.arch": "amd64",
"host.cpus": "16",
"host.lang": "go1.22.4",
"host.name": "MINI-PC",
"host.os": "linux",
"mem.heap.total": "27 MiB",
"mem.heap.used": "16 MiB",
"mem.total": "44 MiB",
"mem.used": "16 MiB"
}
}
}Steps to reproduce the behavior
- Create a bucket that uses tiered storage
- Allow some data to transition to the tiered storage
- Try and replicate the bucket to another bucket on the same host using
mc mirrorfrom a remote client machine - Not all data will be replicated. The command will exit complaining of an
i/o timeout
mc --version
mc version RELEASE.2024-06-24T19-40-33Z (commit-id=3548007d5bd1ba8e2ed98f35d95c4776dd9a7ff9)
Runtime: go1.22.4 linux/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
System information
My minio deployment consists of two SNSD instances. The primary instance runs on an OCI VPS. Due to limited storage on the VPS, I have a second instance that runs on my LAN. Here I have much more storage but a slower network. This is where I transition things to. The tiers are connected via a Wireguard VPN tunnel that is constrained primarily by my home network connection (1000/120mbps). Due to some overhead, this means the maximum rate at which server1 can pull data from server2 is ~80-100mbps
I originally ran nginx in front of both minio servers, but during my troubleshooting I circumvented it in case it was the cause of this issue. Both servers communicate directly now with each other and clients.
Both minio servers are on RELEASE.2024-06-24T19-40-33Z and run in docker. An example docker compose:
minio:
image: minio/minio:RELEASE.2024-06-28T09-06-49Z
container_name: minio
restart: unless-stopped
user: ${PUID}:${PGID}
depends_on:
authentik-checker:
condition: service_completed_successfully
networks:
- general
- nginx
ports:
- ${WG_IP}:9000:9000
command: server --console-address ":9090" /data
volumes:
- ${DBULK}/minio:/data
environment:
- MINIO_ROOT_USER=rootuser
- MINIO_ROOT_PASSWORD=${MINIO_ROOT_PW}
- MINIO_SERVER_URL=https://minio.domain.com
- MINIO_BROWSER_REDIRECT_URL=https://minio.domain.com/web/uiI have captured the output of mc mirror --debug --json running various different tests. I'm happy to share if useful, but would need instructions on which details to remove from them, since all headers are of course visible.
Things that work
- Server-side replication
rclone sync myminio/bucket1 myminio/bucket2mc mirror myminio/bucket1 myminio/bucket2when all objects are on the hot tiermc mirror myminio/bucket1 ./localfoldermc cp myminio/bucket1 ./localfolder
Things I have tried that had no effect
- Removing nginx
- Enabling traffic shaping on my LAN to ensure the limited upload bandwidth is not overwhelmed (without this, the network would be overwhelmed and significant packet loss would occur, resetting the connection. I thought this was the root cause initially)
- Using
--disable-multipart - Running
mc mirrordirectly in the minio container (docker compose exec minio mc mirror...) - this hung after about 1100 objects, but otherwise behaved as described above
mc in all instances is being run from in Debian WSL on my LAN. My tests initially started on the Windows client, but I moved to Linux to check it wasn't limited to just the Windows client.
While I first thought this was an issue with my minio deployment, now I don't believe it is