8000 Repeatedly restarting containers Error: could not find config file supervisord/worker.export.conf · Issue #9428 · cvat-ai/cvat · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Repeatedly restarting containers Error: could not find config file supervisord/worker.export.conf #9428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
2 tasks done
hhhfsj opened this issue May 16, 2025 · 1 comment
Closed
2 tasks done

Repeatedly restarting containers Error: could not find config file supervisord/worker.export.conf #9428

hhhfsj opened this issue May 16, 2025 · 1 comment

Comments

@hhhfsj
Copy link
hhhfsj commented May 16, 2025 •
edited
Loading

Actions before raising this issue

  • I searched the existing issues and did not find anything similar.
  • I read/searched the docs

Steps to Reproduce

I am experiencing an issue where several containers that rely on supervisord are failing to start. Previously, everything was working fine, but at some point (I'm not sure exactly when), the services stopped and now cannot be restarted.

Here are some logs from the affected containers:

cvat_worker_export  | wait-for-it: waiting for cvat_db:5432 without a timeout
cvat_worker_export  | wait-for-it: cvat_db:5432 is available after 0 seconds
cvat_worker_export  | waiting for migrations to complete...
cvat_worker_export  | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_export  | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_export  | waiting for Redis migrations to complete...
cvat_worker_export  | Additional supervisor configs that will be included: 
cvat_worker_export  | Error: could not find config file supervisord/worker.export.conf
cvat_worker_export  | For help, use /usr/bin/supervisord -h

cvat_worker_webhooks  | wait-for-it: waiting for cvat_db:5432 without a timeout
cvat_worker_webhooks  | wait-for-it: cvat_db:5432 is available after 0 seconds
cvat_worker_webhooks  | waiting for migrations to complete...
cvat_worker_webhooks  | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_webhooks  | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_webhooks  | waiting for Redis migrations to complete...
cvat_worker_webhooks  | Additional supervisor configs that will be included: 
cvat_worker_webhooks  | Error: could not find config file supervisord/worker.webhooks.conf
cvat_worker_webhooks  | For help, use /usr/bin/supervisord -h

I don't know how to reliably reproduce the issue, as it seemed to occur spontaneously. Any help or suggestions would be appreciated.

Expected Behavior

No response

Possible Solution

No response

Context

full logs

WARN[0000] Found orphan containers ([nuclio]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. 
[+] Running 19/0
 ✔ Container cvat_grafana                   Created                                                                                                                                                                0.0s 
 ✔ Container cvat_db                        Created                                                                                                                                                                0.0s 
 ✔ Container cvat_opa                       Created                                                                                                                                                                0.0s 
 ✔ Container traefik                        Created                                                                                                                                                                0.0s 
 ✔ Container cvat_clickhouse                Created                                                                                                                                                                0.0s 
 ✔ Container cvat_redis_ondisk              Created                                                                                                                                                                0.0s 
 ✔ Container cvat_redis_inmem               Created                                                                                                                                                                0.0s 
 ✔ Container cvat_vector                    Created                                                                                                                                                                0.0s 
 ✔ Container cvat_worker_import             Created                                                                                                                                                                0.0s 
 ✔ Container cvat_worker_annotation         Created                                                                                                                                   
10000
                             0.0s 
 ✔ Container cvat_worker_chunks             Created                                                                                                                                                                0.0s 
 ✔ Container cvat_worker_webhooks           Created                                                                                                                                                                0.0s 
 ✔ Container cvat_worker_export             Created                                                                                                                                                                0.0s 
 ✔ Container cvat_worker_quality_reports    Created                                                                                                                                                                0.0s 
 ✔ Container cvat_utils                     Created                                                                                                                                                                0.0s 
 ✔ Container cvat_worker_analytics_reports  Created                                                                                                                                                                0.0s 
 ✔ Container cvat_server                    Created                                                                                                                                                                0.0s 
 ✔ Container cvat_ui                        Created                                                                                                                                                                0.0s 
 ✔ Container cvat_worker_consensus          Created                                                                                                                                                                0.0s 
Attaching to cvat_clickhouse, cvat_db, cvat_grafana, cvat_opa, cvat_redis_inmem, cvat_redis_ondisk, cvat_server, cvat_ui, cvat_utils, cvat_vector, cvat_worker_analytics_reports, cvat_worker_annotation, cvat_worker_chunks, cvat_worker_consensus, cvat_worker_export, cvat_worker_import, cvat_worker_quality_reports, cvat_worker_webhooks, traefik
cvat_redis_inmem               | 1:C 16 May 2025 06:38:42.632 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
cvat_redis_inmem               | 1:C 16 May 2025 06:38:42.632 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
cvat_redis_inmem               | 1:C 16 May 2025 06:38:42.632 * Redis version=7.2.3, bits=64, commit=00000000, modified=0, pid=1, just started
cvat_redis_inmem               | 1:C 16 May 2025 06:38:42.632 * Configuration loaded
cvat_redis_inmem               | 1:M 16 May 2025 06:38:42.633 * monotonic clock: POSIX clock_gettime
cvat_redis_inmem               | 1:M 16 May 2025 06:38:42.633 * Running mode=standalone, port=6379.
cvat_redis_inmem               | 1:M 16 May 2025 06:38:42.633 * Server initialized
cvat_redis_inmem               | 1:M 16 May 2025 06:38:42.634 * Reading RDB base file on AOF loading...
cvat_redis_inmem               | 1:M 16 May 2025 06:38:42.634 * Loading RDB produced by version 7.2.3
cvat_redis_inmem               | 1:M 16 May 2025 06:38:42.634 * RDB age 729 seconds
cvat_redis_inmem               | 1:M 16 May 2025 06:38:42.634 * RDB memory usage when created 0.83 Mb
cvat_redis_inmem               | 1:M 16 May 2025 06:38:42.634 * RDB is base AOF
cvat_redis_inmem               | 1:M 16 May 2025 06:38:42.634 * Done loading RDB, keys loaded: 0, keys expired: 0.
cvat_redis_inmem               | 1:M 16 May 2025 06:38:42.634 * DB loaded from base file appendonly.aof.1.base.rdb: 0.000 seconds
cvat_redis_inmem               | 1:M 16 May 2025 06:38:42.634 * DB loaded from incr file appendonly.aof.1.incr.aof: 0.000 seconds
cvat_redis_inmem               | 1:M 16 May 2025 06:38:42.634 * DB loaded from append only file: 0.000 seconds
cvat_redis_inmem               | 1:M 16 May 2025 06:38:42.634 * Opening AOF incr file appendonly.aof.1.incr.aof on server start
cvat_redis_inmem               | 1:M 16 May 2025 06:38:42.634 * Ready to accept connections tcp
traefik                        | time="2025-05-16T06:38:42Z" level=info msg="Configuration loaded from environment variables."
traefik                        | {"entryPointName":"websecure","level":"error","msg":"entryPoint \"websecure\" doesn't exist","routerName":"grafana_https@file","time":"2025-05-16T06:38:43Z"}
traefik                        | {"level":"error","msg":"no valid entryPoint for this router","routerName":"grafana_https@file","time":"2025-05-16T06:38:43Z"}
traefik                        | {"entryPointName":"websecure","level":"error","msg":"entryPoint \"websecure\" doesn't exist","routerName":"grafana_https@file","time":"2025-05-16T06:38:43Z"}
traefik                        | {"level":"error","msg":"no valid entryPoint for this router","routerName":"grafana_https@file","time":"2025-05-16T06:38:43Z"}
cvat_redis_ondisk              | I20250516 06:38:43.335147     7 main.cc:141] kvrocks version 2.7.0 (commit 33b49591)
cvat_db                        | 
cvat_db                        | PostgreSQL Database directory appears to contain a database; Skipping initialization
cvat_db                        | 
cvat_db                        | 2025-05-16 06:38:43.619 UTC [1] LOG:  starting PostgreSQL 15.13 on x86_64-pc-linux-musl, compiled by gcc (Alpine 14.2.0) 14.2.0, 64-bit
cvat_db                        | 2025-05-16 06:38:43.619 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
cvat_db                        | 2025-05-16 06:38:43.619 UTC [1] LOG:  listening on IPv6 address "::", port 5432
cvat_redis_ondisk              | I20250516 06:38:43.711129     7 storage.cc:335] [storage] Success to load the data from disk: 368 ms
cvat_redis_ondisk              | I20250516 06:38:43.715867     7 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
cvat_redis_ondisk              | I20250516 06:38:43.716436     7 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
cvat_redis_ondisk              | I20250516 06:38:43.716853     7 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
cvat_redis_ondisk              | I20250516 06:38:43.717247     7 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
cvat_redis_ondisk              | I20250516 06:38:43.717640     7 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
cvat_redis_ondisk              | I20250516 06:38:43.718021     7 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
cvat_redis_ondisk              | I20250516 06:38:43.718427     7 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
cvat_redis_ondisk              | I20250516 06:38:43.718814     7 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
cvat_redis_ondisk              | I20250516 06:38:43.721256     7 worker.cc:555] [worker] Thread #140463238413104 started
cvat_redis_ondisk              | I20250516 06:38:43.721374     7 worker.cc:555] [worker] Thread #140463238236976 started
cvat_redis_ondisk              | I20250516 06:38:43.721479     7 worker.cc:555] [worker] Thread #140463238060848 started
cvat_redis_ondisk              | I20250516 06:38:43.721576     7 worker.cc:555] [worker] Thread #140463237884720 started
cvat_opa                       | {"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server on 127.0.0.11:53: no such host","name":"cvat","plugin":"bundle","time":"2025-05-16T06:38:43Z"}
cvat_opa                       | {"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server on 127.0.0.11:53: no such host","name":"cvat","plugin":"bundle","time":"2025-05-16T06:38:43Z"}
cvat_db                        | 2025-05-16 06:38:43.778 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
cvat_db                        | 2025-05-16 06:38:43.861 UTC [29] LOG:  database system was shut down at 2025-05-16 06:38:31 UTC
cvat_opa                       | {"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp: lookup cvat-server on 127.0.0.11:53: no such host","name":"cvat","plugin":"bundle","time":"2025-05-16T06:38:43Z"}
cvat_clickhouse                | /entrypoint.sh: create new user 'user' instead 'default'
cvat_clickhouse                | ClickHouse Database directory appears to contain a database; Skipping initialization
cvat_db                        | 2025-05-16 06:38:44.007 UTC [1] LOG:  database system is ready to accept connections
cvat_clickhouse                | Processing configuration file '/etc/clickhouse-server/config.xml'.
cvat_clickhouse                | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
cvat_clickhouse                | Logging trace to /var/log/clickhouse-server/clickhouse-server.log
cvat_clickhouse                | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
cvat_clickhouse                | Processing configuration file '/etc/clickhouse-server/config.xml'.
cvat_clickhouse                | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
cvat_clickhouse                | Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/config.xml'.
cvat_clickhouse                | Processing configuration file '/etc/clickhouse-server/users.xml'.
cvat_clickhouse                | Merging configuration file '/etc/clickhouse-server/users.d/default-user.xml'.
cvat_clickhouse                | Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.
cvat_worker_quality_reports    | wait-for-it: waiting for cvat_db:5432 without a timeout
cvat_worker_quality_reports    | wait-for-it: cvat_db:5432 is available after 0 seconds
cvat_db                        | 2025-05-16 06:38:45.533 UTC [33] LOG:  incomplete startup packet
cvat_worker_quality_reports    | waiting for migrations to complete...
cvat_server                    | wait-for-it: waiting for cvat_db:5432 without a timeout
cvat_opa                       | {"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.29.0.17:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2025-05-16T06:38:45Z"}
cvat_server                    | wait-for-it: cvat_db:5432 is available after 0 seconds
cvat_db                        | 2025-05-16 06:38:45.979 UTC [34] LOG:  incomplete startup packet
cvat_worker_import             | wait-for-it: waiting for cvat_db:5432 without a timeout
cvat_db                        | 2025-05-16 06:38:46.282 UTC [35] LOG:  incomplete startup packet
cvat_worker_import             | wait-for-it: cvat_db:5432 is available after 0 seconds
cvat_worker_export             | wait-for-it: waiting for cvat_db:5432 without a timeout
cvat_worker_import             | waiting for migrations to complete...
cvat_db                        | 2025-05-16 06:38:46.290 UTC [36] LOG:  incomplete startup packet
cvat_worker_export             | wait-for-it: cvat_db:5432 is available after 0 seconds
cvat_worker_export             | waiting for migrations to complete...
cvat_worker_analytics_reports  | wait-for-it: waiting for cvat_db:5432 without a timeout
cvat_db                        | 2025-05-16 06:38:46.415 UTC [37] LOG:  incomplete startup packet
cvat_worker_analytics_reports  | wait-for-it: cvat_db:5432 is available after 0 seconds
cvat_worker_analytics_reports  | waiting for migrations to complete...
cvat_opa                       | {"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.29.0.17:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2025-05-16T06:38:46Z"}
cvat_opa                       | {"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.29.0.17:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2025-05-16T06:38:47Z"}
traefik                        | {"entryPointName":"websecure","level":"error","msg":"entryPoint \"websecure\" doesn't exist","routerName":"grafana_https@file","time":"2025-05-16T06:38:47Z"}
traefik                        | {"level":"error","msg":"no valid entryPoint for this router","routerName":"grafana_https@file","time":"2025-05-16T06:38:47Z"}
traefik                        | {"entryPointName":"websecure","level":"error","msg":"entryPoint \"websecure\" doesn't exist","routerName":"grafana_https@file","time":"2025-05-16T06:38:47Z"}
traefik                        | {"level":"error","msg":"no valid entryPoint for this router","routerName":"grafana_https@file","time":"2025-05-16T06:38:47Z"}
cvat_worker_webhooks           | wait-for-it: waiting for cvat_db:5432 without a timeout
cvat_db                        | 2025-05-16 06:38:47.382 UTC [38] LOG:  incomplete startup packet
cvat_worker_webhooks           | wait-for-it: cvat_db:5432 is available after 0 seconds
cvat_worker_webhooks           | waiting for migrations to complete...
cvat_worker_consensus          | wait-for-it: waiting for cvat_db:5432 without a timeout
cvat_db                        | 2025-05-16 06:38:47.901 UTC [39] LOG:  incomplete startup packet
cvat_worker_consensus          | wait-for-it: cvat_db:5432 is available after 0 seconds
cvat_worker_consensus          | waiting for migrations to complete...
cvat_utils                     | wait-for-it: waiting for cvat_db:5432 without a timeout
cvat_db                        | 2025-05-16 06:38:48.227 UTC [40] LOG:  incomplete startup packet
cvat_utils                     | wait-for-it: cvat_db:5432 is available after 0 seconds
cvat_utils                     | waiting for migrations to complete...
cvat_opa                       | {"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.29.0.17:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2025-05-16T06:38:48Z"}
cvat_worker_chunks             | wait-for-it: waiting for cvat_db:5432 without a timeout
cvat_db                        | 2025-05-16 06:38:48.301 UTC [41] LOG:  incomplete startup packet
cvat_worker_chunks             | wait-for-it: cvat_db:5432 is available after 0 seconds
cvat_worker_chunks             | waiting for migrations to complete...
cvat_worker_annotation         | wait-for-it: waiting for cvat_db:5432 without a timeout
cvat_db                        | 2025-05-16 06:38:48.558 UTC [42] LOG:  incomplete startup packet
cvat_worker_annotation         | wait-for-it: cvat_db:5432 is available after 0 seconds
cvat_worker_annotation         | waiting for migrations to complete...
cvat_vector                    | 2025-05-16T06:38:48.744498Z  INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
cvat_vector                    | 2025-05-16T06:38:48.744632Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
cvat_vector                    | 2025-05-16T06:38:48.744721Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
cvat_vector                    | 2025-05-16T06:38:48.782825Z  INFO vector::topology::running: Running healthchecks.
cvat_vector                    | 2025-05-16T06:38:48.783037Z  INFO vector: Vector has started. debug="false" version="0.26.0" arch="x86_64" revision="c6b5bc2 2022-12-05"
cvat_vector                    | 2025-05-16T06:38:48.783081Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
cvat_vector                    | 2025-05-16T06:38:48.783836Z  INFO source{component_kind="source" component_id=http-events component_type=http_server component_name=http-events}: vector::sources::util::http::prelude: Building HTTP server. address=0.0.0.0:80
cvat_vector                    | 2025-05-16T06:38:48.787619Z ERROR vector::topology::builder: msg="Healthcheck: Failed Reason." error=Unexpected status: 404 Not Found component_kind="sink" component_type="clickhouse" component_id=clickhouse component_name=clickhouse
cvat_ui                        | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
cvat_ui                        | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
cvat_ui                        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
cvat_ui                        | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
cvat_ui                        | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf differs from the packaged version
cvat_ui                        | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
cvat_ui                        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
cvat_ui                        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
cvat_ui                        | /docker-entrypoint.sh: Configuration complete; ready for start up
cvat_ui                        | 2025/05/16 06:38:48 [warn] 1#1: duplicate MIME type "text/html" in /etc/nginx/conf.d/default.conf:18
cvat_ui                        | nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/conf.d/default.conf:18
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: using the "epoll" event method
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: nginx/1.27.4
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: built by gcc 14.2.0 (Alpine 14.2.0) 
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: OS: Linux 6.6.47-0-virt
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker processes
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 29
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 30
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 31
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 32
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 33
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 34
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 35
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 36
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 37
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 38
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 39
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 40
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 41
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 42
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 43
cvat_ui                        | 2025/05/16 06:38:48 [notice] 1#1: start worker process 44
cvat_worker_quality_reports    | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_quality_reports    | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_quality_reports    | waiting for Redis migrations to complete...
traefik                        | {"entryPointName":"websecure","level":"error","msg":"entryPoint \"websecure\" doesn't exist","routerName":"grafana_https@file","time":"2025-05-16T06:38:49Z"}
traefik                        | {"level":"error","msg":"no valid entryPoint for this router","routerName":"grafana_https@file","time":"2025-05-16T06:38:49Z"}
traefik                        | {"entryPointName":"websecure","level":"error","msg":"entryPoint \"websecure\" doesn't exist","routerName":"grafana_https@file","time":"2025-05-16T06:38:49Z"}
traefik                        | {"level":"error","msg":"no valid entryPoint for this router","routerName":"grafana_https@file","time":"2025-05-16T06:38:49Z"}
cvat_opa                       | {"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.29.0.17:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2025-05-16T06:38:49Z"}
cvat_server                    | Operations to perform:
cvat_server                    |   Apply all migrations: account, admin, auth, authtoken, consensus, contenttypes, dataset_repo, db, django_rq, engine, iam, organizations, quality_control, sessions, sites, socialaccount, webhooks
cvat_server                    | Running migrations:
cvat_server                    |   No migrations to apply.
cvat_worker_analytics_reports  | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_analytics_reports  | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_analytics_reports  | waiting for Redis migrations to complete...
cvat_worker_import             | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_import             | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_import             | waiting for Redis migrations to complete...
cvat_worker_export             | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_export             | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_export             | waiting for Redis migrations to complete...
cvat_server                    | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_server                    | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_webhooks           | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_webhooks           | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_webhooks           | waiting for Redis migrations to complete...
cvat_worker_consensus          | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_consensus          | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_consensus          | waiting for Redis migrations to complete...
cvat_worker_chunks             | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_chunks             | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_chunks             | waiting for Redis migrations to complete...
cvat_utils                     | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_utils                     | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_utils                     | waiting for Redis migrations to complete...
cvat_worker_annotation         | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_annotation         | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_annotation         | waiting for Redis migrations to complete...
cvat_opa                       | {"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.29.0.17:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2025-05-16T06:38:52Z"}
cvat_worker_quality_reports    | Additional supervisor configs that will be included: 
cvat_worker_quality_reports    | Error: could not find config file supervisord/worker.quality_reports.conf
cvat_worker_quality_reports    | For help, use /usr/bin/supervisord -h
cvat_server                    | No migrations to apply
cvat_worker_analytics_reports  | Additional supervisor configs that will be included: 
cvat_worker_import             | Additional supervisor configs that will be included: 
cvat_worker_export             | Additional supervisor configs that will be included: 
cvat_worker_analytics_reports  | Error: could not find config file supervisord/worker.analytics_reports.conf
cvat_worker_analytics_reports  | For help, use /usr/bin/supervisord -h
cvat_worker_import             | Error: could not find config file supervisord/worker.import.conf
cvat_worker_import             | For help, use /usr/bin/supervisord -h
cvat_worker_export             | Error: could not find config file supervisord/worker.export.conf
cvat_worker_export             | For help, use /usr/bin/supervisord -h
cvat_worker_webhooks           | Additional supervisor configs that will be included: 
cvat_worker_webhooks           | Error: could not find config file supervisord/worker.webhooks.conf
cvat_worker_webhooks           | For help, use /usr/bin/supervisord -h
cvat_worker_consensus          | Additional supervisor configs that will be included: 
cvat_worker_consensus          | Error: could not find config file supervisord/worker.consensus.conf
cvat_worker_consensus          | For help, use /usr/bin/supervisord -h
cvat_utils                     | Additional supervisor configs that will be included: 
cvat_worker_chunks             | Additional supervisor configs that will be included: 
cvat_worker_annotation         | Additional supervisor configs that will be included: 
cvat_opa                       | {"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.29.0.17:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2025-05-16T06:38:56Z"}
cvat_utils                     | Error: could not find config file supervisord/utils.conf
cvat_utils                     | For help, use /usr/bin/supervisord -h
cvat_worker_chunks             | Error: could not find config file supervisord/worker.chunks.conf
cvat_worker_chunks             | For help, use /usr/bin/supervisord -h
cvat_worker_annotation         | Error: could not find config file supervisord/worker.annotation.conf
cvat_worker_annotation         | For help, use /usr/bin/supervisord -h
cvat_db                        | 2025-05-16 06:38:57.034 UTC [53] LOG:  incomplete startup packet
cvat_worker_quality_reports exited with code 0
cvat_server                    | Processing queue import...
cvat_server                    | Processing queue export...
cvat_server                    | Processing queue annotation...
cvat_server                    | Processing queue webhooks...
cvat_server                    | Processing queue notifications...
cvat_server                    | Processing queue quality_reports...
cvat_server                    | Processing queue cleaning...
cvat_server                    | Job clean_up_sessions is unchanged
cvat_server                    | Job cron_export_cache_directory_cleanup is unchanged
cvat_server                    | Job cron_tmp_directory_cleanup is unchanged
cvat_server                    | Processing queue chunks...
cvat_server                    | Processing queue consensus...
cvat_db                        | 2025-05-16 06:38:59.537 UTC [54] LOG:  incomplete startup packet
cvat_db                        | 2025-05-16 06:38:59.578 UTC [55] LOG:  incomplete startup packet
cvat_worker_analytics_reports exited with code 0
cvat_worker_export exited with code 0
cvat_db                        | 2025-05-16 06:39:00.028 UTC [56] LOG:  incomplete startup packet
cvat_worker_import exited with code 0
cvat_worker_quality_reports    | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_quality_reports    | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_quality_reports    | waiting for Redis migrations to complete...
cvat_server                    | 
cvat_server                    | 0 static files copied to '/home/django/static', 165 unmodified.
cvat_db                        | 2025-05-16 06:39:01.158 UTC [58] LOG:  incomplete startup packet
cvat_db                        | 2025-05-16 06:39:01.185 UTC [59] LOG:  incomplete startup packet
cvat_worker_webhooks exited with code 0
cvat_worker_consensus exited with code 0
cvat_server                    | wait-for-it: waiting for cvat_db:5432 without a timeout
cvat_server                    | wait-for-it: cvat_db:5432 is available after 0 seconds
cvat_server                    | waiting for migrations to complete...
cvat_db                        | 2025-05-16 06:39:01.735 UTC [60] LOG:  incomplete startup packet
cvat_db                        | 2025-05-16 06:39:01.985 UTC [61] LOG:  incomplete startup packet
cvat_db                        | 2025-05-16 06:39:02.121 UTC [62] LOG:  incomplete startup packet
cvat_db                        | 2025-05-16 06:39:02.224 UTC [63] LOG:  incomplete startup packet
cvat_worker_annotation exited with code 0
cvat_worker_chunks exited with code 0
cvat_utils exited with code 0
cvat_opa                       | {"level":"error","msg":"Bundle load failed: request failed: Get \"http://cvat-server:8080/api/auth/rules\": dial tcp 172.29.0.17:8080: connect: connection refused","name":"cvat","plugin":"bundle","time":"2025-05-16T06:39:03Z"}
cvat_worker_export             | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_export             | waiting for Redis migrations to complete...
cvat_worker_export             | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_analytics_reports  | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_analytics_reports  | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_analytics_reports  | waiting for Redis migrations to complete...
cvat_worker_import             | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_import             | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_import             | waiting for Redis migrations to complete...
cvat_worker_quality_reports    | Additional supervisor configs that will be included: 
cvat_worker_quality_reports    | Error: could not find config file supervisord/worker.quality_reports.conf
cvat_worker_quality_reports    | For help, use /usr/bin/supervisord -h
cvat_worker_webhooks           | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_webhooks           | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_webhooks           | waiting for Redis migrations to complete...
cvat_worker_consensus          | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_consensus          | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_consensus          | waiting for Redis migrations to complete...
cvat_server                    | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_server                    | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_server                    | waiting for Redis migrations to complete...
cvat_worker_chunks             | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_chunks             | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_chunks             | waiting for Redis migrations to complete...
cvat_worker_annotation         | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_worker_annotation         | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_worker_annotation         | waiting for Redis migrations to complete...
cvat_utils                     | wait-for-it: waiting for cvat_redis_inmem:6379 without a timeout
cvat_utils                     | wait-for-it: cvat_redis_inmem:6379 is available after 0 seconds
cvat_utils                     | waiting for Redis migrations to complete...
cvat_db                        | 2025-05-16 06:39:07.394 UTC [73] LOG:  incomplete startup packet
cvat_worker_quality_reports exited with code 0
cvat_worker_export             | Additional supervisor configs that will be included: 
cvat_worker_export             | Error: could not find config file supervisord/worker.export.conf
cvat_worker_export             | For help, use /usr/bin/supervisord -h
cvat_worker_import             | Additional supervisor configs that will be included: 
cvat_worker_analytics_reports  | Additional supervisor configs that will be included: 
cvat_worker_import             | Error: could not find config file supervisord/worker.import.conf
cvat_worker_import             | For help, use /usr/bin/supervisord -h
cvat_worker_analytics_reports  | Error: could not find config file supervisord/worker.analytics_reports.conf
cvat_worker_analytics_reports  | For help, use /usr/bin/supervisord -h
cvat_worker_webhooks           | Additional supervisor configs that will be included: 
cvat_worker_consensus          | Additional supervisor configs that will be included: 
cvat_worker_webhooks           | Error: could not find config file supervisord/worker.webhooks.conf
cvat_worker_webhooks           | For help, use /usr/bin/supervisord -h
cvat_worker_consensus          | Error: could not find config file supervisord/worker.consensus.conf
cvat_worker_consensus          | For help, use /usr/bin/supervisord -h
cvat_server                    | Additional supervisor configs that will be included: reusable/smokescreen.conf 
cvat_server                    | 2025-05-16 06:39:09,905 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
cvat_server                    | 2025-05-16 06:39:09,905 INFO Included extra file "/home/django/supervisord/reusable/supervisord.conf" during parsing
cvat_server                    | 2025-05-16 06:39:09,905 INFO Included extra file "/home/django/supervisord/reusable/smokescreen.conf" during parsing
cvat_server                    | 2025-05-16 06:39:09,907 INFO Creating socket unix:///tmp/uvicorn.sock
cvat_server                    | 2025-05-16 06:39:09,907 INFO Closing socket unix:///tmp/uvicorn.sock
cvat_server                    | 2025-05-16 06:39:09,909 INFO RPC interface 'supervisor' initialized
cvat_server                    | 2025-05-16 06:39:09,909 CRIT Server 'unix_http_server' running without any HTTP authentication checking
cvat_server                    | 2025-05-16 06:39:09,909 INFO supervisord started with pid 1
cvat_worker_annotation         | Additional supervisor configs that will be included: 
cvat_utils                     | Additional supervisor configs that will be included: 
cvat_worker_chunks             | Additional supervisor configs that will be included: 
cvat_worker_annotation         | Error: could not find config file supervisord/worker.annotation.conf
cvat_worker_annotation         | For help, use /usr/bin/supervisord -h
cvat_utils                     | Error: could not find config file supervisord/utils.conf
cvat_utils                     | For help, use /usr/bin/supervisord -h
cvat_worker_chunks             | Error: could not find config file supervisord/worker.chunks.conf
cvat_worker_chunks             | For help, use /usr/bin/supervisord -h

Environment

docker-server:~/docker-composes/cvat# docker-compose ps
NAME                            IMAGE                                       COMMAND                  SERVICE                         CREATED          STATUS                                  PORTS
cvat_clickhouse                 clickhouse/clickhouse-server:23.11-alpine   "/entrypoint.sh"         cvat_clickhouse                 26 minutes ago   Up 38 seconds                           8123/tcp, 9000/tcp, 9009/tcp
cvat_db                         postgres:15-alpine                          "docker-entrypoint.s…"   cvat_db                         26 minutes ago   Up 38 seconds                           5432/tcp
cvat_grafana                    grafana/grafana-oss:10.1.2                  "sh -euc 'mkdir -p /…"   cvat_grafana                    26 minutes ago   Up 37 seconds                           3000/tcp
cvat_opa                        openpolicyagent/opa:0.63.0                  "/opa run --server -…"   cvat_opa                        26 minutes ago   Up 38 seconds                           
cvat_redis_inmem                redis:7.2.3-alpine                          "docker-entrypoint.s…"   cvat_redis_inmem                26 minutes ago   Up 39 seconds                           6379/tcp
cvat_redis_ondisk               apache/kvrocks:2.7.0                        "kvrocks -c /var/lib…"   cvat_redis_ondisk               26 minutes ago   Up 38 seconds (healthy)                 6666/tcp
cvat_server                     cvat/server:dev                             "./backend_entrypoin…"   cvat_server                     26 minutes ago   Up 34 seconds                           8080/tcp
cvat_ui                         cvat/ui:dev                                 "/docker-entrypoint.…"   cvat_ui                         26 minutes ago   Up 32 seconds                           80/tcp
cvat_utils                      cvat/server:dev                             "./backend_entrypoin…"   cvat_utils                      26 minutes ago   Up 8 seconds                            8080/tcp
cvat_vector                     timberio/vector:0.26.0-alpine               "/usr/local/bin/vect…"   cvat_vector                     26 minutes ago   Up 36 seconds                           
cvat_worker_analytics_reports   cvat/server:dev                             "./backend_entrypoin…"   cvat_worker_analytics_reports   26 minutes ago   Up 8 seconds                            8080/tcp
cvat_worker_annotation          cvat/server:dev                             "./backend_entrypoin…"   cvat_worker_annotation          26 minutes ago   Up 6 seconds                            8080/tcp
cvat_worker_chunks              cvat/server:dev                             "./backend_entrypoin…"   cvat_worker_chunks              26 minutes ago   Up 5 seconds                            8080/tcp
cvat_worker_consensus           cvat/server:dev                             "./backend_entrypoin…"   cvat_worker_consensus           26 minutes ago   Up 6 seconds                            8080/tcp
cvat_worker_export              cvat/server:dev                             "./backend_entrypoin…"   cvat_worker_export              26 minutes ago   Up Less than a second                   8080/tcp
cvat_worker_import              cvat/server:dev                             "./backend_entrypoin…"   cvat_worker_import              26 minutes ago   Up 5 seconds                            8080/tcp
cvat_worker_quality_reports     cvat/server:dev                             "./backend_entrypoin…"   cvat_worker_quality_reports     26 minutes ago   Restarting (2) Less than a second ago   
cvat_worker_webhooks            cvat/server:dev                             "./backend_entrypoin…"   cvat_worker_webhooks            26 minutes ago   Up 2 seconds                            8080/tcp
nuclio                          quay.io/nuclio/dashboard:1.13.0-amd64       "/docker-entrypoint.…"   nuclio                          4 weeks ago      Up 2 days (healthy)                     80/tcp, 0.0.0.0:8070->8070/tcp, :::8070->8070/tcp
traefik                         traefik:v2.10                               "/entrypoint.sh trae…"   traefik                         26 minutes ago   Up 39 seconds                           0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 80/tcp, 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp
@hhhfsj hhhfsj added the bug Something isn't working label May 16, 2025
@azhavoro azhavoro removed the bug Something isn't working label May 16, 2025
@azhavoro
Copy link
Contributor

you need to update your docker compose configuration
I would also recommend using release versions instead of dev to avoid such errors

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@azhavoro @hhhfsj
0