8000 [molecule] retry if galaxy access fails · Issue #8402 · kiali/kiali · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[molecule] retry if galaxy access fails #8402

New issue

Have a question about this project? Sign up for a free GitHub account to op 8000 en an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jmazzitelli opened this issue May 8, 2025 · 3 comments · Fixed by #8447
Closed

[molecule] retry if galaxy access fails #8402

jmazzitelli opened this issue May 8, 2025 · 3 comments · Fixed by #8447
Assignees
@jmazzitelli
Labels
enhancement This is the preferred way to describe new end-to-end features.

Comments

@jmazzitelli
Copy link
Collaborator
jmazzitelli commented May 8, 2025 •
edited
Loading

Sometimes a molecule test run fails because it cannot access galaxy, which is usually only a temporary problem (I've seen the next test run fine).

So it would be nice to get whatever is doing this to retry at least a few times to see if it can workaround the temp failure - that would help solve the issue around false test failures.

This is what the logs look like when this happens:

/bin/bash: line 1: go: command not found
Pulling down latest helm-charts from remote repo to here: /home/jmazzite/source/kiali/helm-charts
Git repo for the helm charts is found here: /home/jmazzite/source/kiali/helm-charts
make[1]: Entering directory '/home/jmazzite/source/kiali/helm-charts'
Will use this helm executable: /home/jmazzite/source/kiali/helm-charts/_output/helm-install/helm
Building Helm Chart for Kiali operator
==> Linting /home/jmazzite/source/kiali/helm-charts/_output/charts/kiali-operator

1 chart(s) linted, 0 chart(s) failed
Successfully packaged chart and saved it to: /home/jmazzite/source/kiali/helm-charts/_output/charts/kiali-operator-2.10.0-SNAPSHOT.tgz
make[1]: Leaving directory '/home/jmazzite/source/kiali/helm-charts'
Minikube repos: external=[192.168.39.167:5000] internal=[localhost:5000]
| registry                    | ci      | enabled ✅   | minikube                       |
Will not auto-detect any hosts for non-OpenShift clusters.
Will not rebuild kiali-molecule image.
podman volume exists molecule-tests-volume && echo "Podman volume already exists; deleting it" && podman volume rm molecule-tests-volume || true
podman volume create molecule-tests-volume
molecule-tests-volume
podman create -v molecule-tests-volume:/data --name molecule-volume-helper quay.io/fedora/fedora-minimal:latest sleep infinity
8592dde204a00d18618c502c07bbf60fd5c204f08cee866e179724a81542beac
podman start molecule-volume-helper
molecule-volume-helper
podman run --rm -v molecule-tests-volume:/data -v "/home/jmazzite/source/kiali/helm-charts":/source:ro,Z quay.io/fedora/fedora-minimal cp -r /source /data/helm-charts-repo
podman run --rm -v molecule-tests-volume:/data -v "$(realpath /home/jmazzite/source/kiali/kiali/operator)":/source:ro,Z quay.io/fedora/fedora-minimal cp -r /source /data/operator
podman run --rm -v molecule-tests-volume:/data -v "/home/jmazzite/.kube/config":/source:ro,Z quay.io/fedora/fedora-minimal cp -r /source /data/kubeconfig
podman stop molecule-volume-helper
time="2025-05-08T01:02:59-04:00" level=warning msg="StopSignal SIGTERM failed to stop container molecule-volume-helper in 10 seconds, resorting to SIGKILL"
molecule-volume-helper
podman rm molecule-volume-helper
molecule-volume-helper
for msn in accessible-namespaces-test; do podman run --rm  --env KUBECONFIG="/tmp/molecule/kubeconfig" --env K8S_AUTH_KUBECONFIG="/tmp/molecule/kubeconfig" --env MOLECULE_CLUSTER_TYPE="minikube" --env MOLECULE_HELM_CHARTS_REPO=/tmp/molecule/helm-charts-repo -v molecule-tests-volume:/tmp/molecule -v /home/jmazzite/.minikube/logs/lastStart.txt:/home/jmazzite/.minikube/logs/lastStart.txt:Z -v /home/jmazzite/.minikube/logs/audit.json:/home/jmazzite/.minikube/logs/audit.json:Z -v /home/jmazzite/.minikube/certs/ca.pem:/home/jmazzite/.minikube/certs/ca.pem:Z -v /home/jmazzite/.minikube/certs/ca-key.pem:/home/jmazzite/.minikube/certs/ca-key.pem:Z -v /home/jmazzite/.minikube/certs/cert.pem:/home/jmazzite/.minikube/certs/cert.pem:Z -v /home/jmazzite/.minikube/certs/key.pem:/home/jmazzite/.minikube/certs/key.pem:Z -v /home/jmazzite/.minikube/machines/server.pem:/home/jmazzite/.minikube/machines/server.pem:Z -v /home/jmazzite/.minikube/machines/server-key.pem:/home/jmazzite/.minikube/machines/server-key.pem:Z -v /home/jmazzite/.minikube/machines/ci/id_rsa:/home/jmazzite/.minikube/machines/ci/id_rsa:Z -v /home/jmazzite/.minikube/machines/ci/id_rsa.pub:/home/jmazzite/.minikube/machines/ci/id_rsa.pub:Z -v /home/jmazzite/.minikube/machines/ci/config.json:/home/jmazzite/.minikube/machines/ci/config.json:Z -v /home/jmazzite/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.31.0-docker-overlay2-amd64.tar.lz4:/home/jmazzite/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.31.0-docker-overlay2-amd64.tar.lz4:Z -v /home/jmazzite/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.31.0-docker-overlay2-amd64.tar.lz4.checksum:/home/jmazzite/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.31.0-docker-overlay2-amd64.tar.lz4.checksum:Z -v /home/jmazzite/.minikube/cache/linux/amd64/v1.31.0/kubectl:/home/jmazzite/.minikube/cache/linux/amd64/v1.31.0/kubectl:Z -v /home/jmazzite/.minikube/config/config.json:/home/jmazzite/.minikube/config/config.json:Z -v /home/jmazzite/.minikube/profiles/ci/events.json:/home/jmazzite/.minikube/profiles/ci/events.json:Z -v /home/jmazzite/.minikube/profiles/ci/client.crt:/home/jmazzite/.minikube/profiles/ci/client.crt:Z -v /home/jmazzite/.minikube/profiles/ci/client.key:/home/jmazzite/.minikube/profiles/ci/client.key:Z -v /home/jmazzite/.minikube/profiles/ci/apiserver.crt.523f01b8:/home/jmazzite/.minikube/profiles/ci/apiserver.crt.523f01b8:Z -v /home/jmazzite/.minikube/profiles/ci/apiserver.key.523f01b8:/home/jmazzite/.minikube/profiles/ci/apiserver.key.523f01b8:Z -v /home/jmazzite/.minikube/profiles/ci/apiserver.crt:/home/jmazzite/.minikube/profiles/ci/apiserver.crt:Z -v /home/jmazzite/.minikube/profiles/ci/apiserver.key:/home/jmazzite/.minikube/profiles/ci/apiserver.key:Z -v /home/jmazzite/.minikube/profiles/ci/proxy-client.crt:/home/jmazzite/.minikube/profiles/ci/proxy-client.crt:Z -v /home/jmazzite/.minikube/profiles/ci/proxy-client.key:/home/jmazzite/.minikube/profiles/ci/proxy-client.key:Z -v /home/jmazzite/.minikube/profiles/ci/config.json:/home/jmazzite/.minikube/profiles/ci/config.json:Z -v /home/jmazzite/.minikube/last_update_check:/home/jmazzite/.minikube/last_update_check:Z -v /home/jmazzite/.minikube/bin/docker-machine-driver-kvm2:/home/jmazzite/.minikube/bin/docker-machine-driver-kvm2:Z -v /home/jmazzite/.minikube/ca.crt:/home/jmazzite/.minikube/ca.crt:Z -v /home/jmazzite/.minikube/ca.key:/home/jmazzite/.minikube/ca.key:Z -v /home/jmazzite/.minikube/proxy-client-ca.crt:/home/jmazzite/.minikube/proxy-client-ca.crt:Z -v /home/jmazzite/.minikube/proxy-client-ca.key:/home/jmazzite/.minikube/proxy-client-ca.key:Z -v /home/jmazzite/.minikube/ca.pem:/home/jmazzite/.minikube/ca.pem:Z -v /home/jmazzite/.minikube/cert.pem:/home/jmazzite/.minikube/cert.pem:Z -v /home/jmazzite/.minikube/key.pem:/home/jmazzite/.minikube/key.pem:Z --env MOLECULE_MINIKUBE_IP=192.168.39.167  -w /tmp/molecule/operator --network="host"  --env DORP=podman --env OPERATOR_IMAGE_PULL_SECRET_NAME= --env MOLECULE_KIALI_CR_SPEC_VERSION=default --env PLUGIN_IMAGE_PULL_SECRET_JSON="" --env MOLECULE_OSSMCONSOLE_CR_SPEC_VERSION="default"  --env MOLECULE_OPERATOR_PROFILER_ENABLED=true --env MOLECULE_OPERATOR_INSTALLER=helm --env MOLECULE_DUMP_LOGS_ON_ERROR=true  --env MOLECULE_WAIT_RETRIES=120 localhost/kiali-molecule:latest molecule --debug test  --scenario-name ${msn}; if [ "$?" != "0" ]; then echo "Molecule test failed: ${msn}"; podman volume rm molecule-tests-volume; exit 1; fi; done
DEBUG    Validating schema /tmp/molecule/operator/molecule/accessible-namespaces-test/molecule.yml.
WARNING  Driver podman does not provide a schema.
INFO     accessible-namespaces-test scenario test matrix: prepare, converge, destroy
INFO     Performing prerun with role_name_check=0...
INFO     Running ansible-galaxy collection install -v -r requirements.yml
ERROR    Starting galax
8000
y collection install process
Process install dependency map
ERROR    No config file found; using defaults
[WARNING]: Collection redhatinsights.insights does not support Ansible version
2.15.3
[WARNING]: Skipping Galaxy server https://galaxy.ansible.com/api/. Got an
unexpected error when getting available versions of collection
community.general: Unknown error when attempting to call Galaxy at 'https://gal
axy.ansible.com/api/v3/collections/community/general/versions/9.0.0/': <urlopen
error [Errno -3] Temporary failure in name resolution>. <urlopen error [Errno
-3] Temporary failure in name resolution>
ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/v3/collections/community/general/versions/9.0.0/': <urlopen error [Errno -3] Temporary failure in name resolution>. <urlopen error [Errno -3] Temporary failure in name resolution>

Traceback (most recent call last):
  File "/usr/local/bin/molecule", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/molecule/command/test.py", line 113, in test
    base.execute_cmdline_scenarios(scenario_name, args, command_args, ansible_args)
  File "/usr/local/lib/python3.11/site-packages/molecule/command/base.py", line 113, in execute_cmdline_scenarios
    scenario.config.runtime.prepare_environment(
  File "/usr/local/lib/python3.11/site-packages/ansible_compat/runtime.py", line 609, in prepare_environment
    self.install_requirements(Path(req_file), retry=retry, offline=offline)
  File "/usr/local/lib/python3.11/site-packages/ansible_compat/runtime.py", line 588, in install_requirements
    raise AnsibleCommandError(result)
ansible_compat.errors.AnsibleCommandError: Got 1 exit code while running: ansible-galaxy collection install -v -r requirements.yml
Molecule test failed: accessible-namespaces-test
molecule-tests-volume
make: *** [make/Makefile.molecule.mk:188: molecule-test] Error 1
@jmazzitelli jmazzitelli added the enhancement This is the preferred way to describe new end-to-end features. label May 8, 2025
@jmazzitelli
Copy link
Collaborator Author

This is really annoying - it happened with the latest CI test (again) - I'm going to try to fix this.

@jmazzitelli jmazzitelli self-assigned this May 21, 2025
@jmazzitelli jmazzitelli moved this from 📋 Backlog to 🏗 In progress in Kiali Sprint 25-08 | Kiali v2.11 May 21, 2025
@jmazzitelli
Copy link
Collaborator Author

And of course nothing is every easy:

ansible_compat.errors.AnsibleCommandError: Got 1 exit code while running: ansible-galaxy collection install -v -r requirements.yml

means ansible is capturing the exit code and not passing it back up - we are losing the real error code. Looks like I might have to examine the output and through some basic heuristics determine if the error is fatal or transient.

@jmazzitelli
Copy link
Collaborator Author

Here is another run with a failure - the error message is slightly different but still Galaxy was inaccessible:

Will not auto-detect any hosts for non-OpenShift clusters.
Will not rebuild kiali-molecule image.
Docker is not supported for running the molecule tests. Ignoring 'dorp=docker' and using podman.
podman volume exists molecule-tests-volume && echo "Podman volume already exists; deleting it" && podman volume rm molecule-tests-volume || true
podman volume create molecule-tests-volume
molecule-tests-volume
podman create -v molecule-tests-volume:/data --name molecule-volume-helper quay.io/fedora/fedora-minimal:latest sleep infinity
8c19fe95255a526d1d8a8dd58be598f8f20bb874176d2d34526e3dfb195644c8
podman start molecule-volume-helper
molecule-volume-helper
podman run --rm -v molecule-tests-volume:/data -v "/tmp/KIALI-GIT-KIND/helm-charts":/source:ro,Z quay.io/fedora/fedora-minimal cp -r /source /data/helm-charts-repo
podman run --rm -v molecule-tests-volume:/data -v "$(realpath /tmp/KIALI-GIT-KIND/kiali/operator)":/source:ro,Z quay.io/fedora/fedora-minimal cp -r /source /data/operator
podman run --rm -v molecule-tests-volume:/data -v "/home/runner/.kube/config":/source:ro,Z quay.io/fedora/fedora-minimal cp -r /source /data/kubeconfig
podman stop molecule-volume-helper
time="2025-05-21T06:22:19Z" level=warning msg="StopSignal SIGTERM failed to stop container molecule-volume-helper in 10 seconds, resorting to SIGKILL"
molecule-volume-helper
podman rm molecule-volume-helper
molecule-volume-helper
Docker is not supported for running the molecule tests. Ignoring 'dorp=docker' and using podman.
for msn in remote-cluster-resources-test; do podman run --rm  --env KUBECONFIG="/tmp/molecule/kubeconfig" --env K8S_AUTH_KUBECONFIG="/tmp/molecule/kubeconfig" --env MOLECULE_CLUSTER_TYPE="kind" --env MOLECULE_HELM_CHARTS_REPO=/tmp/molecule/helm-charts-repo -v molecule-tests-volume:/tmp/molecule   --env MOLECULE_KIND_IP=172.18.0.2 -w /tmp/molecule/operator --network="host"  --env DORP=podman --env OPERATOR_IMAGE_PULL_SECRET_NAME= --env MOLECULE_KIALI_CR_SPEC_VERSION=default --env PLUGIN_IMAGE_PULL_SECRET_JSON="" --env MOLECULE_OSSMCONSOLE_CR_SPEC_VERSION="default"  --env MOLECULE_OPERATOR_PROFILER_ENABLED=true --env MOLECULE_OPERATOR_INSTALLER=skip --env MOLECULE_DUMP_LOGS_ON_ERROR=true  --env MOLECULE_WAIT_RETRIES=120 localhost/kiali-molecule:latest molecule --debug test  --scenario-name ${msn}; if [ "$?" != "0" ]; then echo "Molecule test failed: ${msn}"; podman volume rm molecule-tests-volume; exit 1; fi; done
DEBUG    Validating schema /tmp/molecule/operator/molecule/remote-cluster-resources-test/molecule.yml.
WARNING  Driver podman does not provide a schema.
INFO     remote-cluster-resources-test scenario test matrix: prepare, converge, destroy
INFO     Performing prerun with role_name_check=0...
INFO     Running ansible-galaxy collection install -v -r requirements.yml
ERROR    Starting galaxy collection install process
Process install dependency map
to see the full traceback, use -vvv

ERROR    No config file found; using defaults
Warning: : Collection redhatinsights.insights does not support Ansible version
2.15.3
Warning: : Skipping Galaxy server [https://galaxy.ansible.com/api/.](https://galaxy.ansible.com/api/) Got an
unexpected error when getting available versions of collection
community.general: 'results'
ERROR! Unexpected Exception, this is probably a bug: 'results'

Traceback (most recent call last):
  File "/usr/local/bin/molecule", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/molecule/command/test.py", line 113, in test
    base.execute_cmdline_scenarios(scenario_name, args, command_args, ansible_args)
  File "/usr/local/lib/python3.11/site-packages/molecule/command/base.py", line 113, in execute_cmdline_scenarios
    scenario.config.runtime.prepare_environment(
  File "/usr/local/lib/python3.11/site-packages/ansible_compat/runtime.py", line 609, in prepare_environment
    self.install_requirements(Path(req_file), retry=retry, offline=offline)
  File "/usr/local/lib/python3.11/site-packages/ansible_compat/runtime.py", line 588, in install_requirements
    raise AnsibleCommandError(result)
ansible_compat.errors.AnsibleCommandError: Got 250 exit code while running: ansible-galaxy collection install -v -r requirements.yml
Molecule test failed: remote-cluster-resources-test
molecule-tests-volume
make: *** [make/Makefile.molecule.mk:186: molecule-test] Error 1

jmazzitelli added a commit to jmazzitelli/kiali that referenced this issue May 21, 2025
@jmazzitelli
retry a molecule test that failed when it appears the failure was due…
7e664c0 
… to Galaxy server being down

fixes: kiali#8402
@jmazzitelli jmazzitelli mentioned this issue May 21, 2025
Merged
@jmazzitelli jmazzitelli moved this from 🏗 In progress to 👀 In review in Kiali Sprint 25-08 | Kiali v2.11 May 21, 2025
jmazzitelli added a commit that referenced this issue May 21, 2025
@jmazzitelli
retry a molecule test that failed when it appears the failure was due…
0e99de0 
… to Galaxy server being down (#8447)

fixes: #8402
@github-project-automation github-project-automation bot moved this from 👀 In review to ✅ Done in Kiali Sprint 25-08 | Kiali v2.11 May 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jmazzitelli jmazzitelli

Labels
enhancement This is the preferred way to describe new end-to-end features.
Projects
Kiali Sprint 25-08 | Kiali v2.11
Status: ✅ Done
Milestone
No milestone
1 participant
@jmazzitelli
0