Error: No route to host #63

Viren-tyagi · 2025-01-09T13:47:25Z

Viren-tyagi
Jan 9, 2025

I am trying to submit psiflow jobs to an HPC cluster but get a No route to host error, even though psiflow is installed properly.

This is the submit script

#!/bin/bash

#SBATCH --job-name=parsl.ModelEvaluation.block-0.1736412340.5722115
#SBATCH --output=/gpfs/home3/vtyagi/test-psiflow/psiflow_internal/000/submit_scripts/parsl.ModelEvaluation.block-0.1736412340.5722115.stdout
#SBATCH --error=/gpfs/home3/vtyagi/test-psiflow/psiflow_internal/000/submit_scripts/parsl.ModelEvaluation.block-0.1736412340.5722115.stderr
#SBATCH --nodes=1
#SBATCH --time=600
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:1

#SBATCH --partition=gpu_h100
#SBATCH --cpus-per-task=64

source $HOME/setup-psiflow/activate.sh
export PARSL_CORES=64


export JOBNAME="parsl.ModelEvaluation.block-0.1736412340.5722115"

 work_queue_worker --parent-death --wall-time=36000 --cores=64 --idle-timeout=20 gcn142.local.snellius.surf.nl 1024

This is what is written out in the stdout file

work_queue_worker: creating workspace /scratch-local/vtyagi.9277551/worker-65553-2687753
work_queue_worker: using 64 cores, 773737 MB memory, 2266620048 MB disk, 4 gpus
work_queue_worker: deleting workspace /gpfs/scratch1/nodespecific/gcn148/vtyagi.9277551/worker-65553-2687753

JOB STATISTICS
==============
Job ID: 9277551
Cluster: snellius
User/Group: vtyagi/vtyagi
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 64
CPU Utilized: 00:00:01
CPU Efficiency: 0.00% of 16:18:08 core-walltime
Job Wall-clock time: 00:15:17
Memory Utilized: 3.26 MB
Memory Efficiency: 0.00% of 720.00 GB

This is what is written out in the stderr file

couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
couldn't connect to 172.18.62.142:1024: No route to host
2025/01/09 10:01:09.34 work_queue_worker[2687753] notice: stopping: could not connect after 900 seconds.

pdobbelaere · 2025-03-14T15:25:01Z

pdobbelaere
Mar 14, 2025
Maintainer

By default, psiflow does not choose a communication port for its work_queue_workers (port=0), leaving it up to workqueue to select some reasonable value (see). This does not work for every HPC depending on system (network) configuration.

I can manually control the port by setting the WORK_QUEUE_PORT environment variable in the shell that runs the main parsl process. Could you check whether this works on your end as well?

Alternatively, we could modify the psiflow config to accept a port argument, which would only be a minor adaptation.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Error: No route to host #63

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Error: No route to host #63

Uh oh!

Viren-tyagi Jan 9, 2025

Replies: 1 comment

Uh oh!

pdobbelaere Mar 14, 2025 Maintainer

Viren-tyagi
Jan 9, 2025

pdobbelaere
Mar 14, 2025
Maintainer