modules using log redirects break when using conda #8455
Comments
|
Minimal example:
`main.nfprocess EXAMPLE
{
input:
val message
output:
path('*.log')
script:
"""
(
echo "error stream ${message}" >&2;
echo "output stream: ${message}";
) 2> >(tee output.log >&2);
"""
}
workflow {
Channel.of('Bonjour', 'Ciao', 'Hello', 'Hola') | EXAMPLE | view
}`nextflow.config`output.log./nextflow run main.nf
N E X T F L O W ~ version 25.04.1
Launching `main.nf` [confident_pike] DSL2 - revision: 9def87aeff
executor > local (4)
ERROR ~ Error executing process > 'EXAMPLE (3)'
Caused by:
Process `EXAMPLE (3)` terminated with an error exit status (1)
Command executed:
(
echo "error stream Hello" >&2;
echo "output stream: Hello";
) 2> >(tee output.log >&2);
Command exit status:
1
Command output:
(empty)
Command error:
.command.sh: line 5: /dev/fd/63: cannot overwrite existing file
Work dir:
/path/to/nextflow/minimal/work/bc/f16f1ca4ed7437717c3807417c7866
Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
-- Check '.nextflow.log' file for details |
|
I think you're right. It's to do with It's the first time I've seen noclobber interacting like that though, particularly as it's only happening in a conda env. Why is it succeeding in the docker container? |
|
And why haven’t others encountered this when -C Has been part of the template for a while now? |
|
🤷🏽 Very few people running pipelines on macs with conda perhaps. I think many use docker on mac. |
|
Ok, so it's bash vs zsh. $ bash
$ set -C
$ ( echo "error stream Ciao" >&2; echo "output stream: Ciao"; ) 2> >(tee output.log >&2);
bash: /dev/fd/63: cannot overwrite existing filewhere as for zsh it prints as expected: $ zsh
$ ( echo "error stream Ciao" >&2; echo "output stream: Ciao"; ) 2> >(tee output.log >&2);
output stream: Ciao
error stream Ciaohmmn |
|
bash fails with a simplified substitution: $ bash --version
GNU bash, version 5.2.37(1)-release (aarch64-apple-darwin24.0.0)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ ( echo "error" >&2 ) 2> >(cat)
bash: /dev/fd/63: cannot overwrite existing file |
|
Using |
|
Fails with bash installed with conda too. I also found someone else having the same problem: https://unix.stackexchange.com/questions/554843/how-to-allow-process-substitution-when-noclobber-is-set. So I'd recommend the solution would be to update all redirects to use the |
|
Yes, that sounds good. Thanks for investigating further. |
Have you checked the docs?
Description of the bug
Processes that include lines like:
2> >(tee output.log >&2)break if run with conda. Specifically discovered when working withfastp.Switching them to
2>| >(tee output.log >&2)will fix the issue.Command used and terminal output
Relevant files
fulcrumgenomics-example.zip
System information
Nextflow version: 24.10.6
nf-core/tools version: 3.2.1
Hardware: M3 mac
Executor: local
Container engine: conda 24.11.3
OS: macOS 14.5
The text was updated successfully, but these errors were encountered: