Open
Description
What happened + What you expected to happen
When streaming log from Ray Job, it fails to print the last log statement. The issue seems to be introduced in Ray 2.12 and continue to persist in 2.31. It works fine in Ray 2.10.
In the following example, it never prints the line I'm Ray
Versions / Dependencies
Ray 2.31
Python 3.11
Linux
Reproduction script
# hello.py
import os,time
print('hello')
time.sleep(2)
print("world")
time.sleep(2)
print("I'm Ray")
# submit.py
from ray.job_submission import JobSubmissionClient
client = JobSubmissionClient("http://127.0.0.1:8265")
job_id = client.submit_job(
# Entrypoint shell command to execute
entrypoint="python hello.py",
# Path to the local directory that contains the script.py file
runtime_env={"working_dir": "./"}
)
print(job_id)
logs_f = client.tail_job_logs(job_id)
async def test():
async for line in logs_f:
print(line)
import asyncio
asyncio.run(test())
ray start --head
python submit.pyIssue Severity
Medium: It is a significant difficulty but I can work around it.