Open
Description
What happened + What you expected to happen
Steps to reproduce:
- serve run dummy:transformer
- curl localhost:8000 ----> succeeds
- Ctrl-C (1) and run it again
- curl localhost:8000 ----> Connection refused
There is no way to recover now except to run "serve shutdown".
You can reproduce this on this workspace: https://console.anyscale.com/o/anyscale-internal/workspaces/expwrk_648fzyx25qt5ih54ylfjfybl2d/ses_hg15be2l74pppi8f42nzflpwi9?command-history-section=command_history
It is not reproducible on my laptop, only that workspace.
Versions / Dependencies
Ray 2.2
Reproduction script
from starlette.requests import Request
import ray
from ray import serve
import argparse
from os import path
@serve.deployment(num_replicas=2, ray_actor_options={"num_cpus": 2})
class Transformer:
def __init__(self, model_dir):
print("Loading", model_dir)
self.model_dir = model_dir
def complete(self, text):
return self.model_dir + " " + text
async def __call__(self, http_request: Request) -> str:
prompt: str = {"text": "foo"}
return self.complete(prompt["text"])
transformer = Transformer.bind("dummy")
You can use this or any other dummy script.
Issue Severity
Medium: It is a significant difficulty but I can work around it.