Tags: Lightning-AI/LitServe
Tags
created shutdown endpoint with API key security and custom passed tes… …ts (#525) * adding shutdown endpoint * adding security to shutdown endpoint * adding security to shutdown endpoint pt 2 * adding tests and changing to bearer token * making new litserve test wrapper * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * upd * upd * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * linter * running CI again * adding ValueError if API key not configured * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * linter * rewriting value error and changing env variable * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Apply suggestions from code review * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Enhance shutdown API key handling and logging. Generate a default shutdown API key if not set, and provide a curl command for server shutdown in logs. Update logging levels for better traceability during server operations. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * Refactor shutdown logic in LitServer for improved clarity and error handling. Updated HTTP status codes for shutdown endpoint responses and enhanced logging for worker termination processes, including descriptive names for better traceability. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor shutdown process in LitServer to improve clarity and error handling. Enhanced logging for Uvicorn worker termination and removed redundant code. Updated curl command formatting for better readability. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor LitServer shutdown process to streamline management handling and improve clarity. Removed unused attributes and consolidated manager initialization. Updated shutdown method to accept worker lists directly for better flexibility during termination. * update * revert * Enhance LitServer shutdown process to handle different shutdown reasons. Updated _perform_graceful_shutdown method to accept a shutdown_reason parameter, allowing for conditional transport closure. Removed unused test_litserve_shutdown context manager and adjusted tests to use wrap_litserve_start for improved clarity and consistency. * changing test API key for shutdown testing due to collision with the shutdown API key environment variable * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor manager initialization in LitServer to simplify code. Updated tests to include a mock manager for improved isolation and reliability during server run tests. * update * fix tests * fix * update * fix ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Aniket Maurya <theaniketmaurya@gmail.com> Co-authored-by: Jirka B <j.borovec+github@gmail.com>
Improve developer experience (#519) * Enhance LitAPI and LitServer initialization documentation - Updated docstrings for LitAPI and LitServer constructors to provide detailed descriptions of parameters and their usage. - Improved clarity on deprecated parameters and their replacements, ensuring better guidance for users. - Added examples for instantiating LitAPI and LitServer, demonstrating various configurations for enhanced usability. * bumo * fix tests
Enable true concurrency in async streaming loop and add tests (#488) * process tasks as async tasks * Add delay in streaming async loop to prevent CPU spinning * add test_async_stream_litapi * add tests for concurrent asycn streaming * update time * update * Update src/litserve/loops/streaming_loops.py * fix test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * trigger ci --------- Co-authored-by: Aniket Maurya <theaniketmaurya@gmail.com> Co-authored-by: Aniket Maurya <aniket@grid.ai> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
enable asynchronous processing in LitAPI (#477) * feat: enable asynchronous processing in LitAPI and update request handling - Added `enable_async` parameter to `LitAPI` constructor to support async methods. - Implemented `__post_init__` to validate coroutine methods when async support is enabled. - Introduced `_async_inject_context` for async context injection. - Enhanced `SingleLoop` to handle requests asynchronously with `_run_single_loop_with_async`. - Updated `_process_single_request` to manage async request processing and error handling. * add tests * update * refactor: enhance get_default_loop to support async processing - Updated `get_default_loop` function to include `enable_async` parameter for better async handling. - Added validation for async streaming and batching conditions. - Adjusted calls to `get_default_loop` in `LitServer` and `inference_worker` to pass the new parameter. * fix tests
PreviousNext