From 5d4d01491abec8ef1e6f45578c25ad037e9818d3 Mon Sep 17 00:00:00 2001 From: Brian Gunnarson <49216024+bgunnar5@users.noreply.github.com> Date: Thu, 12 Jun 2025 17:43:23 -0700 Subject: [PATCH 1/2] Version/1.13.0b1 (#512) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * remove github text that was causing errors * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning Co-authored-by: Joe Koning * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning Co-authored-by: Joe Koning * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks * Fix filenames for OpenFoam tutorial (#475) * bugfix/deep-merge-existing-keys (#476) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * remove DeepMergeException and add conflict_handler to dict_deep_merge * add conflict handler to dict_deep_merge * fix broken tests for detailed-status * use caplog fixture rather than IO stream * add ability to define module-specific fixtures * add tests for read/write status files and conlict handling * add caplog explanation to docstrings * update CHANGELOG * run fix-style * add pytest-mock as dependency for test suite * clean up input check in dict_deep_merge * Improved Info (#477) * Add merlin version to banner * Add python package info to and clean up 'merlin info' * Add some unit tests * Force GitHub runner checkout to grab the whole history, fixing CHANGELOG test bug * Update CHANGELOG to show bugfix to CHANGELOG test * Target is in source's history (#478) * New github action test to make sure target has been merged into source * Fix link to merlin banner image (#479) * bugfix/status_nested_workspace (#480) * remove a merge conflict statement that was missed * have status ignore nested workspaces and modify merge rules * update CHANGELOG * fixed issue with escape sequences in ascii art * apply Luc's suggestion * add setuptools as a requirement since python 3.12 doesn't have it natively * modify unit tests for status to use pytest rather than unittest * update CHANGELOG * add fixtures for status testing and add nested workflow test * update CHANGELOG * bugfix/celery-chord-error (#481) * remove a merge conflict statement that was missed * add celery results backend patch to stop ChordErrors * add MERLIN_RAISE_ERROR return code * add tests to ensure chord error isn't raised * add RAISE_ERROR to docs * update CHANGELOG * fix lint issues * up the sleep time on the chord error test * add new steps to the chord err test spec * add tree statement to the new test for debugging * upping sleep time to see if that fixes github action for python 3.7 * change sleep time for new test based on python version * run fix style * remove specific sleep time for diff python versions * release/1.12.2b1 (#482) * remove a merge conflict statement that was missed * bump version to 1.12.2b1 * bugfix/flux-nodes (#484) * remove a merge conflict statement that was missed * fix flux node allocation issue * allow for vars to be used with nodes settings of workers/batch * add tests for var usage with nodes * update CHANGELOG * run fix-style * bugfix/flux-nodes-prior-versions (#487) * add a version check for flux when getting node count * update CHANGELOG * add major version check for flux * Change Task ID to directory path (#486) * Modifying task id to include directory * Adding Several New Unit Tests (#490) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * split up create_server_config and write tests for it * add tests for config_merlin_server function * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * update CHANGELOG * add Ryan and Joe's suggestions * update tests to use newly named functions * fix linter issue * release/1.12.2 (#491) * update version to 1.12.2 * fix style issue * Refactor/distributed-tests (#493) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * Include celerymanager and update celeryadapter to check the status of celery workers. * Fixed issue where the update status was outside of if statement for checking workers * Include worker status stop and add template for merlin restart * Added comment to the CeleryManager init * Increment db_num instead of being fixed * Added other subprocess parameters and created a linking system for redis to store env dict * Implemented stopping of celery workers and restarting workers properly * Update stopped to stalled for when the worker doesn't respond to restart * Working merlin manager run but start and stop not working properly * Made fix for subprocess to start new shell and fixed manager start and stop * Added comments and update changelog * Include style fixes * Fix style for black * Revert launch_job script that was edited when doing automated lint * Move importing of CONFIG to be within redis_connection due to error of config not being created yet * Added space to fix style * Revert launch_jobs.py: * Update import of all merlin.config to be in the function * suggested changes plus beginning work on monitor/manager collab * move managers to their own folder and fix ssl problems * final PR touch ups * Fix lint style changes * Fixed issue with context manager * Reset file that was incorrect changed * Check for ssl cert before applying to Redis connection * Comment out Active tests for celerymanager * split up create_server_config and write tests for it * add tests for config_merlin_server function * Fix lint issue with unused import after commenting out Active celery tests * Fixed style for import * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * Fixed kwargs being modified when making a copy for saving to redis worker args. * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * Added password check and omit if a password doesn't exist * update CHANGELOG * change testing log level to debug * add debug statement for redis_connection * change debug log to info so github ci will display it * attempt to fix password missing from Namespace error * run checks for all necessary configurations * convert stop-workers tests to pytest format * update github wf and comment out stop-workers tests in definitions.py * add missing key to GH wf file * fix invalid syntax in definitions.py * comment out stop_workers tests * playing with new caches for workflow CI * fix yaml syntax error * fix typo for getting runner os * fix test and add python version to CI cache * add in common-setup step again with caches this time * run fix-style * update CHANGELOG * fix remaining style issues * run without caches to compare execution time of test suite * allow redis config to not use ssl * remove stop-workers and query-workers tests from definitions.py * create helper_funcs file with common testing functions * move query-workers to pytest and add base class w/ stop-workers tests * update CHANGELOG * final changes for the stop-workers & query-workers tests * run fix-style * move stop and query workers tests to the same file * run fix-style * go back to original cache setup * try new cache for singularity install * fix syntax issue in github workflow * attempt to fix singularity cache * remove ls statement that breaks workflow * revert back to no common setup * remove unnecessary dependency * update github actions versions to use latest * update action versions that didn't save * run fix-style * move distributed test suite actions back to v2 * add 'merlin run' tests and port existing ones to pytest * update CHANGELOG * add aliased fixture types for typehinting * add tests for the purge command * update CHANGELOG * update run command tests to use conditions when appropriate * start work on adding workflow tests * create function and class scoped config fixtures * add Tuple fixture type * get e2e test of feature_demo workflow running * add check for proper variable substitution in e2e test * generalize functionality to run workflows * add create_testing_dir fixture * port chord error workflow to pytest * create dataclasses to house common fixtures and reduce fixture import requirements * fix lint issues * remove hard requirement of Annotated type for python 3.7 and 3.8 * remove distributed test CI and add unit test CI * fix typo in fixture_types and fix lint issues * run fix-style * add check for python2 before adding that condition check * convert local run test to use StepFinishedFilesCount condition * update CHANGELOG.md * fix problem created by merge conflict when mergin develop * remove manager functionality from this PR * update README for test suite * change SIGTERM to SIGKILL * update Makefile to include new changes to test suite --------- Co-authored-by: Ryan Lee Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> * Drop Python 3.7 and Add Python 3.12 & 3.13 (#495) * drop support for py 3.7, add support for py 3.12 and 3.13 * fix docs build issue * remove py 3.7 and add py 3.12/3.13 to integration tests * update Makefile to use target_version variable and update spellbook requirements in examples * Requirements fixes (#501) * add fix for broken deepdiff dependency in py38 and remove py37 specific requirements that were missed * remove try/except imports that are no longer necessary * update CHANGELOG * docs/api_docs (#496) * add/update docstrings for top-level files and enable API docs * remove Generator type hint * fix invalid escape sequences * add docstrings for the common/ directory * fix styling and add in cross-references to portions of Merlin codebase * give code blocks types for formatting * add api docs for half of the study directory * add API docs for study.py * ignore the data path for API docs * finish API docs for study directory * finish api docs for spec/ directory * update CHANGELOG * final cleanup of API docs for common, spec, and study folders * finish API docs for examples folder * began work on API docs for config folder * write API docs for server utils * finish api docs for the server directory * finish API docs for config directory * final cleanup * fix doc build issues * add section explaining API docs to the docs' README file * run fix-style * update readthedocs to build with latest python * fix too few arguments to generator issue * rename in MerlinStepRecord so that it's hidden again * add most of Charles' suggestions * remove unused openfilelist.py, opennpylib.py, and merlin_templates.py files, and add remainder of Charles suggestions * run fix-style * rename load_default_user_names to set_username_and_vhost * Update README.md to remove lgtm banner (#488) * Update README.md to remove lgtm banner * Update CHANGELOG.md * Delete .lgtm.yml * move changelog update to the unreleased section * remove lgtm.yml --------- Co-authored-by: Brian Gunnarson * Refactor/simplified-ci (#504) * split python and singularity setup into individual actions * add shell to new actions * try to fix shell issue * remove install-deps * reorder cache step and check pip version in the output * update CHANGELOG and make path to definitions.py relative * fix style issues * move cache check to setup-python action * fix an issue with docs build on python 3.9 (#503) * fix an issue with docs build on python 3.9 * adding ChatGPT suggestion to use get-pip.py script * add python version check to reinstall pip CI step * add reinstall pip step to all jobs that need it * save get-pip.py to tmp folder instead of directly to the repo * update CHANGELOG * fix .wci.yml links (#505) * Refactor/config (#498) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * Refactor/config-defaults (#497) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * remove debug step from github ci * add user permission check to local test suite * create a CI workflow for push and pull_request_target so secrets can be accessed * add link to article where I found this solution * add a merlin info test so we can see what server CI is connecting to * add unit tests for new config commands * add MerlinConfigManager class to replace config-related functions * add unit tests for MerlinConfigManager and update integration tests * comment out the pr-target workflow since it's not working * run fix-style * rewrite tests for find_config_file * remove uses of CONFIGFILE_DIR and chdir from configfile unit tests * run fix-style * re-order the find_config_file function to be local, config_path, merlin home * remove pr-target workflow, add missing docstrings, and run fix-style * add create and use commands, update MerlinConfigManager as necessary * update test suite for config command * update all documentation where 'merlin config' is mentioned * change merlin config to merlin config create * update CHANGELOG * run fix-style * update GitHub workflow to use 'merlin config create' * try to fix f-string issue * add Charles' suggestions * fix broken tests and security issue with new log statement * remove app.yaml file that was accidentally pushed * fix config file validation as Charles suggested * Refactor/server-config (#506) * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * change print to log statement * bugfix/local-config (#507) * add local mode config initialization * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * add unit tests for new functions in configfile.py * run fix-style and update CHANGELOG * change defaults to not use getpass * run fix-style * add changes that Charles suggested * feature/monitor-auto-restart (#494) * add classes for backends to help abstract them * establish templates and basic functionality for database interactions * flush out creation and retrieval of studies and runs * finish study/run storage, retrieval, and deletion to db functionality * add StudyNotFoundError and RunNotFoundError * add ability to dump run metadata to merlin_info directory * add new Celery signature to mark a run as complete * create merlin database commands * integrate database changes with monitor command * run fix-style and fix lint issues * update CHANGELOG and run fix-style with python 12 * last run of style fixes before push * run fix style with python 3.13 * create foundation for monitor auto-restart test * move get_backend_password import inside RedisBackend class * fix lint issue and comment out monitor test for now * fix redis connection so it works for ssl and non-ssl connections * add test for workflow auto-restart functionality * run linter * clean up print for DatabaseStudy and add -k option to delete all-studies * add docs for new monitor and database command * run fix-style * update built-in celery settings and retry exceptions to better handle timeouts * add new Monitor classes to handle the monitor command * update CHANGELOG and fix monitor test * fix lint issues * add worker entries and interactions with the database * add ABC class for database entities * add multi-run and multi-study queries, queries by study id, and queries by run workspace * add ability to delete multiple runs, studies, and workers. Also add ability to delete study by id and run by workspace * update docstrings for db and backend files * run fix-style * add repr method to HasRegex to make parsing GitHub CI easier * add ability to delete everything from db and retrieve connection string from MerlinDatabase * rename DatabaseRun and DatabaseStudy to more appropriate RunEntity and StudyEntity * split WorkerModel into LogicalWorkerModel and PhysicalWorkerModel * split redis backend class up into store classes for different entities. Removes db worker interaction temporarily * add logical worker functionality to database * add CLI functionality for physical workers and clean up MerlinDatabase * add physical worker db entry creation on celery worker startup * comment out code that's breaking worker launch * fix bug with get_all_runs method not existing anymore * run fix-style * fix bug with load/delete of physical workers and add functionality to set worker status to stopped * fix issue where ids were passed to wait_for_workers instead of names * move stop-worker celery functionality to a signal handler * add Logical Worker with ID 7620359f-6deb-0056-8956-39495dba8e59 ------------------------------------------------ Name: worker2 Runs: ['2e12e7cd-e7b9-468b-92c7-1ba1dffba852'] Queues: ['[merlin]_sim_queue', '[merlin]_seq_queue'] Physical Workers: ['dc863387-6b54-4a93-8a27-507bf0fd789e', '41a28c9e-f58d-49da-87bf-cea1569398b9'] Additional Data: {} Physical Worker with ID 41a28c9e-f58d-49da-87bf-cea1569398b9 ------------------------------------------------ Name: celery@worker2.%rzadams1011 Logical Worker ID: 7620359f-6deb-0056-8956-39495dba8e59 Launch Command: None Args: {} Process ID: None Status: WorkerStatus.RUNNING Last Heartbeat: 2025-04-14 12:40:05.786023 Last Spinup: 2025-04-14 12:40:05.786028 Host: rzadams1011 Restart Count: 0.0 Additional Data: {} Physical Worker with ID dc863387-6b54-4a93-8a27-507bf0fd789e ------------------------------------------------ Name: celery@worker2.%rzadams1010 Logical Worker ID: 7620359f-6deb-0056-8956-39495dba8e59 Launch Command: None Args: {} Process ID: None Status: WorkerStatus.RUNNING Last Heartbeat: 2025-04-14 12:40:05.027347 Last Spinup: 2025-04-14 12:40:05.027353 Host: rzadams1010 Restart Count: 0.0 Additional Data: {} Run with ID 2e12e7cd-e7b9-468b-92c7-1ba1dffba852 ------------------------------------------------ Workspace: /usr/WS1/gunny/scalability_testing/studies/long_running_wf_20250414-124613 Study ID: 64def1c2-8ee8-40a4-9f74-36d5b4872b9e Queues: ['[merlin]_seq_queue', '[merlin]_sim_queue'] Workers: ['7620359f-6deb-0056-8956-39495dba8e59'] Parent: None Child: None Run Complete: False Additional Data: {} Study with ID 64def1c2-8ee8-40a4-9f74-36d5b4872b9e ------------------------------------------------ Name: long_running_wf Runs: ['2e12e7cd-e7b9-468b-92c7-1ba1dffba852'] Additional Data: {} command * factor out common code from entity classes into mixin classes * run fix-style * remove process kill from celery shutdown * write tests for high-level modules of backends folder * add tests for the RedisBackend class * write unit tests for redis_logical_worker_store.py * add unit tests for RedisPhysicalWorkerStore * add tests for RedisRunStore * move db entities to their own folder * update version to 1.13.0a1 * add init.py file to backends/redis folder * change the monitor test so that it has more samples and purges tasks rather than stopping workers * run fix-style * fix error from merging develop * fix import issue with enums * condense store logic into base class and mixin class * add tests for new combined logic, remove tests for separated logic * run fix-style * fix imports and string representations of entities * update logical worker queue entry to be a set instead of list * make worker queues a set and fix str representations * update docs for database command * run fix style * change some info logs to debug * fix the majority of the docs warnings * fix docs warnings and run fix style * finish the database command docs * fix log messages and bug with deleting physical worker entry * add log files per Charles' suggestion * refactor db entities to share additional common logic * fix bug when deleting runs by workspace after workspace has been removed * add unit tests and some integration tests for data_models and db_commands * add unit tests for entity and mixin classes * refactor MerlinDatabase to use newly added entity managers * move mixins folder into entities folder * run fix-style * move mixin tests and remove todos from physical worker test file * add tests for the merlin_db module * add tests for entity managers * run fix-style * update CHANGELOG * fix unit tests for python 3.8 * fix typo * log warning if workspace doesn't exist rather than error * remove a function that was added by accident in merge from develop * fix import issue * reorganize redis_utils as they're no longer needed * move serialize and deserialize to common utils * add base store and move save, retrieve, delete methods up to ResultsBackend * initial work on adding sqlite support for local runs * finish implementing sqlite stores * update docstrings for the backend package * move existing tests of redis/results backend after refactor * add unit tests for sqlite modules and move around fixtures * add tests for StoreBase * fix unit tests for configfile * remove duplicate fixture * add some docs about sqlite and move database docs out of monitoring section * first attempt at fixing port issue * fix lint issues * second attempt at fixing the tests and style issues * implement Charles comments * update CHANGELOG * Update copyright (#510) * add COPYRIGHT file, update LICENSE, and change copyright header in all files * update Makefile commands related to copyright/copyright headers * Update CHANGELOG * add a check for the copyright header in 'make check-style' * update CHANGELOG * release/1.13.0b1 (#511) * Potential fix for code scanning alert no. 23: Workflow does not contain permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * codeql-fixes (#513) * fix logging security vulnerabilities * update CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning Co-authored-by: Joe Koning Co-authored-by: Jane Herriman Co-authored-by: Luc Peterson Co-authored-by: Ryan Lee Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> Co-authored-by: Wout De Nolf Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> --- .github/actions/setup-python/action.yml | 45 + .github/actions/setup-singularity/action.yml | 45 + .github/workflows/push-pr_workflow.yml | 346 ++- .gitignore | 1 + .lgtm.yml | 35 - .readthedocs.yaml | 3 +- .wci.yml | 6 +- CHANGELOG.md | 87 + COPYRIGHT | 10 + LICENSE | 4 +- Makefile | 128 +- README.md | 7 +- config.mk | 2 + docs/README.md | 82 +- docs/api_reference/index.md | 4 +- .../monitor-flowchart.png | Bin 49390 -> 164955 bytes docs/faq.md | 4 +- docs/gen_ref_pages.py | 21 +- docs/index.md | 2 +- docs/tutorial/2_installation.md | 112 +- docs/tutorial/3_hello_world.md | 2 +- docs/tutorial/4_run_simulation.md | 2 +- docs/user_guide/celery.md | 2 +- docs/user_guide/command_line.md | 613 ++++- .../configuration/containerized_server.md | 85 +- .../configuration/external_server.md | 2 +- docs/user_guide/configuration/index.md | 26 +- .../user_guide/configuration/merlin_server.md | 177 -- docs/user_guide/database/database_cmd.md | 1103 +++++++++ docs/user_guide/database/entities.md | 90 + docs/user_guide/database/index.md | 28 + docs/user_guide/installation.md | 2 +- docs/user_guide/interpreting_output.md | 2 +- .../monitoring/monitor_for_allocation.md | 65 +- docs/user_guide/running_studies.md | 2 +- docs/user_guide/specification.md | 2 +- lgtm.yml | 25 - merlin/__init__.py | 36 +- merlin/ascii_art.py | 34 +- merlin/backends/__init__.py | 28 + merlin/backends/backend_factory.py | 90 + merlin/backends/redis/__init__.py | 19 + merlin/backends/redis/redis_backend.py | 106 + merlin/backends/redis/redis_store_base.py | 252 ++ merlin/backends/redis/redis_stores.py | 91 + merlin/backends/results_backend.py | 239 ++ merlin/backends/sqlite/__init__.py | 21 + merlin/backends/sqlite/sqlite_backend.py | 107 + merlin/backends/sqlite/sqlite_connection.py | 96 + merlin/backends/sqlite/sqlite_store_base.py | 232 ++ merlin/backends/sqlite/sqlite_stores.py | 65 + merlin/backends/store_base.py | 78 + merlin/backends/utils.py | 137 ++ merlin/celery.py | 162 +- merlin/common/__init__.py | 55 +- merlin/common/abstracts/__init__.py | 29 - merlin/common/abstracts/enums/__init__.py | 51 - merlin/common/dumper.py | 146 +- merlin/common/enums.py | 59 + merlin/common/openfilelist.py | 198 -- merlin/common/opennpylib.py | 391 --- merlin/common/sample_index.py | 549 ++++- merlin/common/sample_index_factory.py | 161 +- merlin/common/security/__init__.py | 43 +- merlin/common/security/encrypt.py | 101 +- .../security/encrypt_backend_traffic.py | 73 +- merlin/common/tasks.py | 513 ++-- merlin/common/util_sampling.py | 90 +- merlin/config/__init__.py | 91 +- merlin/config/broker.py | 168 +- merlin/config/celeryconfig.py | 61 +- merlin/config/config_filepaths.py | 18 + merlin/config/configfile.py | 360 ++- merlin/config/merlin_config_manager.py | 227 ++ merlin/config/results_backend.py | 223 +- merlin/config/utils.py | 126 +- merlin/data/celery/__init__.py | 34 +- merlin/data/celery/app.yaml | 69 +- merlin/data/celery/app_redis.yaml | 72 +- merlin/data/celery/app_test.yaml | 44 - merlin/db_scripts/__init__.py | 30 + merlin/db_scripts/data_models.py | 456 ++++ merlin/db_scripts/db_commands.py | 171 ++ merlin/db_scripts/entities/__init__.py | 32 + merlin/db_scripts/entities/db_entity.py | 219 ++ .../entities/logical_worker_entity.py | 176 ++ merlin/db_scripts/entities/mixins/__init__.py | 25 + merlin/db_scripts/entities/mixins/name.py | 42 + .../entities/mixins/queue_management.py | 45 + .../entities/mixins/run_management.py | 107 + .../entities/physical_worker_entity.py | 304 +++ merlin/db_scripts/entities/run_entity.py | 345 +++ merlin/db_scripts/entities/study_entity.py | 114 + merlin/db_scripts/entity_managers/__init__.py | 31 + .../entity_managers/entity_manager.py | 237 ++ .../entity_managers/logical_worker_manager.py | 188 ++ .../physical_worker_manager.py | 153 ++ .../db_scripts/entity_managers/run_manager.py | 174 ++ .../entity_managers/study_manager.py | 143 ++ merlin/db_scripts/merlin_db.py | 267 +++ merlin/display.py | 270 ++- merlin/examples/__init__.py | 43 +- .../dev_workflows/multiple_workers.yaml | 6 +- merlin/examples/examples.py | 43 +- merlin/examples/generator.py | 87 +- .../workflows/feature_demo/requirements.txt | 3 +- .../workflows/optimization/requirements.txt | 3 +- .../remote_feature_demo/requirements.txt | 3 +- merlin/exceptions/__init__.py | 102 +- merlin/log_formatter.py | 44 +- merlin/main.py | 811 ++++++- merlin/merlin_templates.py | 75 - merlin/monitor/__init__.py | 24 + merlin/monitor/celery_monitor.py | 188 ++ merlin/monitor/monitor.py | 198 ++ merlin/monitor/monitor_factory.py | 75 + merlin/monitor/task_server_monitor.py | 86 + merlin/router.py | 331 +-- merlin/server/__init__.py | 43 +- merlin/server/server_commands.py | 150 +- merlin/server/server_config.py | 273 ++- merlin/server/server_util.py | 2125 +++++++++++++++-- merlin/spec/__init__.py | 51 +- merlin/spec/all_keys.py | 46 +- merlin/spec/defaults.py | 46 +- merlin/spec/expansion.py | 314 ++- merlin/spec/override.py | 74 +- merlin/spec/specification.py | 690 ++++-- merlin/study/__init__.py | 57 +- merlin/study/batch.py | 295 ++- merlin/study/celeryadapter.py | 718 ++++-- merlin/study/dag.py | 327 ++- merlin/study/script_adapter.py | 519 ++-- merlin/study/status.py | 636 +++-- merlin/study/status_constants.py | 39 +- merlin/study/status_renderers.py | 280 ++- merlin/study/step.py | 401 +++- merlin/study/study.py | 531 ++-- merlin/utils.py | 806 +++++-- mkdocs.yml | 31 +- requirements/dev.txt | 1 + requirements/release.txt | 2 - setup.py | 38 +- tests/README.md | 6 +- tests/__init__.py | 5 + tests/conftest.py | 452 +++- tests/constants.py | 6 + tests/context_managers/__init__.py | 5 + tests/context_managers/celery_task_manager.py | 161 ++ .../celery_workers_manager.py | 81 +- tests/context_managers/server_manager.py | 8 +- tests/fixture_data_classes.py | 105 + tests/fixture_types.py | 87 + tests/fixtures/__init__.py | 6 + tests/fixtures/chord_err.py | 120 + tests/fixtures/config.py | 29 + tests/fixtures/examples.py | 26 +- tests/fixtures/feature_demo.py | 142 ++ tests/fixtures/monitor.py | 64 + tests/fixtures/run_command.py | 29 + tests/fixtures/server.py | 68 +- tests/fixtures/status.py | 33 +- tests/fixtures/stores.py | 122 + tests/integration/__init__.py | 5 + tests/integration/commands/__init__.py | 5 + tests/integration/commands/pgen.py | 42 + tests/integration/commands/test_monitor.py | 120 + tests/integration/commands/test_purge.py | 369 +++ tests/integration/commands/test_run.py | 474 ++++ .../commands/test_stop_and_query_workers.py | 386 +++ tests/integration/conditions.py | 157 +- .../integration/database/test_data_models.py | 74 + tests/integration/definitions.py | 314 +-- tests/integration/helper_funcs.py | 170 ++ tests/integration/run_tests.py | 37 +- tests/integration/test_celeryadapter.py | 339 ++- tests/integration/test_specs/chord_err.yaml | 3 +- .../test_specs/monitor_auto_restart_test.yaml | 36 + .../test_specs/multiple_workers.yaml | 56 + .../integration/workflows/test_chord_error.py | 69 + .../workflows/test_feature_demo.py | 138 ++ .../unit/backends/redis/test_redis_backend.py | 133 ++ .../backends/redis/test_redis_store_base.py | 495 ++++ .../unit/backends/redis/test_redis_stores.py | 86 + .../backends/sqlite/test_sqlite_backend.py | 88 + .../backends/sqlite/test_sqlite_connection.py | 92 + .../backends/sqlite/test_sqlite_store_base.py | 457 ++++ .../backends/sqlite/test_sqlite_stores.py | 96 + tests/unit/backends/test_backend_factory.py | 86 + tests/unit/backends/test_results_backend.py | 344 +++ tests/unit/backends/test_store_base.py | 66 + tests/unit/backends/test_utils.py | 47 + tests/unit/common/test_dumper.py | 6 + tests/unit/common/test_encryption.py | 25 +- tests/unit/common/test_sample_index.py | 6 + tests/unit/common/test_util_sampling.py | 6 + tests/unit/config/__init__.py | 5 + tests/unit/config/test_broker.py | 92 +- tests/unit/config/test_config_object.py | 6 + tests/unit/config/test_configfile.py | 624 +++-- .../unit/config/test_merlin_config_manager.py | 208 ++ tests/unit/config/test_results_backend.py | 62 +- tests/unit/config/test_utils.py | 18 +- tests/unit/db_scripts/__init__.py | 5 + tests/unit/db_scripts/entities/__init__.py | 5 + .../db_scripts/entities/test_db_entity.py | 351 +++ .../entities/test_logical_worker_entity.py | 223 ++ tests/unit/db_scripts/entities/test_mixins.py | 216 ++ .../entities/test_physical_worker_entity.py | 324 +++ .../db_scripts/entities/test_run_entity.py | 333 +++ .../db_scripts/entities/test_study_entity.py | 171 ++ .../db_scripts/entity_managers/__init__.py | 5 + .../entity_managers/test_entity_manager.py | 409 ++++ .../test_logical_worker_manager.py | 256 ++ .../test_physical_worker_manager.py | 259 ++ .../entity_managers/test_run_manager.py | 271 +++ .../entity_managers/test_study_manager.py | 226 ++ tests/unit/db_scripts/test_data_models.py | 589 +++++ tests/unit/db_scripts/test_db_commands.py | 668 ++++++ tests/unit/db_scripts/test_merlin_db.py | 449 ++++ tests/unit/server/__init__.py | 5 + tests/unit/server/test_RedisConfig.py | 6 + tests/unit/server/test_server_commands.py | 6 + tests/unit/server/test_server_config.py | 46 +- tests/unit/server/test_server_util.py | 9 +- tests/unit/spec/__init__.py | 5 + tests/unit/spec/test_specification.py | 6 + tests/unit/study/__init__.py | 34 +- .../status_test_files/combine_status_files.py | 35 +- .../study/status_test_files/shared_tests.py | 35 +- .../status_test_variables.py | 35 +- tests/unit/study/test_detailed_status.py | 35 +- tests/unit/study/test_status.py | 35 +- tests/unit/study/test_study.py | 6 + tests/unit/test_examples_generator.py | 6 + tests/unit/utils/test_dict_deep_merge.py | 6 + tests/unit/utils/test_get_package_version.py | 6 + tests/unit/utils/test_time_formats.py | 6 + tests/utils.py | 6 + 239 files changed, 29076 insertions(+), 6470 deletions(-) create mode 100644 .github/actions/setup-python/action.yml create mode 100644 .github/actions/setup-singularity/action.yml delete mode 100644 .lgtm.yml create mode 100644 COPYRIGHT delete mode 100644 docs/user_guide/configuration/merlin_server.md create mode 100644 docs/user_guide/database/database_cmd.md create mode 100644 docs/user_guide/database/entities.md create mode 100644 docs/user_guide/database/index.md delete mode 100644 lgtm.yml create mode 100644 merlin/backends/__init__.py create mode 100644 merlin/backends/backend_factory.py create mode 100644 merlin/backends/redis/__init__.py create mode 100644 merlin/backends/redis/redis_backend.py create mode 100644 merlin/backends/redis/redis_store_base.py create mode 100644 merlin/backends/redis/redis_stores.py create mode 100644 merlin/backends/results_backend.py create mode 100644 merlin/backends/sqlite/__init__.py create mode 100644 merlin/backends/sqlite/sqlite_backend.py create mode 100644 merlin/backends/sqlite/sqlite_connection.py create mode 100644 merlin/backends/sqlite/sqlite_store_base.py create mode 100644 merlin/backends/sqlite/sqlite_stores.py create mode 100644 merlin/backends/store_base.py create mode 100644 merlin/backends/utils.py delete mode 100644 merlin/common/abstracts/__init__.py delete mode 100644 merlin/common/abstracts/enums/__init__.py create mode 100644 merlin/common/enums.py delete mode 100644 merlin/common/openfilelist.py delete mode 100644 merlin/common/opennpylib.py create mode 100644 merlin/config/config_filepaths.py create mode 100644 merlin/config/merlin_config_manager.py delete mode 100644 merlin/data/celery/app_test.yaml create mode 100644 merlin/db_scripts/__init__.py create mode 100644 merlin/db_scripts/data_models.py create mode 100644 merlin/db_scripts/db_commands.py create mode 100644 merlin/db_scripts/entities/__init__.py create mode 100644 merlin/db_scripts/entities/db_entity.py create mode 100644 merlin/db_scripts/entities/logical_worker_entity.py create mode 100644 merlin/db_scripts/entities/mixins/__init__.py create mode 100644 merlin/db_scripts/entities/mixins/name.py create mode 100644 merlin/db_scripts/entities/mixins/queue_management.py create mode 100644 merlin/db_scripts/entities/mixins/run_management.py create mode 100644 merlin/db_scripts/entities/physical_worker_entity.py create mode 100644 merlin/db_scripts/entities/run_entity.py create mode 100644 merlin/db_scripts/entities/study_entity.py create mode 100644 merlin/db_scripts/entity_managers/__init__.py create mode 100644 merlin/db_scripts/entity_managers/entity_manager.py create mode 100644 merlin/db_scripts/entity_managers/logical_worker_manager.py create mode 100644 merlin/db_scripts/entity_managers/physical_worker_manager.py create mode 100644 merlin/db_scripts/entity_managers/run_manager.py create mode 100644 merlin/db_scripts/entity_managers/study_manager.py create mode 100644 merlin/db_scripts/merlin_db.py delete mode 100644 merlin/merlin_templates.py create mode 100644 merlin/monitor/__init__.py create mode 100644 merlin/monitor/celery_monitor.py create mode 100644 merlin/monitor/monitor.py create mode 100644 merlin/monitor/monitor_factory.py create mode 100644 merlin/monitor/task_server_monitor.py create mode 100644 tests/context_managers/celery_task_manager.py create mode 100644 tests/fixture_data_classes.py create mode 100644 tests/fixture_types.py create mode 100644 tests/fixtures/chord_err.py create mode 100644 tests/fixtures/config.py create mode 100644 tests/fixtures/feature_demo.py create mode 100644 tests/fixtures/monitor.py create mode 100644 tests/fixtures/run_command.py create mode 100644 tests/fixtures/stores.py create mode 100644 tests/integration/commands/__init__.py create mode 100644 tests/integration/commands/pgen.py create mode 100644 tests/integration/commands/test_monitor.py create mode 100644 tests/integration/commands/test_purge.py create mode 100644 tests/integration/commands/test_run.py create mode 100644 tests/integration/commands/test_stop_and_query_workers.py create mode 100644 tests/integration/database/test_data_models.py create mode 100644 tests/integration/helper_funcs.py create mode 100644 tests/integration/test_specs/monitor_auto_restart_test.yaml create mode 100644 tests/integration/test_specs/multiple_workers.yaml create mode 100644 tests/integration/workflows/test_chord_error.py create mode 100644 tests/integration/workflows/test_feature_demo.py create mode 100644 tests/unit/backends/redis/test_redis_backend.py create mode 100644 tests/unit/backends/redis/test_redis_store_base.py create mode 100644 tests/unit/backends/redis/test_redis_stores.py create mode 100644 tests/unit/backends/sqlite/test_sqlite_backend.py create mode 100644 tests/unit/backends/sqlite/test_sqlite_connection.py create mode 100644 tests/unit/backends/sqlite/test_sqlite_store_base.py create mode 100644 tests/unit/backends/sqlite/test_sqlite_stores.py create mode 100644 tests/unit/backends/test_backend_factory.py create mode 100644 tests/unit/backends/test_results_backend.py create mode 100644 tests/unit/backends/test_store_base.py create mode 100644 tests/unit/backends/test_utils.py create mode 100644 tests/unit/config/test_merlin_config_manager.py create mode 100644 tests/unit/db_scripts/__init__.py create mode 100644 tests/unit/db_scripts/entities/__init__.py create mode 100644 tests/unit/db_scripts/entities/test_db_entity.py create mode 100644 tests/unit/db_scripts/entities/test_logical_worker_entity.py create mode 100644 tests/unit/db_scripts/entities/test_mixins.py create mode 100644 tests/unit/db_scripts/entities/test_physical_worker_entity.py create mode 100644 tests/unit/db_scripts/entities/test_run_entity.py create mode 100644 tests/unit/db_scripts/entities/test_study_entity.py create mode 100644 tests/unit/db_scripts/entity_managers/__init__.py create mode 100644 tests/unit/db_scripts/entity_managers/test_entity_manager.py create mode 100644 tests/unit/db_scripts/entity_managers/test_logical_worker_manager.py create mode 100644 tests/unit/db_scripts/entity_managers/test_physical_worker_manager.py create mode 100644 tests/unit/db_scripts/entity_managers/test_run_manager.py create mode 100644 tests/unit/db_scripts/entity_managers/test_study_manager.py create mode 100644 tests/unit/db_scripts/test_data_models.py create mode 100644 tests/unit/db_scripts/test_db_commands.py create mode 100644 tests/unit/db_scripts/test_merlin_db.py diff --git a/.github/actions/setup-python/action.yml b/.github/actions/setup-python/action.yml new file mode 100644 index 000000000..d412562bf --- /dev/null +++ b/.github/actions/setup-python/action.yml @@ -0,0 +1,45 @@ +name: Setup Python Environment +description: Setup Python, reinstall pip, and install dependencies +inputs: + python-version: + required: true + +runs: + using: "composite" + steps: + - name: Cache pip dependencies + uses: actions/cache@v4 + with: + path: ~/.cache/pip + key: ${{ hashFiles('requirements/release.txt') }}-${{ hashFiles('requirements/dev.txt') }} + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: ${{ inputs.python-version }} + + - name: Reinstall pip + shell: bash + run: | + PY_VER=$(python -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')") + PY_MAJOR=$(echo $PY_VER | cut -d. -f1) + PY_MINOR=$(echo $PY_VER | cut -d. -f2) + + if [ "$PY_MAJOR" -eq 3 ] && [ "$PY_MINOR" -le 8 ]; then + URL="https://bootstrap.pypa.io/pip/${PY_VER}/get-pip.py" + else + URL="https://bootstrap.pypa.io/get-pip.py" + fi + + curl -sS "$URL" -o /tmp/get-pip.py + python /tmp/get-pip.py --force-reinstall + + pip --version + pip3 --version + + - name: Install dependencies + shell: bash + run: | + python3 -m pip install --upgrade pip + if [ -f requirements.txt ]; then pip install -r requirements.txt; fi + pip3 install -r requirements/dev.txt diff --git a/.github/actions/setup-singularity/action.yml b/.github/actions/setup-singularity/action.yml new file mode 100644 index 000000000..14e376e7d --- /dev/null +++ b/.github/actions/setup-singularity/action.yml @@ -0,0 +1,45 @@ +name: Install Singularity +description: Install Go and Singularity +inputs: + go-version: + required: true + singularity-version: + required: true + os: + default: linux + arch: + default: amd64 + +runs: + using: "composite" + steps: + - name: Install system dependencies + shell: bash + run: | + sudo apt-get update && sudo apt-get install -y \ + build-essential \ + libssl-dev \ + uuid-dev \ + libgpgme11-dev \ + squashfs-tools \ + libseccomp-dev \ + pkg-config + + - name: Download and install Go + shell: bash + run: | + wget https://go.dev/dl/go${{ inputs.go-version }}.${{ inputs.os }}-${{ inputs.arch }}.tar.gz + sudo tar -C /usr/local -xzf go${{ inputs.go-version }}.${{ inputs.os }}-${{ inputs.arch }}.tar.gz + rm go${{ inputs.go-version }}.${{ inputs.os }}-${{ inputs.arch }}.tar.gz + echo "/usr/local/go/bin" >> $GITHUB_PATH + + - name: Download and install Singularity + shell: bash + run: | + wget https://github.com/sylabs/singularity/releases/download/v${{ inputs.singularity-version }}/singularity-ce-${{ inputs.singularity-version }}.tar.gz + tar -xzf singularity-ce-${{ inputs.singularity-version }}.tar.gz + cd singularity-ce-${{ inputs.singularity-version }} + ./mconfig + make -C ./builddir + sudo make -C ./builddir install + diff --git a/.github/workflows/push-pr_workflow.yml b/.github/workflows/push-pr_workflow.yml index 1d3c9d958..d28320982 100644 --- a/.github/workflows/push-pr_workflow.yml +++ b/.github/workflows/push-pr_workflow.yml @@ -2,6 +2,9 @@ name: Python CI on: [push, pull_request] +permissions: + contents: read + jobs: Changelog: name: CHANGELOG.md updated @@ -9,29 +12,28 @@ jobs: if: github.event_name == 'pull_request' steps: - - name: Checkout code - uses: actions/checkout@v2 - with: - fetch-depth: 0 # Checkout the whole history, in case the target is way far behind - - - name: Check if target branch has been merged - run: | - if git merge-base --is-ancestor ${{ github.event.pull_request.base.sha }} ${{ github.sha }}; then - echo "Target branch has been merged into the source branch." - else - echo "Target branch has not been merged into the source branch. Please merge in target first." - exit 1 - fi - - - name: Check that CHANGELOG has been updated - run: | - # If this step fails, this means you haven't updated the CHANGELOG.md file with notes on your contribution. - if git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.sha }} | grep -q '^CHANGELOG.md$'; then - echo "Thanks for helping keep our CHANGELOG up-to-date!" - else - echo "Please update the CHANGELOG.md file with notes on your contribution." - exit 1 - fi + - uses: actions/checkout@v4 + with: + fetch-depth: 0 # Checkout the whole history, in case the target is way far behind + + - name: Check if target branch has been merged + run: | + if git merge-base --is-ancestor ${{ github.event.pull_request.base.sha }} ${{ github.sha }}; then + echo "Target branch has been merged into the source branch." + else + echo "Target branch has not been merged into the source branch. Please merge in target first." + exit 1 + fi + + - name: Check that CHANGELOG has been updated + run: | + # If this step fails, this means you haven't updated the CHANGELOG.md file with notes on your contribution. + if git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.sha }} | grep -q '^CHANGELOG.md$'; then + echo "Thanks for helping keep our CHANGELOG up-to-date!" + else + echo "Please update the CHANGELOG.md file with notes on your contribution." + exit 1 + fi Lint: runs-on: ubuntu-latest @@ -40,51 +42,34 @@ jobs: MAX_COMPLEXITY: 15 steps: - - uses: actions/checkout@v2 - - name: Set up Python - uses: actions/setup-python@v2 - with: - python-version: '3.x' - - - name: Check cache - uses: actions/cache@v2 - with: - path: ~/.cache/pip - key: ${{ hashFiles('requirements/release.txt') }}-${{ hashFiles('requirements/dev.txt') }} - - - name: Install dependencies - run: | - python3 -m pip install --upgrade pip - if [ -f requirements.txt ]; then pip install --upgrade -r requirements.txt; fi - pip3 install --upgrade -r requirements/dev.txt - - - name: Lint with flake8 - run: | - # stop the build if there are Python syntax errors or undefined names - flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics - # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide - flake8 . --count --max-complexity=$MAX_COMPLEXITY --statistics --max-line-length=$MAX_LINE_LENGTH - - - name: Lint with isort - run: | - python3 -m isort --check --line-length $MAX_LINE_LENGTH merlin - python3 -m isort --check --line-length $MAX_LINE_LENGTH tests - python3 -m isort --check --line-length $MAX_LINE_LENGTH *.py - - - name: Lint with Black - run: | - python3 -m black --check --line-length $MAX_LINE_LENGTH --target-version py38 merlin - python3 -m black --check --line-length $MAX_LINE_LENGTH --target-version py38 tests - python3 -m black --check --line-length $MAX_LINE_LENGTH --target-version py38 *.py - - - name: Lint with PyLint - run: | - python3 -m pylint merlin --rcfile=setup.cfg --exit-zero - python3 -m pylint tests --rcfile=setup.cfg --exit-zero + - uses: actions/checkout@v4 + + - uses: ./.github/actions/setup-python + with: + python-version: '3.x' + + - name: Lint with flake8 + run: | + # stop the build if there are Python syntax errors or undefined names + flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics + # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide + flake8 . --count --max-complexity=$MAX_COMPLEXITY --statistics --max-line-length=$MAX_LINE_LENGTH + + - name: Lint with isort + run: | + isort --check --line-length $MAX_LINE_LENGTH merlin tests *.py + + - name: Lint with Black + run: | + black --check --line-length $MAX_LINE_LENGTH --target-version py38 merlin tests *.py + + - name: Lint with PyLint + run: | + pylint merlin tests --rcfile=setup.cfg --exit-zero Local-test-suite: runs-on: ubuntu-latest - env: + env: GO_VERSION: 1.18.1 SINGULARITY_VERSION: 3.9.9 OS: linux @@ -92,138 +77,115 @@ jobs: strategy: matrix: - python-version: ['3.7', '3.8', '3.9', '3.10', '3.11'] + python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13'] steps: - - uses: actions/checkout@v2 - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v2 - with: - python-version: ${{ matrix.python-version }} - - - name: Check cache - uses: actions/cache@v2 - with: - path: ${{ env.pythonLocation }} - key: ${{ env.pythonLocation }}-${{ hashFiles('requirements/release.txt') }}-${{ hashFiles('requirements/dev.txt') }} - - - name: Install dependencies - run: | - python3 -m pip install --upgrade pip - if [ -f requirements.txt ]; then pip install -r requirements.txt; fi - pip3 install -r requirements/dev.txt - pip freeze - - - name: Install singularity - run: | - sudo apt-get update && sudo apt-get install -y \ - build-essential \ - libssl-dev \ - uuid-dev \ - libgpgme11-dev \ - squashfs-tools \ - libseccomp-dev \ - pkg-config - wget https://go.dev/dl/go$GO_VERSION.$OS-$ARCH.tar.gz - sudo tar -C /usr/local -xzf go$GO_VERSION.$OS-$ARCH.tar.gz - rm go$GO_VERSION.$OS-$ARCH.tar.gz - export PATH=$PATH:/usr/local/go/bin - wget https://github.com/sylabs/singularity/releases/download/v$SINGULARITY_VERSION/singularity-ce-$SINGULARITY_VERSION.tar.gz - tar -xzf singularity-ce-$SINGULARITY_VERSION.tar.gz - cd singularity-ce-$SINGULARITY_VERSION - ./mconfig && \ - make -C ./builddir && \ - sudo make -C ./builddir install - - - name: Install merlin to run unit tests - run: | - pip3 install -e . - merlin config - - - name: Install CLI task dependencies generated from the 'feature demo' workflow - run: | - merlin example feature_demo - pip3 install -r feature_demo/requirements.txt - - - name: Run pytest over unit test suite - run: | - python3 -m pytest -v --order-scope=module tests/unit/ - - - name: Run integration test suite for local tests - run: | - python3 tests/integration/run_tests.py --verbose --local - - Distributed-test-suite: + - uses: actions/checkout@v4 + + - uses: ./.github/actions/setup-python + with: + python-version: ${{ matrix.python-version }} + + - uses: ./.github/actions/setup-singularity + with: + go-version: ${{ env.GO_VERSION }} + singularity-version: ${{ env.SINGULARITY_VERSION }} + os: ${{ env.OS }} + arch: ${{ env.ARCH }} + + - name: Install merlin and setup + run: | + pip3 install -e . + merlin config create + + - name: Install CLI task dependencies from 'feature_demo' workflow + run: | + merlin example feature_demo + pip3 install -r feature_demo/requirements.txt + + - name: Run integration test suite for local tests + run: | + python3 tests/integration/run_tests.py --verbose --local + + Unit-tests: runs-on: ubuntu-latest - services: - # rabbitmq: - # image: rabbitmq:latest - # ports: - # - 5672:5672 - # options: --health-cmd "rabbitmqctl node_health_check" --health-interval 10s --health-timeout 5s --health-retries 5 - # Label used to access the service container - redis: - # Docker Hub image - image: redis - # Set health checks to wait until redis has started - options: >- - --health-cmd "redis-cli ping" - --health-interval 10s - --health-timeout 5s - --health-retries 5 - ports: - - 6379:6379 + env: + GO_VERSION: 1.18.1 + SINGULARITY_VERSION: 3.9.9 + OS: linux + ARCH: amd64 + + strategy: + matrix: + python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13'] + + steps: + - uses: actions/checkout@v4 + + - uses: ./.github/actions/setup-python + with: + python-version: ${{ matrix.python-version }} + + - uses: ./.github/actions/setup-singularity + with: + go-version: ${{ env.GO_VERSION }} + singularity-version: ${{ env.SINGULARITY_VERSION }} + os: ${{ env.OS }} + arch: ${{ env.ARCH }} + + - name: Install merlin and setup + run: | + pip3 install -e . + merlin config create + + - name: Install CLI task dependencies from 'feature_demo' workflow + run: | + merlin example feature_demo + pip3 install -r feature_demo/requirements.txt + + - name: Run pytest over unit test suite + run: | + python3 -m pytest -v --order-scope=module tests/unit/ + + Integration-tests: + runs-on: ubuntu-latest + env: + GO_VERSION: 1.18.1 + SINGULARITY_VERSION: 3.9.9 + OS: linux + ARCH: amd64 strategy: matrix: - python-version: ['3.7', '3.8', '3.9', '3.10', '3.11'] + python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13'] steps: - - uses: actions/checkout@v2 - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v2 - with: - python-version: ${{ matrix.python-version }} - - - name: Check cache - uses: actions/cache@v2 - with: - path: ${{ env.pythonLocation }} - key: ${{ env.pythonLocation }}-${{ hashFiles('requirements/release.txt') }}-${{ hashFiles('requirements/dev.txt') }} - - - name: Install dependencies - run: | - python3 -m pip install --upgrade pip - if [ -f requirements.txt ]; then pip install -r requirements.txt; fi - pip3 install -r requirements/dev.txt - - - name: Install merlin and setup redis as the broker - run: | - pip3 install -e . - merlin config --broker redis - - - name: Install CLI task dependencies generated from the 'feature demo' workflow - run: | - merlin example feature_demo - pip3 install -r feature_demo/requirements.txt - - - name: Run integration test suite for distributed tests - env: - REDIS_HOST: redis - REDIS_PORT: 6379 - run: | - python3 tests/integration/run_tests.py --verbose --distributed - - # - name: Setup rabbitmq config - # run: | - # merlin config --test rabbitmq - - # - name: Run integration test suite for rabbitmq - # env: - # AMQP_URL: amqp://localhost:${{ job.services.rabbitmq.ports[5672] }} - # RABBITMQ_USER: Jimmy_Space - # RABBITMQ_PASS: Alexander_Rules - # ports: - # - ${{ job.services.rabbitmq.ports['5672'] }} - # run: | - # python3 tests/integration/run_tests.py --verbose --ids 31 32 + - uses: actions/checkout@v4 + + - uses: ./.github/actions/setup-python + with: + python-version: ${{ matrix.python-version }} + + - uses: ./.github/actions/setup-singularity + with: + go-version: ${{ env.GO_VERSION }} + singularity-version: ${{ env.SINGULARITY_VERSION }} + os: ${{ env.OS }} + arch: ${{ env.ARCH }} + + - name: Install merlin + run: | + pip --version + pip3 --version + pip3 install -e . + merlin config create + + - name: Install CLI task dependencies from 'feature_demo' workflow + run: | + merlin example feature_demo + pip3 install -r feature_demo/requirements.txt + + # TODO remove the --ignore statement once those tests are fixed + - name: Run integration test suite for distributed tests + run: | + python3 -m pytest -v --ignore tests/integration/test_celeryadapter.py tests/integration/ diff --git a/.gitignore b/.gitignore index cec577a85..6ce8997ed 100644 --- a/.gitignore +++ b/.gitignore @@ -42,6 +42,7 @@ docs/build/ # Test files .tox/* .coverage +**/setup_env.sh # Jupyter jupyter/.ipynb_checkpoints diff --git a/.lgtm.yml b/.lgtm.yml deleted file mode 100644 index b1b073e2b..000000000 --- a/.lgtm.yml +++ /dev/null @@ -1,35 +0,0 @@ -########################################################################################## -# Customize file classifications. # -# Results from files under any classifier will be excluded from LGTM # -# statistics. # -########################################################################################## - -########################################################################################## -# Use the `path_classifiers` block to define changes to the default classification of # -# files. # -########################################################################################## - -path_classifiers: - test: - # Exclude all files to ovveride lgtm defaults - - exclude: / - # Classify all files in the top-level directories tests/ - # and merlin/examples as test code. - - tests - - merlin/examples - -######################################################################################### -# Use the `queries` block to change the default display of query results. # -# The py/clear-text-logging-sensitive-data exclusion is due to cert and password # -# keywords being flagged as security leaks. # -######################################################################################### - -queries: - - include: "*" - - exclude: "py/clear-text-logging-sensitive-data" - -extraction: - python: - python_setup: - version: "3" - diff --git a/.readthedocs.yaml b/.readthedocs.yaml index c3cfbbe07..7c5ed2c04 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -3,7 +3,7 @@ version: 2 build: os: "ubuntu-20.04" tools: - python: "3.8" + python: "3.13" python: install: @@ -11,5 +11,6 @@ python: mkdocs: fail_on_warning: false + configuration: mkdocs.yml formats: [pdf] \ No newline at end of file diff --git a/.wci.yml b/.wci.yml index 0ced2c93b..6b1502f8b 100644 --- a/.wci.yml +++ b/.wci.yml @@ -1,6 +1,6 @@ name: Merlin -icon: https://raw.githubusercontent.com/LLNL/merlin/main/docs/images/merlin_icon.png +icon: https://raw.githubusercontent.com/LLNL/merlin/main/docs/assets/images/merlin_icon.png headline: Enabling Machine Learning HPC Workflows @@ -8,5 +8,5 @@ description: The Merlin workflow framework targets large-scale scientific machin documentation: general: https://merlin.readthedocs.io/ - installation: https://merlin.readthedocs.io/en/latest/modules/installation/installation.html - tutorial: https://merlin.readthedocs.io/en/latest/tutorial.html + installation: https://merlin.readthedocs.io/en/latest/user_guide/installation/ + tutorial: https://merlin.readthedocs.io/en/latest/tutorial/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 5bd534b1b..f2ee0eccb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,93 @@ All notable changes to Merlin will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [1.13.0b1] +### Added +- API documentation for Merlin's core codebase +- New `merlin database` command to interact with new database functionality + - When running locally, SQLite will be used as the database. Otherwise your current results backend will be used + - `merlin database info`: prints some basic information about the database + - `merlin database get`: allows you to retrieve and print entries in the database + - `merlin database delete`: allows you to delete entries in the database +- Added `db_scripts/` folder containing several new files all pertaining to database interaction + - `data_models`: a module that houses dataclasses that define the format of the data that's stored in Merlin's database. + - `db_commands`: an interface for user commands of `merlin database` to be processed + - `merlin_db`: houses the `MerlinDatabase` class, used as the main point of contact for interactions with the database + - `entities/`: A folder containing modules that define a structured interface for interacting with persisted data. + - `entity_managers/`: A folder containing classes responsible for managing high-level database operations across all entities. +- Added `backends/` folder containing a new OOP way to interact with results backend databases + - `results_backend`: houses an abstract class `ResultsBackend` that defines what every supported backend implement in Merlin + - `redis/`: A folder containing the `RedisBackend` class that defines specific interactions with the Redis database + - `sqlite/`: A folder containing the `SQLiteBackend` class that defines specific interactions with the SQLite database + - `backend_factory`: houses a factory class `MerlinBackendFactory` that initializes an appropriate `ResultsBackend` instance +- Added `monitors/` folder containing a refactored, OOP approach to handling the `merlin monitor` command + - `celery_monitor`: houses the `CeleryMonitor` class a concrete subclass of `TaskServerMonitor` for monitoring Celery task servers + - `monitor_factory`: houses a factory class `MonitorFactory` that initializes an appropriate `TaskServerMonitor` instance + - `monitor`: houses the `Monitor` class, used as the top-level point of interaction for the monitor command + - `task_server_monitor`: houses the `TaskServerMonitor` ABC class, which serves as a common interface for monitoring task servers +- A new celery task called `mark_run_as_complete` that is automatically added to the task queue associated with the final step in a workflow +- Added support for Python 3.12 and 3.13 +- Added additional tests for the `merlin run` and `merlin purge` commands +- Aliased types to represent different types of pytest fixtures +- New test condition `StepFinishedFilesCount` to help search for `MERLIN_FINISHED` files in output workspaces +- Added "Unit-tests" GitHub action to run the unit test suite +- Added `CeleryTaskManager` context manager to the test suite to ensure tasks are safely purged from queues if tests fail +- Added `command-tests`, `workflow-tests`, and `integration-tests` to the Makefile +- Added tests and docs for the new `merlin config` options +- Python 3.8 now requires `orderly-set==5.3.0` to avoid a bug with the deepdiff library +- New step 'Reinstall pip to avoid vendored package corruption' to CI workflow jobs that use pip +- New GitHub actions to reduce common code in CI +- COPYRIGHT file for ownership details +- New check for copyright headers in the Makefile + +### Changed +- Updated the `merlin monitor` command + - it will now attempt to restart workflows automatically if a workflow is hanging + - it utilizes an object oriented approach in the backend now +- Celery's default settings have been updated to add: + - `interval_max: 300` -> tasks will retry for up to 5 minutes instead of 1 minute like it previously was + - new `broker_transport_options`: + - `socket_timeout: 300` -> increases the socket timeout to 5 minutes instead of the default 2 minutes + - `retry_policy: {timeout: 600}` -> sets the maximum amount of time that Celery will keep trying to connect to the broker to 10 minutes + - `broker_connection_timeout: 60` -> establishing a connection to the broker will not timeout for an entire minute now instead of the previous 4 seconds + - new generic backend settings: + - `result_backend_always_retry: True` -> backend will now auto-retry on the event of recoverable exceptions + - `result_backend_max_retries: 20` -> maximum number of retries in the event of recoverable exceptions + - new Redis specific settings: + - `redis_retry_on_timeout: True` -> retries read/write operations on TimeoutError to the Redis server + - `redis_socket_connect_timeout: 300` -> 5 minute socket timeout for connections to Redis + - `redis_socket_timeout: 300` -> 5 minute socket timeout for read/write operations to Redis + - `redis_socket_keepalive: True` -> socket TCP keepalive to keep connections healthy to the Redis server +- The `merlin config` command: + - Now defaults to the LaunchIT setup + - No longer required to have configuration named `app.yaml` + - New subcommands: + - `create`: Creates a new configuration file + - `update-broker`: Updates the `broker` section of the configuration file + - `update-backend`: Updates the `results_backend` section of the configuration file + - `use`: Point your active configuration to a new configuration file +- The `merlin server` command no longer modifies the `~/.merlin/app.yaml` file by default. Instead, it modifies the `./merlin_server/app.yaml` file. +- Dropped support for Python 3.7 +- Ported all distributed tests of the integration test suite to pytest + - There is now a `commands/` directory and a `workflows/` directory under the integration suite to house these tests + - Removed the "Distributed-tests" GitHub action as these tests will now be run under "Integration-tests" +- Removed `e2e-distributed*` definitions from the Makefile +- Modified GitHub CI to use shared testing servers hosted by LaunchIT rather than the jackalope server +- CI to use new actions +- Copyright headers in all files + - These now point to the LICENSE and COPYRIGHT files + - LICENSE: Legal permissions (e.g., MIT terms) + - COPYRIGHT: Ownership, institutional metadata + - Make commands that change version/copyright year have been modified + +### Fixed +- Running Merlin locally no longer requires an `app.yaml` configuration file +- Removed dead lgtm link +- Potential security vulnerabilities related to logging + +### Deprecated +- The `--steps` argument of the `merlin monitor` command is now deprecated and will be removed in Version 1.14.0. + ## [1.12.2] ### Added - Conflict handler option to the `dict_deep_merge` function in `utils.py` diff --git a/COPYRIGHT b/COPYRIGHT new file mode 100644 index 000000000..b17a5373e --- /dev/null +++ b/COPYRIGHT @@ -0,0 +1,10 @@ +Copyright (c) 2019–2025 Lawrence Livermore National Laboratory + +Produced at the Lawrence Livermore National Laboratory + +LLNL-CODE-797170 + +All rights reserved. + +See the CONTRIBUTORS file for a full list of authors and contributors. +See the LICENSE file for license details. diff --git a/LICENSE b/LICENSE index 3adc85cb9..3d37c857e 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,8 @@ MIT License -Copyright (c) 2023 Lawrence Livermore National Laboratory +See the COPYRIGHT file for ownership and authorship information. + +For details, see https://github.com/LLNL/merlin. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/Makefile b/Makefile index a261b1d99..282767eb6 100644 --- a/Makefile +++ b/Makefile @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + include config.mk .PHONY : virtualenv @@ -34,12 +11,13 @@ include config.mk .PHONY : install-workflow-deps .PHONY : install-dev .PHONY : unit-tests +.PHONY : command-tests +.PHONY : workflow-tests +.PHONY : integration-tests .PHONY : e2e-tests .PHONY : e2e-tests-diagnostic .PHONY : e2e-tests-local .PHONY : e2e-tests-local-diagnostic -.PHONY : e2e-tests-distributed -.PHONY : e2e-tests-distributed-diagnostic .PHONY : tests .PHONY : check-flake8 .PHONY : check-black @@ -89,6 +67,18 @@ unit-tests: . $(VENV)/bin/activate; \ $(PYTHON) -m pytest -v --order-scope=module $(UNIT); \ +command-tests: + . $(VENV)/bin/activate; \ + $(PYTHON) -m pytest -v $(TEST)/integration/commands/; \ + + +workflow-tests: + . $(VENV)/bin/activate; \ + $(PYTHON) -m pytest -v $(TEST)/integration/workflows/; \ + + +integration-tests: command-tests workflow-tests + # run CLI tests - these require an active install of merlin in a venv e2e-tests: @@ -111,18 +101,8 @@ e2e-tests-local-diagnostic: $(PYTHON) $(TEST)/integration/run_tests.py --local --verbose -e2e-tests-distributed: - . $(VENV)/bin/activate; \ - $(PYTHON) $(TEST)/integration/run_tests.py --distributed; \ - - -e2e-tests-distributed-diagnostic: - . $(VENV)/bin/activate; \ - $(PYTHON) $(TEST)/integration/run_tests.py --distributed --verbose - - # run unit and CLI tests -tests: unit-tests e2e-tests +tests: unit-tests integration-tests e2e-tests check-flake8: @@ -135,9 +115,9 @@ check-flake8: check-black: . $(VENV)/bin/activate; \ - $(PYTHON) -m black --check --line-length $(MAX_LINE_LENGTH) --target-version py38 $(MRLN); \ - $(PYTHON) -m black --check --line-length $(MAX_LINE_LENGTH) --target-version py38 $(TEST); \ - $(PYTHON) -m black --check --line-length $(MAX_LINE_LENGTH) --target-version py38 *.py; \ + $(PYTHON) -m black --check --line-length $(MAX_LINE_LENGTH) --target-version $(PY_TARGET_VER) $(MRLN); \ + $(PYTHON) -m black --check --line-length $(MAX_LINE_LENGTH) --target-version $(PY_TARGET_VER) $(TEST); \ + $(PYTHON) -m black --check --line-length $(MAX_LINE_LENGTH) --target-version $(PY_TARGET_VER) *.py; \ check-isort: @@ -155,8 +135,23 @@ check-pylint: $(PYTHON) -m pylint tests --rcfile=setup.cfg; \ +check-copyright: + @echo "🔍 Checking for required copyright header..." + @missing_files=$$(find $(MRLN) $(TEST) \ + \( -path 'merlin/examples/workflows' -o -name '__pycache__' \) -prune -o \ + -name '*.py' -print | \ + xargs grep -L "Copyright (c) Lawrence Livermore National Security, LLC and other Merlin" || true); \ + if [ -n "$$missing_files" ]; then \ + echo "❌ The following files are missing the required copyright header:"; \ + echo "$$missing_files"; \ + exit 1; \ + else \ + echo "✅ All files contain the required header."; \ + fi + + # run code style checks -check-style: check-flake8 check-black check-isort check-pylint +check-style: check-copyright check-flake8 check-black check-isort check-pylint check-push: tests check-style @@ -179,34 +174,27 @@ fix-style: $(PYTHON) -m isort -w $(MAX_LINE_LENGTH) $(MRLN); \ $(PYTHON) -m isort -w $(MAX_LINE_LENGTH) $(TEST); \ $(PYTHON) -m isort -w $(MAX_LINE_LENGTH) *.py; \ - $(PYTHON) -m black --target-version py38 -l $(MAX_LINE_LENGTH) $(MRLN); \ - $(PYTHON) -m black --target-version py38 -l $(MAX_LINE_LENGTH) $(TEST); \ - $(PYTHON) -m black --target-version py38 -l $(MAX_LINE_LENGTH) *.py; \ + $(PYTHON) -m black --target-version $(PY_TARGET_VER) -l $(MAX_LINE_LENGTH) $(MRLN); \ + $(PYTHON) -m black --target-version $(PY_TARGET_VER) -l $(MAX_LINE_LENGTH) $(TEST); \ + $(PYTHON) -m black --target-version $(PY_TARGET_VER) -l $(MAX_LINE_LENGTH) *.py; \ -# Increment the Merlin version. USE ONLY ON DEVELOP BEFORE MERGING TO MASTER. -# Use like this: make VER=?.?.? version +# # Increment the Merlin version. USE ONLY ON DEVELOP BEFORE MERGING TO MASTER. +# Usage: make version VER=1.13.0 FROM=1.13.0-beta +# (defaults to FROM=Unreleased if not set) version: -# do merlin/__init__.py - sed -i 's/__version__ = "$(VSTRING)"/__version__ = "$(VER)"/g' merlin/__init__.py -# do CHANGELOG.md - sed -i 's/## \[Unreleased\]/## [$(VER)]/g' CHANGELOG.md -# do all file headers (works on linux) - find merlin/ -type f -print0 | xargs -0 sed -i 's/Version: $(VSTRING)/Version: $(VER)/g' - find *.py -type f -print0 | xargs -0 sed -i 's/Version: $(VSTRING)/Version: $(VER)/g' - find tests/ -type f -print0 | xargs -0 sed -i 's/Version: $(VSTRING)/Version: $(VER)/g' - find Makefile -type f -print0 | xargs -0 sed -i 's/Version: $(VSTRING)/Version: $(VER)/g' - -# Increment copyright year + @echo "Updating Merlin version from [$(FROM)] to [$(VER)]..." + sed -i 's/__version__ = "\(.*\)"/__version__ = "$(VER)"/' merlin/__init__.py + @if grep -q "## \[$(FROM)\]" CHANGELOG.md; then \ + sed -i 's/## \[$(FROM)\]/## [$(VER)]/' CHANGELOG.md; \ + else \ + echo "⚠️ No matching '## [$(FROM)]' found in CHANGELOG.md"; \ + fi + +# Increment copyright year - Usage: make year YEAR=2026 year: -# do LICENSE (no comma after year) - sed -i 's/$(YEAR) Lawrence Livermore/$(NEW_YEAR) Lawrence Livermore/g' LICENSE - -# do all file headers (works on linux) - find merlin/ -type f -print0 | xargs -0 sed -i 's/$(YEAR), Lawrence Livermore/$(NEW_YEAR), Lawrence Livermore/g' - find *.py -type f -print0 | xargs -0 sed -i 's/$(YEAR), Lawrence Livermore/$(NEW_YEAR), Lawrence Livermore/g' - find tests/ -type f -print0 | xargs -0 sed -i 's/$(YEAR), Lawrence Livermore/$(NEW_YEAR), Lawrence Livermore/g' - find Makefile -type f -print0 | xargs -0 sed -i 's/$(YEAR), Lawrence Livermore/$(NEW_YEAR), Lawrence Livermore/g' + @echo "Updating COPYRIGHT file to year $(YEAR)..." + sed -i -E 's/(Copyright \(c\) 2019–)[0-9]{4}( Lawrence Livermore National Laboratory)/\1$(YEAR)\2/' COPYRIGHT # Make a list of all dependencies/requirements reqlist: diff --git a/README.md b/README.md index e47f7744b..6756257cf 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,6 @@ ![Activity](https://img.shields.io/github/commit-activity/m/LLNL/merlin) [![Issues](https://img.shields.io/github/issues/LLNL/merlin)](https://github.com/LLNL/merlin/issues) [![Pull requests](https://img.shields.io/github/issues-pr/LLNL/merlin)](https://github.com/LLNL/merlin/pulls) -[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/LLNL/merlin.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/LLNL/merlin/context:python) ![Merlin](https://raw.githubusercontent.com/LLNL/merlin/main/docs/assets/images/merlin_banner_white.png) @@ -86,7 +85,7 @@ Need help? ## Quick Start -Note: Merlin supports Python 3.6+. +Note: Merlin supports Python 3.8+. To install Merlin and its dependencies, run: @@ -94,7 +93,9 @@ To install Merlin and its dependencies, run: Create your application config file: - $ merlin config + $ merlin config create + +Open the newly created config file at `~/.merlin/app.yaml` and edit it to point to a RabbitMQ/Redis server. More instructions on this can be found on the [Configuration page](https://merlin.readthedocs.io/en/stable/user_guide/configuration/) of Merlin's docs. That's it. diff --git a/config.mk b/config.mk index 1d6455ae3..83c0cdc2c 100644 --- a/config.mk +++ b/config.mk @@ -1,4 +1,5 @@ PYTHON?=python3 +PY_TARGET_VER?=py311 PYV=$(shell $(PYTHON) -c "import sys;t='{v[0]}_{v[1]}'.format(v=list(sys.version_info[:2]));sys.stdout.write(t)") PYVD=$(shell $(PYTHON) -c "import sys;t='{v[0]}.{v[1]}'.format(v=list(sys.version_info[:2]));sys.stdout.write(t)") VENV?=venv_merlin_py_$(PYV) @@ -20,6 +21,7 @@ endif VER?=1.0.0 VSTRING=[0-9]\+\.[0-9]\+\.[0-9]\+\(b[0-9]\+\)\? YEAR=20[0-9][0-9] +FROM?=Unreleased NEW_YEAR?=2023 CHANGELOG_VSTRING="## \[$(VSTRING)\]" INIT_VSTRING="__version__ = \"$(VSTRING)\"" diff --git a/docs/README.md b/docs/README.md index cbddc54a4..e427b7d0e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -41,4 +41,84 @@ MkDocs relies on an `mkdocs.yml` file for almost everything to do with configura ## How Do API Docs Work? -Coming soon... +The API documentation in this project is automatically generated using a combination of MkDocs plugins and a custom Python script. This ensures that the documentation stays up-to-date with your codebase and provides a structured reference for all Python modules, classes, and functions in the merlin directory. + +This section will discuss: + +- [Code Reference Generation](#code-reference-generation) + - [How the Script Works](#how-the-script-works) +- [Viewing the API Docs](#viewing-the-api-docs) +- [Keeping API Docs Up-to-Date](#keeping-api-docs-up-to-date) +- [Example Docstring](#example-docstring) +- [Plugins Involved](#plugins-involved) + +### Code Reference Generation + +The script `docs/gen_ref_pages.py` is responsible for generating the API reference pages. It scans the `merlin` directory for Python files and creates Markdown files for each module. These Markdown files are then included in the `api_reference` section of the documentation. + +#### How the Script Works + +1. File Scanning: + + The script recursively scans all Python files in the `merlin` directory using `Path.rglob("*.py")`. + +2. Ignore Patterns: + + Certain files and directories are excluded from the API docs based on the `IGNORE_PATTERNS` list. For example: + + - `merlin/examples/workflows` + - `merlin/examples/dev_workflows` + - `merlin/data` + - Files like `ascii_art.py` + + The `should_ignore()` function checks each file against these patterns and skips them if they match. + +3. Markdown File Creation: + + For each valid Python file: + + - The script determines the module path (e.g., merlin.module_name) and the corresponding Markdown file path. + - Special cases like `__init__.py` are handled by renaming the generated file to index.md for better navigation. + - Files like `__main__.py` are ignored entirely. + + The script then writes the mkdocstrings syntax (::: module_name) into the Markdown file, which tells the mkdocstrings plugin to generate the documentation for that module. + +4. Navigation File: + + The script builds a navigation structure using the `mkdocs_gen_files.Nav` class. This structure is saved into a `SUMMARY.md` file, which is used by the `literate-nav` plugin to define the navigation for the API reference section. + +### Viewing the API Docs + +Once the script generates the Markdown files, they are included in the documentation site under the `api_reference` section. You can explore the API docs in the navigation bar under `Reference Guide` -> `API Reference`, with the navigation organized based on the module hierarchy. + +### Keeping API Docs Up-to-Date + +To ensure the API documentation remains accurate: + +Update the docstrings in Merlin's code whenever changes are made to functions, classes, or modules. The `docs/gen_ref_pages.py` file will run automatically when docs are created (i.e. when you run `mkdocs serve`). + +### Example Docstring + +The API documentation relies on properly formatted docstrings. Here’s an example using the Google-style docstring format: + +```python +def add_numbers(a: int, b: int) -> int: + """ + Adds two numbers. + + Args: + a (int): The first number. + b (int): The second number. + + Returns: + The sum of the two numbers. + """ +``` + +### Plugins Involved + +Several MkDocs plugins work together to generate and display the API documentation: + +- `mkdocstrings`: Parses Python docstrings and generates the actual API content. +- `mkdocs-gen-files`: Handles the creation of Markdown files and navigation structure. +- `literate-nav`: Uses the `SUMMARY.md` file to organize the API reference section in the documentation sidebar. diff --git a/docs/api_reference/index.md b/docs/api_reference/index.md index 6707457de..072a7b679 100644 --- a/docs/api_reference/index.md +++ b/docs/api_reference/index.md @@ -1,5 +1,3 @@ # Merlin API Reference -Coming soon! - - +Welcome to the Application Program Interface (API) Reference Guide for Merlin! This comprehensive guide is designed to provide developers with a detailed understanding of the various modules, classes, and functions available within the Merlin API. diff --git a/docs/assets/images/monitoring/monitor_for_allocation/monitor-flowchart.png b/docs/assets/images/monitoring/monitor_for_allocation/monitor-flowchart.png index ab8d596ea1d4b4ef74d647c2f694baffc9ef5e04..4be57303ed4434dbe230cf4c055be44eaba926b2 100644 GIT binary patch literal 164955 zcmeFZc~nyS|2Ik=I~}ZJ9m>i~ZBk1dN>g(#I}K`DIiTR2Q;I`rs7RufW~a%_a2_f% zlR-_zSurauCrkxrsZ1$RF%d}+ynE~U{_cJLyX#r&x$Cay`MH*h&HljNpZ)o~$JhJy z=AR3;Xa1DgB?Ey#{eig)-zOZR)=iIkko|mf)P#MZv6KEuonNl zB%nb2w=k@##s4S5@X8w&n2OQaW zg{_^tJ3#?5a18px?8>*O4Zz9!YR5q%Sqf?PhGdQFvW!O>=H^MDx#?D28~JD#GaX59 z7+6>O(>-d5@QLLH3_i7q+XTFLrUVrJURt9R{WSwLN|(2;nxg@qGfC#Gb*0$l(e;4% z{al;0{$MXX&;weU@jsjez6F8&Znb)~)nm`6Yajpb0dRp=|2&_L4FJ0jfwa2{XUl6r zp}`3UchuF%ni~o~3gMjV-v<7fy|KTQ>6@cF>9}KBNi15tm888q7))IPzt4gdSm~k|}q>72Ly}%LW)n0RJ-Cam!ak!m+UH%tQoYGG? zf2w3o&<|=vla=}-@DVl*cx2NGMP01|N|gDDbn}(r-@@0sg)~flovn>wlYw)O1tZs= z4r`f3E@9j8=S!#B8gQoxA8Z}5A>zYM*C%m_98Y00b=c}xWbCZmJ4vMEk!V{rca$r&grctuUL|!+$+|B%-N`)Q zRbIRPYXqj(G)W6hiv*M_sPgkUPc1^7P}|R`gg3JxcGQ(U{TA83WGA~dhuM+z>ybo?* z8p_dnwDd#~aFe=!C$oLc;kVR=ac7Ixl>y(%)tQ0ci;wYxcD>ezI#Y*CiWvx8#3Bs1 zQ7Hksx&wz1DEqPb&n2gC5U+*|#JX1@pic|FIyGN@57?U?aql?&+UDJcWbBZ8pL3T{hY(XwS%%{|+u=KUXmF30IWuD4Lh!x(Nlpoq32k^DyVK z`+lDA9#3#>$MRs;r}R?~>Dckf?P9~_pY`lBoIV;$mvwFEmD|@Ft*Rv~@+6XNQa55f z6#K9@5QkzKqFsnXGbZB;a_q%^!~TPNE9~@2N>;GehJ`C8-_U*)6} zQBiAGjE!+gWv2f|;2DFo*DT$Aj1KwQnB9d}2n%%*p?iSR?|QWI<{(?kJak*-M}F&i zL~*UB;O_QJLo@F&@ek`WCy0%^vyRZ_Zk@?o_ly|@y`(bm5py z7A1qK`x#dT1J%tbh?^c6z17EUJ)}+WY zbiCe9M?ItgA{zoPVrRsJ2@1`-H@&CM9+J zg>&e7{GK94u~uHc7a>$p$bVHfaJ0y^V!oChX)wzB{(VT^V7zZ{>awIM*}F6KP7D+< z2PsPjwp}ZeW7oi&oof}yIi(N3Nxd)IkI*H}{?ZnC??SU46FsOf~-~-6;06n|}j- zIY@4*ZGf~Clt~VsbXoLdQk0eMnWRa#2D*fTnJW11CtC)fWABxr zC*5bwk0aCrR6h5`n)hghGN+s{2US(6kIoGcWvM>*XJB@yPH~scX%AA4>bT0$#hP}y zg5#%Qb;2j%yqE~TF=5OkqIOWk7h}(;{YacZ5tyno{advMao}A>*L(qZm*z|2DGO-9 z3xb=XvwXkS=E_Wma5VMyhL79*K2&7#!4|< zU*)oXk;Nws*rdGEn;;aQGIQB3>W4!;i?R=AK-tUToB!+~<+p`GPns^W&R^TSGMO)U|`G)ioubYaet>I&JoVgTIAa4kjXer{~V&^iz*kpKE!3cPP?A%ZIM|J(X z`6bYu@h&^U>m-lEGg~XA%nf5ddF5dbjM&qw96wU3Ls)IpsPF%*>W=A7Z#8R~Jp6;ijIW=BmIe*vCZYvv6>ZvC(~l{i_TgGAc^H;kJ(wHo%ufp?QN ziMT#{*a^NRrQ1RAZ_8s7#>%X!uLE_mE0x1Ulvif_wA}=8afahGHvFfE?sDWb@vP~> zJKDA7i&7U94$-~rvOe)6IC?uG*<5$`2{>I0WC%GwmNz^kFDp$dq4vJ77nEz`t3{t4 zC{Ib2NKOyj#clotQ=Y7hP@|XyH*)_s&PwZhl(Q7zZw?*bXbmoes@40Fcq`PGSD}iT zB{9C)AMi8qj$55K*xsru~PE77gM$ZxYbn1p!eu-$d zGym{-T1ixB6*7z>tlLr)mwKiD*s9b3*@=lTvS`~tUSy;3lC26PlY4H_9d%LF89}1i z^PQ8q<8)dDx?*-PX(>kcb*hDFaHxlk&5kYF%UnSKtLB^W3hF+`e32#$eg$yt?K3Pw4FU_b@wG$=G;4 zmcgP+YIlq`7Xb5VT6~tE0sti?>zK}eO*9o6f^U9@vL?Zz>laJE?mY=#8OXsQ2SS%nk>>HPCwX__K4dNK&#AW&OJ&tjzMT29KPx z^s?8fKCQGXZ)N7jKZ3`Q3(N!Fn`7n{!^%aQ%q$0oO~;}m{amIL~1jsMDNgEFg@`J;;P0zG%syHRvp==id$>7 z+mt1eA<$epTmGAd0`@G(Y&BZB+`H;qfI(o_|2vlim*M{p>9uDH;13NVAwju-F$6X9 zcoqaqfSOc|V+F;sah<%ys>T9PRfqlq7XaB^3NR4PSYKxhoc}M#!7cWq;V59XD8G4* z7X;5?L|q_y8>hdz{c$!ClUcUhM&8dDMTi2o-Ko9`ylBgs7jce*3ub=+X-~+8rWVVn zbNnN~So*m&Avyu1p_d3(A^INx3(GUu4h;E~{@=KaTVmdH2zJhCw3prpfFb(E^kWp! zB0udKVB<$Y|IxeVRD=G4K30sHc8*;51@+R2Gt35`o6Q{?1i}v>d0%WOH|E(Cf?!Wm z(AuNMWlRse_Fm{+dGQBu_R3V$Nf4P?{cGB@cG1;0o1UyL+1kMI{~iDq;(uaE`V>HE zg2c(!FAkx>kOoq?U2YXt1CRe#E6dJ`t!lQ{Sj&?Jb)wn25>Phdo(=vS;;e=q1IKC0 z6WVF(E41q8nrEj-CGJ;8?9ZDrVgz+8H=$|kw}GK&ClgPXe4TICR&GFyXNPymyBUNu zE=|tDJ(jGh&hmF%pHIn~9{c$)^4g=P-cm`)rpXPUaF!z0QsbcNc=w2zA?kvtE62Ew zofnILe-)Y$GwrPmKt3%Bp}Q0O0->p+u`6P08Vkp5=sZ4j6pB_kjyMn>G57h6Z(iCJ zlsya_aU}-~|HQuUNSEm2{i% z4f%P<{=v=v@J$y7yxZ{}!A@ty49K{SWzrGT^YL~DR3&1R3uMf0vtV8SBAgu zIEYR*y{w)tYflxAg`PkzFBF;sxv1MZGT*(~8(k!m8j+l2Cj9qG%uFTjakT7cDrg@bOoV)+eYTnMD=yV zL(LVf65&8thsm2_>|}3AvHp);;v?ZBCWJO2+B86mzw5;%7c5PB6yYMKn75elSocNl zGuRX+sRPU8_yvbuZ#g*LK{RJ!`$1jaskA0p1KThKrAB`+iD$QC9Lt2&wR)3m^OdDp zVE=w-)qa0oz)Y!&nAxS+FY{_*U|yI-Tgv)`3n<87akgj5!@04%EY`GNGH0Tevh++ydE~nybil#)l{gwk5g50zqZy%%EGq;HES+6X7xw0M1wuSW4iXV z=!O@WllqOk`#5Q@!#l68Yif*7haE?E#`c(mb=I5DHinDi$*4S=TJHTcItN34-P@I= zax6{sab)VqJf8V1m7l@>)hMmz!ahByfIJDzH!gF{US_?N{hR;xthltteEF@phPbX` z1->$y7$Y24x!zkML#RxXXK&5>^{xzEdGj`m0CXV{@T;{I{;i6zN2v=AZ3YG4&%`eB8n9PfhRC zyMf5W=wRWDrHc+?B|UrPLFjew_qs&(MX5_-ss@j4BIMXR4H5-^cXjeOd1+TU-KZ{( z`MokG-@UDV${!KyjylWa5cuRg`-a;{PTA)0>E83^qxH6C9B#g=HV0J_{IJ~!)bE9c z*Q*+1^Z2u^d6Q$a!P4Mn;kW8|f2$_d6Xg!hM3O5Xp4l+hyfHr5_q4xNbIonuwsELhH$w4oY1S?M}ap;|@Dgr>%MG}es_EZfm{+3k}xnf3R@2j5K zBf`DJ4)NzVx8Sp1lvjQ#r{Pn#AmBUs-}94-U;RQ9XW+VMqGwf&(e);IWvWXV%zN_t z*=*!?%pV%z-|6OiDX?SKchu3zN5UGyJd&~OvG!%y3w7KPUY8P_UZ>pBMh)u>bAR~5 zX}*oG8)Dt{qa-r6_~H=uv|Fc!AUUSDmX2zu98{Pv^lhx6k$+VCgO9!`OJ=pq5tJ$I zEOdNpb%&_D37#i->qQwCwzf0~#*Ae+2d3?vmWabb#t8}^k(4(TC>Fr(wz=}AoI0mA z+<-yUZ{$fo5X!xSX-p;FnCIfRP#nVB4aHj=-6$GD^ngd9(UT|WMRh*^Ds0EH@5(id-;9Gn>WRJuUxMAElaTcpEYCE zqT>F#u{7~h*U?;EIr`M=mZE-rueTCjyQ!}mPqFCMec)*zYj}BS5zI*a%`;fzkHiS8 z_6Gdf!r1G{bk65<@+60&*6p}^3qe_-ylJ3MTIdp(zKXrbPiPAm`#n451*xKAOT ze%ic02RoUOpNHA0?nSNb^j@iNG?MNQ6%AyLA1!#`EpI@7iZUMgp+=cz4(Ee`r8N!)IFiPdeyGQP=o?E}WsE8~koFe_rgd9o}u zSw~>>N)8J3b%%<4ip8n%Dt8>7+`S|wqE6ks$|uE!K$RJC!+xgPFBAA};sqNtcQ7zd z9$Sp{Qx*-hp61N=rEhVb9I8!BXv7QJxa!}k?%htnYv=fB(aLi3bKbcJNX84D(_J7hl$El3WajZSqJD} zKdR;JrdUedFr>*YP2jg4ur_WtdPVR@ZPH5}_kK&x&FWw9iFR30UblWTLQP^SCwu?HXUTLzj%(GtUiY8#s**l#? zug|pmNm-FpcshB^Rw*BAZ%{kB=upz&3k@ldaf3Gd+W0jj8Z9_7YqUAev(HQl$@Vng z%N9<2J&q@0wCtLm4y$Z~)vIFK2GqKDsU!BWvEK~&!!FElFnaA?L-=eAicyY=_pJ^W z%Do`cOMbKE8IfYA5(DWB2XrP4W5R^BR495y@?o~YgTg%s9_ zA1IO?Wbo*&kuyVjM%rkUSxG;fCb)HGNeue)GVhlOLsOU^tk6kh;)eDiL2IXJALofc zD7Y}`PA^n`uFkXcttR`sPefy!HP*EV>m|DBdfoaqicKW=qz&;V7taPWBT%Um351aTkiZLIfeBHtt@QjlCj+2kCIGDj5 zKZR``G%q@r$`a+Nq$4cC3)+y#D%bsN)pL%llf_O_{xu=RNFtNKYikg&rw0K_iHzs4-%H1_7F2817iU27Akes-jV zIZUP+ZkLC9+sgdqYg&d-*@NigjM%XvXDJo358_`;-Ss}z>|K8QmCoshc?Sd+tM=HW z6J>W-cC}~nNyVM>8Z9|#m@Gt%7ANN^vl_c7KpqL8Q5U0E2BbGw?<9}yFE?S61G+M{ zFM>w@_Ng_fYWwpLq_dM8eCnxkY2gSPFBYgQI7Pm2jq)pC2zfxIrm9hQh&lcrxMej7vE0habENAU0MFY;Xz$L z5~)H#@*@XDrF*d4U6CtTb2ei<^)M>nr!#yn`6}B6Rh8YpIf;_kq06!N(4`wZdvB>~ z${smyhNfr&(~MhlP1$y^cUBPfXBl|-JO0k9`1(oc7auJ#kGbpfA;Avbn0H2d1y9K6 z781U+QFmaKv#zdiFk`4EIM+@-6GsOnv~VrWg?L&HnZ6A6PbPZx4%vbMcWF$37ZiAf zXjsOPOBf-m0K0wSS_D-3qQX|A!W@KUj#1Yl1;{U_qT|fX4~kX|%ay?4|KuRxH6XvA zSzA}|_H&<|GPBSFzI1h?8pL2)uGc=D8f`UOJhB(`y>Y>3*N7C(VM`Nfk=&?OCV%R1oxFY2IT9638Jt7LLa?{7q6YnvfB_#e=)Nk@F#8UDAb37CFe znHu`m})>`7%$|9nb+_^Lo*A4(&TYO<~AF(zQ;MX?y&1KLM@=VvVCR z01K&QC(iowWti6gOc|Mz6exH)R z(yIV*&WeAnd@QUUus3nowN2Ql0xS%N{r{xPbgXe$oYNhYMWf*W)u%}q1yU1$c0f=iJAH3pQJooi%?xIa@LwJg;|>Y3xd|KneaC9|FHT@$OYudy2OtTGl$Kvj;^ zu3rvpU3>)b3^ty5LmjH~XCI7WII&_IPD2BN6XKYIp*Y@mQGfJpfG5Fp!?nI0J)Q2` z32sT}zhtU6lCR6AqmgEfq3=b|l;tIsQab!7AE?lA8n3^{Io0P)xp8Zjj#s(4R&ld^ zX*4k&o4Fa!W+2ls=uM0U@QFE9%mK3{uE zo!1sw?7JVnZY;XsrG6GDsp#t5hmK2$lTX7c;Jls0c*sHt?#g_U|jZ-eCv)k@!N6(4;Y;$}DD zthpN^;d?@whdebK_xZ!mkfGfi$S^?6G0o(4>DqFNhXq}}R=bYJ+IZtxh-ZtXwjZ017y8lJzQi~t-2dBspBVb&8mu)0MQ_- zuw$>|XB6)17nVtHPE-JfVo^9cd^BSP zUkar?!)z_~q)3G$N&9YOAP%dy`rOAuI7#80T9=ZV%STs6c-7rD9DI_J`6u!Bmd%=V ztoXr6XwA83r5eAffAohaQ(zA0imKTySUUM+T5LA=Bkk%^ZdY9%AW)uv ztPx9`V)P0paIP^81yIBESAB zm`7+*)XNJ|n((a_*#!wx6>V4BTa9Dii!z+hvjxGvXgWKZce}17K}i?S*`?xI^T&j- z$@$AiI9XRlBMx(NKMJmm=w5a_e*fx>6MIPkO#e=zwYAGvlh?lEXR2 z&ljh77e46E8wU%Q!bmA6OhPI7;?Sp6>W+RM;uLXsvn0B&Qp=GQc8*zK-U_oV>Tkr* z)Jm2D1#k0PCH&m;I5FFLRHKJk;y9zhdCYuLQ%H6s9-&lzWdP1Vv-4;1Tbkl%_@pzF zBq!vQocH;Y6K6A)#S{XWFN+R*ME|6_TB~s_8RmgL&(2s7*tmVs=s))=lvx>%{T(y# z{ljZ^9hw^z74TqwiB4Pi;>&qbGciaVDhPRjcwaZK*j1k|cc<+-4VN(CA^PZD-?3krR_B8o@DO($4Y{j>u;Uw;a*_M0jPO8LvZX%e(p!p z3E=kUW?xS)?c;=j)Wex9W+a|Ty9%uO;cfpieIIvQ6F16H9$6{Qsx#7GemsWRqsCr* z>$H?%_~f&b2fPD8YP}afjGCF$$-_49Du6jisZd=@91juYl-k7r-j{K5azy5jPFNUv zlt%`lE=BkloOcL@|A%2+Jc{bJ$=kPT|Ru53D?eoVq709Jx-_o=6+D_ z`CW&z>U#TU2jV-`qrCwruig>nulz$KCZ^NZ+D6mn!Eiv=@W`{ za9w6&hoNKI)Z9mnb!Cj(t+o$W3gKgDQ#x$Q3f+h$OgW7En4-`sf&M` ze;%lHU-YEiMI<-%2U0BpG)WCBdEt5+ z!TresT@bDm|_S-yl9R_h^eXQH#MnqyD{PmuK5CswKP93@377l@)1ZeMupUwf&&h^UtvtroE_fX5(-mLg90uY1Q&f_L-ZMRi)Xpng^F&D3Eb9|6o7l z%#_P)aomDNCSB-2{K>4?G=q17KcmD~yrgckNhs-*Ri40I0W>*DonW)Ei!e6~kV_-` z5Z7|IT$qi8JM44`rRZH=4+mqHz<-qz;pA>ry~^vEPIuUmSxEU(?}$NtDPmNjUE^n> zXuTG$JcM%A`|3Gose1u`E)B$*FX{m4qjj0Vv-?hD4@Jo2eZ!+`SXcE$_*Uwb4&Bi8 z391P%5%4!f(H@G1i`UC<_SaJ&`!#p8j-H2C=oKk?6K4o_aq_x!IGqOsJ;i@8Re}~1 zludN(?;K_eCTgxPcD;o+XYg*b^)1kJACCi`{XIjv!v*VS0xOtofV|#XycwL{*cu6I zJCLx^a(^wEFj}9>?Jjz&d+Medru3D=r6DaHmeKJiqFw#OIUpKvs*ePsh|$x5rf;=9 zuHtzE3X6IDlqx3bfC)p!-G=m#`FP6N$*r_-2?>9+2;^C%3*3tKbB8rbdUDI8)pcp) z(JEPSEeQBDrRe zUXG|(v_3|Xs~r4}3we-tfd$Z=xIJ8~Qb2h%Qmz!G`Ir3++Vw&}COJ~EOTJ&WTzLQ_ z5!S^@rX2H~hJq#;GW`Wst|YsWK$pKNXU*xJdU69ym!fk6Y3v@B_Nm2l6uyC{ZV=)a ziNUO@Fr!Rlq{i?8`d3sy2`G?iYe-S)`UBfi4*v~bzJOZlGtkf)oz^?2*dD^UgXdC2!p}ASdT-Cw zu&^hid4ZkU^Y`c5mi2Y6_XaMdH8~L<`I;Z_;Q{%Ni#@;GBt<&B=k#H^3XViA)J>B# zcJV&C3d;<13t>+?oRNjmw91p1Xpu2lsQ(-%tT58DeUtOpwBt)D20+e+tY9M$g|SM<=h6$=hrC|4LW|4eSC)A){#!21o~s1bk{xRC*wy$j)&?BTV{`*CXk7 zZNdYkk89KpI4QG;Lk3b!gwH63Xuo&p9{8Vk0>62f+m6VmDECpkSYUu%1^X zY)YZu_H~In7p6}fCg+ZjQb;#qfCh#-WQ9d$M(Cn-gTKB}hXILfX%UP$RYB&y&>Opd z+^1sE;fbDmJF5igGtHRJ>OiB0J%MTOglTGkn^$=Up;ADV&a(0>a!|;cd}m$qka$sL zTbbjF$BY*cE5(P_Dn;=h{0OhX;J3E(F$c_0JKJpDf5b&qxP%UFF1oB|bJLSViWUS4 zxRUm__M{uW=c`)i;aPocRKz03|pV(gA{1LN3INazx$&RE|`$_a{n@|p2zT6c$d^XwKG>1}O z=-5k4i+9jxVU2c#}U0 zG*Q$~i>qk-&iwuZ0O9_uppNaniYR(EG&}=S6NvsP;vG=j^6y?M2Z*u$wSt%#S_A-- zjDeEdbo#<7*BYY#2@Lez101mvW22*kp)T$>romo{Ub@6<+b-)hNKyas8g#93916C+ zL7Mjci*JKBI(%VGhrwgWZ)=@BtH=K@2ly=@CImU=3pAQk%mT@{fe;J!P<45j489hx zc3A-dm8noLvq5Cn`4RYT1xl@*l$L=1y3st!r;|~3o6PU9t*;0(tr89u zK1LA>`2__;;-xG$h&KcLqFR7BQVT){YbYzk_9Arhg%tx}#VLw&yFZ7Th`CXMY|e5^$Bp^@1^6 z0?4*wHntDh)~02rh;`{yA+D0< zf0yotpS7DPABk<*6`9A+O`kAxDT@_xJQ`2(ieuWrXrYXy+gc$L-1#iz&|vLK0BcEz z{d=jgq<^Sta?FD^G7dD6mZ-mr07wO=<>scScU<#3a6@C$B-&PzIJdIViTALa08IkK z?(oL7=*2LHhGXA_SX#&_n+V9W04-l>y~^EqE7<(mr(`oW03*R;8{|UQ2LIQm@BPSo)ob}fW8>@JF;fbh5@;D+&onzX5-TF@8$aOZ5`r68HmY1zpB3{0R zsRLrSGBbd?CmjM*LN=fOGwIailb?vXl{V_jgw}n$kKyt={y;qdF{pE#)CLjQt}{*L z;V1Qfn9XnUuc2F__7*y z37($X9$n*ZmQ`qRzW9@j&4f_^z*LJlpb_@@wRihu))CaYM*%z={tNtJf4%WP7E&>?>+v9b*- zuBoul;$PRTl1>ovI**AVgBs)C(rKdh3C0A>_Q!_wvKES`HR`i3c{2)S6Nb1c*=uXwc_7=7oxLoVlKZPe=nY03Z zvz`kQ5aGMq?KZ2ex$gW}E4;ns)h%a=X)>Ri*b(OLE^cC0Lwp||qWD;I`l*QX?ZU}M zwU`|!LW?ku&vsXKk=*XML`AdgZU)uh7ms60_yVt7^$@3Jc8bWDqQRl9^Z4~<|LEo< zrMv+Apa46;1!`|mfeQlabb>I$kJfjhfhhnQ4OINGN)&Fu;^jshuVQ9oDi<&OQN$Gk z)XkYm!j{1d|J<5w$J*)mCh~IKGEkNFME?tJs$u3X{{`D_IYHj{ED>Q%Gb-(o89T7t zm-h6%=4K_u^}c?8zE~)zrRUIo&2Y@mE4yi*c;uR|{NiIZRLi$3hO)(#;x2Odr*`6v zZfa+hY+S``os!xON=Ho3vGHba=*w+2bd$UaPx38i@nMHqid>M3B*wx8e3Op%w4I|)zfT~aY^Je+2)B_dD0jMSjfhddlHdZ zQfxS%2DHpPTZs(X*k12JVsTd7B{5&${WF`e!+oc?gLEB%*Disb&f0Eatx|!1MOHpm z2QdB;Isn(G|Ib?9Z*+Ex*QYNxrLjT)o#qjLLt;D&=`Re6Eh0qse{89um3PMM_c)L z@F>U^zSsme7ws*pb(^RbOaghG_yTe%(!!S_Kry=S!ZfTM%ca}qxiF1(Fjrx&ks+Fl z&dOzb6VIaLa%Bb$!X{*rpn#jT@@nMe@SvwZ{icCLixD|zf^^7p5XNiU1MTHD>`CE z8aqYkU~!F7i#P|p{sTbCYd~NKyCy+3q0;JLG_nhzGor zG&g2Uh%Yh^2R}foOtsjjGt;c=} z`C~OxG6-IW!HK+3#pvtX4i~YrL}&RW*cQXNf!Dr{_fRuOT^#!)$GX}e=VBF6gvQAL z%NkSwH%+gThqU&6jifSz$DX3(y~p(0c^`FM+hhF98QxDG*}45m+3nJNSB*rN+0~FY z=Myvi{M=C0up8;F*Jw{zCekG$MCINe$({Kgr=~%g`%f(NG2cVQGu3mj`U+CYXNT*R zH{aP4X&ze6;KTeI^W}g30Coz#qOd9;=Wp(;au5Ic%Br6lX>!!5`Vga_#W6keOhzaw z+?I6r$bpL%wa7fb4g+P)V(|puw}H z=r><^H7?=G1xkqZv6*t9W}jFP>)*jy)Lb#?-&NHDbk~#z(S6LyZ$AQP(czZeljV_0 z?nhdpoeo+bq@?0X1aCE97Gwg@JDF{8XwR`Eqix$)1RnK~$f%iTY7*ZCT>`u8rLUrH zVzf>9B2PLq-3_h$ZK-p645SmeY%aARHZj=F?>zMFt={K;bwx3x3tv^!oF;k<1tb(;8Fmm?rM%4VSXYK16pw^*miey+7->}NSA{F}YCt?xUp5K7G*OQ~ zRaEa}e(OVI5jBLvS46uBz#nUfIA`v|1gfnx21|v}s&5OO=NB-(S3{7w0gcKOab*tR zrur}NZ~5k}EVIvW`6qp`gh1a3{wNn8`D4W4Ae^msp;B=u4nPh%*0Mm)trbKWmE(_( zGyCxiq>IP)uFHr&k0R9S+uek%EHg-{4QYFO*%*yi>|GMbSv*kKj#6)gy;5^X3 zT2y@X=iS$n0qYD&usbf4j#$@?Zvc3=ZUF}*6hR4%KowgBZjlzSAC-HOBkwa^&!R0> z`dS=EOr+*9u2sqn##gUb9!)$lgfZG4js zIvr7!9gSkNO)=42vRVmYeEG&<1t>oA?UtVmN3PE;R4!19FI|v{PmH%d)Hjm@l ztLLoDH#t5WCbyX+U-{eZ0V)2&%bb3hP1FjhYtW>=XL-oxYvAuY++{wXJay7Mk&KU1 zy^$H69bf-4VW4O7wKgHoPEkW{LI|@c;+AL&hGbqIIGgerG5Yn&P!#XBE9%P|*L?nn z+WI_U&%<(e+I^&yA<($&WG$Ch7K!qz37|ivhRDO|PaO%Au^k!SB*OQ=bHrcFz@+p& zZXE7$Xd2}dyd(9ILk;lzZ2J07^*XC%(7yPB;OQ2(m4WYEw74PDm)5zp<6ED@P^^0l zpJB}fVqDM-I234j>D&OgXr6W0)t|_;R4Fxb2-cOgh6rgOSM%Y_)76rFyoR(2FbDckkHM#=Gq1pY4j%+nW~u^X2-q z$-y*aujSsSMkWMYHe23DtAFh072WFe*Doo9(Xp*CrQMMH zv%_oeY1Ll7cKxP*Kng%KEp=SOT7sU(tJ(31ArldR6FrmHu);BScfMC6eSXzp;r+}s zfp#f%L~oC#AZ_k6dRh9i&Mf)ML*Vx=CWn>%v+t*P_Em_wMhZI*2{It|Et!Z0%~oI9 zIT;^b$DX|uxPYlD!Tu>I26yz$`=3*V0%C-x`YD+kdW=4HDhBU4Sb;C zFH}66p}hETed`Pl=<;q;{8v6!;x#H2QuH@rFv2g}JfwxX}-`PAKWR zcX`thj#J3aq0pBS z9}Z3nIE+8cqU6|(4?nz^A&iV7GcmL{*Jym1LYlYHERy~isXUOe!X z@)fXRSME;v1Ux`W`7Rj%FhTU1;oA<9xb$yB1{Vo|KH|ejB0;WzF`;Jm^Pf*Zd>zMM zE1_!K2c{9UX7fzF+# z=(Se;|GG;NHx1~9e5I3d7-$s!w|}vz8IZcBmMwb}QZ2p8lJE=pqhKT5CmOI%sQ>X{ z^j)DTF<|2yUf7$bwekOA>)ivH-v9XV(d{Uyk4{Q1)hS&tisO=N>XbsM+|4!SK7_en zBXzovgSk^j$Y#damdj9`5Jtu3GD8VvOc9G2-`CsJIp5#+=fAb~e!t$Y*Yo;#KAz8a zs(tADTdW5fLz-BlBPQVCzpYa;51rl&lR6uFUo+^_o2M!^GDqq_uVJle^lWM&DDTCt z@9FxQpF#^}>936HCq+w?{(-sc3i`&5p4S(G#X1W{F8)})14_fye#vBrXzQons1)Hg7+u;Ncc>8(^{T1=;Y=OU*5DLK>M+Q>KEg z9o7gez63P8e_!qA)s3`u&N_UH%Gqt^)b}dr>BD=$XF}vNwS<`B8o`>H#%(lV2_;)* z&52v3zV;iBUBF|Mf<<$+u6pNbZJe;XFtq!ckW!C*=3V6v-8A)kP22x+x%;N%@eq%FZRxX2l98? z*xHB6L--w^S>kJ?Yt%2$$8I^UOr|r9$NHtZGU$68cmvt|F-<3pwq;ag)zMT7ZbF zxhxQp4Tnw#e?M$K$M5I$QWuJFy={gKp(N=vpKkr!)zR(F1q0psOP*m}g~P|Dl1!=G zX*!Nq4XKu|7Ruw1AmY}!%Psr9L1LM<*aQGo^sS&xl+FZADnm-d0la-zQZU~S;#EuK% z(}Q0c>U;_Ep6Fc%q)e*#4u@gsOeu#5W3;KCWFeVRAneK`gh^kc16K52^@+|roG)la zO?ce^8jh_fwNzv@D2YhR66C>pVg47W?@5HAD^ZG%bE=1V->e#~WTUUw=*$gz3BiJ` z?9-`33#l+W(N#*a)M0|`^(9%XSw<1 z3%Uhe>b-;zW2foPR0>|I;8RF$9ony#)}_1JR_kk?yhVmP&YS$Y@8kC%TuVj2mKsN= z--gNHoE$J|o`~lz57nNkNMAHlE=E2Yyz1^n1{t*&$z2}i1dz2g!cpCB zy@rn(YpG-J-d(fEQ=Ene79bq-BNlulrKA*qIq?@*&%U?>zmvC(t9{YZS=BeA?TW(cqSi+P>oVBg+{7cxJ5-O=cM*0{Ayp*{M` zoGDk7(4!(tn;#=3M}GBU^LAbLYS0(TrwO~FT;6y~)wYbgxP+8#GeIhfCoSEci>ZIV zSN*J+NCC2jB~)$OI68>0A@p|vUw>QN3DMZ$m>iL0_2Q2hZV8vvNmA5+YX?f1H=eAQ zEUUqX+adWYV)dVZA!=#xHmCo@(ORVM$J`kTDredS z^{^%-Y^ilap>9U{ylc+8%^{Q*cm1X14r34l*5eGB!#v9B!f^Q}K;({n|7dMEOOv|X zogN7tuT*X%d)KP(S#-H`%9V2qMOJSP5xBfCJU-lvFlj`MCfkJiP_~PkTx?Q!oErVS zxHX_bKVmiUgtdOM&X=V88A6LLdwRDl4ymuF7j1azm;O%j7i(@;RRMqe#Uq#8)3(h| z@Bsq{R4(%wseuQXDo>^B49YjyI5P6M91PVigz&{0uO~)ru5=i7>%Fgaf$80`D@gjA zR$ae3vOYj&$j7F?7@yyAx~`gH;sGS1q_`M6VX8ZCPG_i0!{-1qd}Ku6?0n1!{(rD8 zs|9guN-%N^E3SXb;f*1%u+eg!lvbb zOIoGu$IPSuV4UpC$fp8pP)xR$tu`rxJyfQ39qwEQ>}uybG741yPDpZQJd?|8S<+&X zr?cEq%-VD(r^6naWse;-eZ-}6a&i>f+uPSHPBecBZ2)5Gz2a-rieNrx&upSKncV!J zXM`oixa{m+*5f<%n4N|`tF7cr|&iQ9eSB*(gU{#Ns7W+&ubkW!1Rj#L&yKdK9zREBk|F!EtMt7rqBkZlg+7T0kzWDW0$0JZLvm#kRFM5=u`)yYVKuBn!>M}&S_KObEwflog-O(C_I0&h~7Ef#716DqX)nLa!bWf zgWIN(>dGnERPxDa@N@EeyS9c2yUC&94xGUTSJdOK?@fF19VohC%To^~5DJcJ^a5N4 zqF|1tGNT{bHG`MBo%8$^f|2w*czo8ktgc`qNkyHswC@c=@7w68pM8k5_&;UwG4t*@ zEyufd?PWVK4g}VSkre~ycF^2MrJn+eAJ z`}Xk#_u&Ycwz|-~TOU1c7`OBAVm1TaMOicvc$pJ>7wS66w=8ldSOmvaXF4V@ z3bybu5vEuXvGMhPEYHj2n1v6C@`zIZREeNc^4!z|y1b}`)wHA~>U|~o#1(GW<`gGm zRaiRA^o7BEc+T#)&3HXBa}wp%rml{Q9m`zUm!nuBdBA%c9BurI+nb$NnY;xe>9ghG zRz`8JHGN>*;LHQ+nY*a;)ptXI4`DV4HJn4s%6ONr`LrbEZxD;n!vosMLwxN#FW$nT zkCYttkWaqG23Pbwx&e8b<`>CVpdfHvK9;l-ygM4xG2ZZsZizCZ_WJlY3Qcr9b$w;(Ve@A5Xu49_9baR${7Zm^aDzaT}` zn?+8l&2N+0uGL#LXs*DV44z4(+J-ca_De9laXprP2y6&SBQ~9lycub8-i+FX-Zmsn zsMn`=Wi$H*8=?=sQg0d4NpwDV0VE5&84D2lg6_bJ2YlXGqC9VX-!TD(em%a$ub?~q zGkGuGvVDo3xDd{oK9CdlFxsLk@nWNwGjsdZlewEqIT)vR6uqyU$@-z0FMS5(5vn5h z=}dC>r2UTQmCtblhQ+luSlMDNn>nqzZkRRV&Hwr$MDE(x1>+SJ71PX0F5@@7zFeSZ zf2$p=fK|BGU)>~j_%!B`xv(^MI4YuM&b5TJ&lLVVapQx{B7aQXHeeoQybJfeQmW2i z37;7K3vtGS)?}>eqb@oi?99J;40SLSpEPr$8|=|8-=b@d1ABd~5Qui$#<^#hD;`Xh zYyFDMdDiOuINu?mej$Lx7dD%oTkdT+IlA!u_Atqr88GA?A^t0ClB^5F+o8s7<n^;c92G|6LwA0m52H4y=-c__Op*Jrq|L$GXK$0o9 zuUO04?1e3gNw#SsH8i`$qq+vCs$$1nyNr+e2KQLQbk}qFa zSZI-{_!vUg*z0Qe-fw?xNo35z$|S}zpoX+beBjKk_(YxXTM7d&EK7qKG905%2bs@o ztl>iWWGC8$UQk!!ykE&~2i4OSTU2YeyYSjEAy- z+f6%pIrMlSo_1(QeW6xw!~CBiRue;kG4oBu=`Xw-!^WEv182r-Id)vz&b(@;!6dR# zY-t*awUFMAiAI@m16avACu*JZkLobgyXrRBqT+DmQy3&ly{lnq|H?D1&Qn9DIH#0Q zy|2jL*F#QJ!dtWW-crouik`e|=G%uH!S^TCQwN!lo03$x@ z{kF_6eH4P<(<^EIFJGD}TE&%Zv;fJ;@zvy*#*@EvOY|AR{+7IdA@<ShlO8ii1c4a+tG{cX_d|51|pMt`EkxcwM5}qQ}4MM~p6diE)D4>AoPS zf!W-27KuXMR~;9TR-14d?^W%>i&f_*WP&XYl#=(5lTO2iqJNQ}m_P6s?>1K58kQBP z#!Ne4F@A0Vb%ixy^wDmy#IzyeCE7`u!Ae6PJ)J&RHLP}k`A$Z-V+lCwCGO#%EVfBB zhF>Qc%8FzX2RSF4$;T&#Xh>Fl=XfY-lUUuvxg?Y?i&@}r->g2T+}^gre=Boa1 z7V-~i9UW;^?p>A!MB@x43j?BpRe9RnL{~;>X{kJfl6_n6>fA13@%si55sfF4R{A1{ zY7)X*NkQD>slqoMU`80&qew{+M*U3r1C+w9$j@SphhyJL-0>){LxtH~Ry~|?rN<3* zGAH}sLtnHe(poO)TZ*y+>q?4zOHr@hIl`#UePS|dPFbjYkVuG-JkY+R+L$wOdEB8< z{Vx&ou|A)3!ue?L7ry;1<6(cjxk6x6T($b0tv4)gWb~!HR8o{(G;LQ(*dSSwtVa%E zb1K!mAL|d?!pB_6`gX#FL>ZK_81TG@(ml`?mK_s z%HP98PDHz(Ul>b7Pjo<5rJY&!ml=brVXS`)L1zt>A5R}lv_Ip^5^A6wj_hV@?!axt zg-;JZg8vrs+gy->OYX?q;AYK(Wl*NKAy2!sf!t5j%ybIr|e?>Z@YSZA^S)&8{AZ&QIeJ}m_7Xni`{r(6T_N!tnTb>!NzsTOCE zo5C94Nyf&;CU4l<6c-_F5lV14Ab~&1TZ8pLg7@$XIqRs#c1Jq>>Cj>pVP3Sp$KcP6 zp5jTKScMdH<7!^PVmO;Or{kQ9-?-mbX~vuGJ|HY167g|@St?;kN%lreoS+bE(}!*t5{j?crBo$Rbaznn!!_3fea*S0JVyhumr zh7f7g7rFCfH~aX>2C?dzxsNC|gFiCSSmVw1ZPVmG2ZYqx<&IRvx?at(d0MmF<5vbf z&j}8_#T-BZ3g zI;#Lp1#J=vh#LYAjV_zO`*MVRr!2=vB@DZEWz`rh@=MEAZPlh#hWZ%})pM zbHYBxg`_1X4_9(X39foeeGJ>3V)yQwo{5^@E7wsiHL52%875t)RzP;qZ+;fxSe2Ae zOvdTC6KP#T)~xh7w>NYCKx@g%t$R_({yHj()qXEeF;F8z?SAx1?+%HP;_9m0f&yh6 zPGiF=8quBrq6F9)o;3oKA;Bc6d=ROuCdzx2e(5GqBcNT0;z0IwFH=)k*(E{3m7uy#@7$8RP^*5SQ}tXHz1@L z+@#znbSflf4lU*22mm1LY#*@KdtkBgjL$)B9UUgn@tp#=jzyeedh4|m6E*O7NvXS( z8~=uuC>`6o+CO@$aAG@wS~SST3@e4OhHN6HTl9J77TZ+2()UC!UK4Vf;V%>RD>gBn zECHx($~IKdqD}mSG0qsQy6xvqZH7ghjTI%kqtPUtrB8)mRLL$`FjT@B+E1*YKNiGn z#0LCWYip}5qFvR!)5BW?)|B#-aBUAd$iu>FK%{4$yf(S8yTp`W0l@y)!aoEwZUD8rY2Ll%cv2-KFi-^W41kAXs%z2${>R=oVDok##^F*fUA}Co0UCB- znq`6$0%5{7aanfxd5pa4=;-L$kslKvBH%pKd+4nIl{7!J0XFjH$9t1vQATQz;ZV`g zu+qv)@&XaRwARnXoHa8uvv}y^GqIWBBaZ(|y1(-@dzMW5CL${;u5z zN-1bH;^)twiM6%1U_E_;>e-sig+gfR*o#}+=yQvUiz-@L*djU5AZPev2Nzm@JXxMz zR5;umNYTsFYhP-ffWYvru!_3n#jP((Y#xB@{Kp2L5G3Ye9eGFn6ktu9*$D+vg83nY zgM$O^JO+dDM1d;a7C|ed<_RFe6zDpfMxLusVlOZwyMNOLM`mMh5AW{oMg|02g9K3w zD(*m7zicfxDD0)T-*Ir|3%oV}HP*sj;=4bm-P@y#0lI?SXCs@OoSacqR0JYUo&XaD z%MG2zwS(Bp6u`(WB}2RgVQE=_MJoVBMEdxYVNYXr_5-PMmi@i4-WK^@lG5T8x{Cr%<5OLwAc*`p9#@~S|J#M`}HdY zV<@{9-t^wm>R@}+fL$kzW?|6=wqKc_vu$z&tPFN`1k@Z#>lYso6oAQsD=*K21^WQR z89w&mv=deY2e{+PY)1Pw<@zhow4~YW-uhy2hz^3*hP_&jwc&W3 zMb<={{wiA;=p$YNHqtBptN|oB%Wq};Q(>@67gRh{< z2HPAA7=4pNF$%&>L*C@-+^nqQK-@qE7u=15OgQ*E5p?Yaqs;P9?|U4+GN zIk~xtdz9>s0gc=B-^(SQwY9x$YHF2ic>a7F&{R_OMwk;qpkA?YK$n`neDd|o`qKB} z#Tfx2U9nq$&w<@JnD(X*G68|(9k!V!#=r`J02wGR^d|nxc3zxncoC97F^fx55V_K z$Z%qLfzl=oL2xbKT&|v?BAg#OrV4O*7wTP_z{Bj7*I^F$ehU3+>b#3xzC4OoY-wr1@e*2;92@L_Edjgx@8%}Z(_1=o zPnq3nJ#T2(zPubM1dA2uCXm)c<>flT?_b=ST>SR!+WA?{eD_n=E%) z0u+8KDz0edT^53kgsz*Y%}~FH;mfu*qpZx#r?$2XxKiK_P);ip3>F9WO7+m8L*9=o z6Iz?W;-KAWN&aDyq|b^Z1FvQU+Nge$l99m=fQR?5|7nv>(t$4mr`!&75jz3&-y8G_ zXv$u$U%m_J4D3WKm||BOh(}$(m{ArG zDjW`nbaj2AsH9{GX`p89e`V)w2!CmQ%}*9+p|M+2#$76qy8u3fv3(a{Y+(5rtu zB21S?qospv$~?KGq@))NDe?dP=XX&cB`Dj};^yuSwW=gN{Na3Iq9dS1hE83z+!#Ch z`NOMw7Q2bn)jA;@J5Z6qD}gPl|Ibsw6ciMY*RTI=*BDK`fB$~kt5@d^s=#8;3dT*t z_GW^^*eQ|>1!>%p%X!!{t~Fa=BVWQSppqf@>$}N$V;zzy3X6UD(-rg)wf?}UW6l6~ zbUWk6VOGG=ccwbBR4vwaqO7b8*g>%!KYWrC6Ua?Ximq zGwj62|3B&IGqJTU-f9zc8yxvwrT+*-Ks8$Fq}Qxev+Q}6;3a0b8xHeI#{C*) zIU)!aN!868(1O++J>QjKij@=$xwDw%DZfMM-=9i@$-bw`T?cIf&pQzN_Nk{%Uk+R{gA7X zE1$$tfYIIxj&Dl%=fA$Wa7S@k@ho(e34tx-PTA<@13R+j@nZ3Ux3hr%sF?*80l+clGPU52-SkZCM}~=hxH(T}v!Jp#_9f*sV}DArCg=+hf5H zNnmcC*fm7f0Npp=-FoE65$s(cUe%Cu27dxkc242(JwY z9<(7~*VpH|fqnREa;Hs6_whZ-4vN4oZvwQNpSyer2ec9BuHdA;tNlEa3sRDj#{&Zc z>7&3_%R>rv>8L|AsLDg@X!>*4QEbF2{uh|^d66VKO3^GG1h!>B5kh{hrsYLXPmk@k z_CNpplSm|912$1xpd_(z(C{VcO{HfrUapW4lZR~g$g}_YDCylG=0}$~T zUxG{(FiG-%fb4E7$a;!?c<0j@wX@b|T-VNUF6gmJNN0g_`W3jr^+>62 zq%B=xY-ygop)m#|Y)hixVkui)fzp=Cc7f0d1e3sp|GKWY?8CpPpFDXLIQ}ibo9cpW ziSC}z3a;h~ji!v4eVDR%J-N1=}Z{>g@dczfW zWeyvqTLMos@>6x~v!7SIdi81&no>ybnj5}L9Rg1DS32Etj++O>`JLc_1?(PJwb!;g zAbY7@>IdYygcew=3lcnq7c;MM^~rVbl@;ix>;6|)?W`-hy?Ch-Ob_(W4O(R#!$VOk z*gar7Ap;0uI!LL&@%GhYA)S@302QYQXl&tz#j4+(eP9b#@<|}dRC1`d z7AU`%)S73Nv+VS0Seu`>po zQ=HMOUtvxmkst3_wQp;TEQM7xg5b5vGD^_^#5x;cnu>x4-H#&FP<~8|hiUa6`29#muue>$g2HWBS%x*3S_OPd+bRPsP8`jL# zNYnCM<4QenVr)Pf+XE9j4|WO|RyrR*1ig^{iwZLSktUtFT#OQ|f*HN6I{jEk?3(;T zdthzDb+ok+z#ao}Iq_4g`p&tyxOmfBAR`Q=MK6D*A5Iz2k|AsXELk4(ZUG|S;UCdN z#TE$(3Gf3D$XRF`|KHz$pvuP1u8cD=F@ci;L+Awb8*n&%he}VY1Jpn_A0!nJ)F8U+ zharyD4T8ceH8l&}d|*<5Q`rP_vi=!%PF0#8gbo9k8p0XiDqgPlUxR#gKjb{&@C*#5 zZ7n@+TF>sG`)z?R6N##-Dhp6l9(nZBbx);MSDI+IUr=aYJGTo0JqN$wd3bqYb58{p zmz9@W9t0t2{J&4!^{!0?4!5$jYy&z2jN!3=e^myu`}PcFEAHIf+}OJ!<4v%U13%X; z9Sf|SimIxa?qx(f5Ly6e@z)#Lfge0AF8u>c0I@h-7uFR0GxX3r?dcpg*;5KR77%f7 z`@g@-yIcbK&|1^XskK`=HU4|civRbP>(&NjC>#HuTRNVB$8bv*5iUIU*qgh1|wf%HS%=;;5W|T zUN1u+9ZcE*BmqfDmdVN>d4v*05UaNNtzj81N{*b>my7zo{Qit`2ZjBMALs#thcn+v zr$PpRzFjjE+?ZR+ZHNM-$ya*n!G>Yq{~ROrLbORK;p=`mVFC8BWnsCH&b3F#0ybi7 zG)Pa|1_$jx%DV<+&Y1(oY2P-1l2&rxP)j0KrlLaf6aiENXTN~mrXSjIAJd6%EKjRi z`d~cv{%C_)?0f&T&bM|}InS}H+?XW?3%Th7a=$Ws&Mi*FtmZMiBbysN_2q91okjsc z)0yuv8w0+};*QUa8*K>Yg~YgdO# zJN_J(;fCN5A)=4fTy_B#g%TomDS>+J|9V>v+O<{N1baZuD830XS z31n8!3NB!=5V){aC0Ns#X`^aXXi;h>GTdWL;T1w`^MGc zS;LU}Qd@VbZkGG~-s(oNL!LahP9)?gd(gea&IU}kDkQ=?3HZyEX^2zpR z>;)z^)?s?QVdY8fj2wGPPcP-`YdG&)WhJE?u8RLWfJx^@eaRn2P&oJJ^;f=>5TpyF zkdz*F*qjt#ub9LvSM9-tz45D8aj(UVjs?!wd-}#SqWn7BKZTS9n!F$#hkJ zmvnr`D``r20pQnV;836P3rRV6VGWKkv(nEO!Bh{L@eAa*G1avtSd$WWSokCCuQ3bF zlbEh)acCFqOa88A96$Lp0j{TXxR9Jh-|&ADYJ zKIFkHXeAOIcj$l3YiMtEpBo}0BfsQ3`n4PGuN$Z((5_r^wIfci`mZGS%5+P2aViY; zc(1vGHSfPO#P!+(0Dm<8QgVpCO&2H+yPQ3NWde;^&@E;;=d?x;FHy&hsMDU5UTXb) z!@!ujmmZhG`~vIrXJ6rvJ45oPd^FYjF}ZtUcyFj|@+o%L0-5&qzA+-UY`V|AJZMYn z0Qum|sT+T%wdgOGZ+~9gQ6qg%`u1RW|CBWCdy8AU=fHH}w{h#?s_DR>s=&i44%E)K z4m@$z?)|Fg5h4$_2V6eaJn_6spZdOjhGKt;Oqwzci7i9vZF?x>f8||U00>d-Kq%nl zehPRckeUgq$xW5%lRB3~@P~V@SQVDaU8u-Y$B107-gK%~ZYt$;vs=5V|8GL2Z>qdzHJH)oUm zbzIBuqi_AnU71z&oPwh}`_oRlDx+H5wA26g>ZjbFFd)ZW+Nm)5&T=;n^XLkpCZ%9| zKoVny#To41F7g`1m6rJ?{b$)IqTqW78~c4a9iMAgq;^khrBfUKeP7L+V8>(e=ZP8xi7iUrIi(`M2jaC~ zH6bpWsds^nix%%0oa&n*aTrTJaO|QglwF+t| z$^#@t>pLj?Ypk}LVk^#awdUr?3m%@^Ag5zx@kTuRJ5E2WrPfO;gcXh0{j2A;8Z|k) zub}XWYjNc7U@O=o?PvG}jgZ}D-= zL^nyA(`CjSRMjNI(diYO`JMu{)n^5l6a@bIz}os7`M?!KRWJ`IgiXC&h)SF|GI&lXr|45q0{%O;oRwjkb()+TwQ;$O3OEh zRfujm2#0T}aS&}km0js!`MqsP|H~~$O1w`_T^K%s!{Gh(#a;DD&fExyFLzWclcZD0 z(`{&C1P@f)dy|%|$NBTYMR%{yJ)2m%R!|V{y9Xxy1hPU8#sJi6@u!ckZ#jgWuSM?u zin>8FrP~R*i!3ScWVSkn75A{#%dCou)E7|-y1lEGWTEEOppm?PdAVC^K<~(vlO7c% zj^91sIGIlaM_Q%yYpT<==0Zub_vRGwj~<(!WgpB{EQub{RgX`KNE$ZW&kJVrvW2pj zo&_~(wCE7j;!jJmIn#qCXKp}+9~|=VKkvXh33OI+iuU_HZN^+_cfevzbT)&DB8nM^Xfbh-+G7{lF*|X#&kk|u)OmVa#WX|dP}Lz|8SBfMt2IW9vx#}kRx)C zqADDb&gut-%~*InV`1oi%-l0Sxzo@N9IWv0#%F}pWSQH0`G&~ncKQN*hX?zp86;)6T}Jj zWku$5NOq>;&wR)`SY_7^f~*)I>lR~`pqvNcGnkAFoD+c4f(4;zOD8{sVrApC85vyo z09keID*XLzx7b5~c7%0(%Tg(4FmC!QlQ#KZe`uKi*s26auFmwvh-q)$gkvvvLZso_ z)?qaTqjNW}pYdp%+FQJR*OwsVW*fz#cY= zRctxQ%lIBrkQuM;QAu;K5?+9s*{~BNBSwlQ~xq&HB$-28d+}cx52<3hNYu)*(Rr(cTjX1aXX}9cuZ<=gvdTVz2H8E zzmV?kTb9#^R-o5fI8*S_1^j?3(;})?vdhaFxJqL>*&M?Tk!iI)a%X-^%-4^POPhkL z;1e=Z4m0_6r9GK^Puym5ugZS#E~01CD|LQ^=4Xx*ma+ocU0Tr(edX* zUrA;{UgF6+y~U?Ob}@K>lRb1|$AbK6?K`>cl3n2@yQ0fpMxH9kMxEdF8%W_so`Gt_ zajSB><4}DCz`+2{+6{{}5ahir=LrkxIh&BzJ{L=x^m)cA3gx+3l)K$2K@6N23Y#Tk z|Guo3!HdT>SIbD^I53-L)h@HQPcKE~idQo~H|PQ8TeadUC46)1q?g^i^^SMy1YRIZ zJjgTWXdc4RUhE93naI<5s^EoYc73rXmmX)P1r|!KuzhQ4947~hg^G~hxOH9zyTVa64uMdYD>uaZtE{pJf z2YowBu3s{qJqEPwy?%V)QGsd_M1j?YrOJA)DMqzi#8&ui0#QFQhhY!LtbI~?US1BSPCa5LHHo5z{TW%-6H|)XG}RX+?{VE0d<@HOgG{k)7`O^tp0LS?0__I?CehZ}jGX zI>`eOXz#s#1iH)HT*@gw73Baiw>n~8-ATV4lhP>Moj&ytRlm)7b*SEw$cFMiGQYAd z;wlh`@74K@B4^!nJr51kW13TksVCnA%avuGOiHj|CIM7g!qdJ6^!<^Dxp{f?HgVYA zJL|!R4O!hV#-ivIP4N9d3|DK4AnmC`ynf%NyM1sQ;C6;)2XC zp5Btra)>mMmqC=K_SzhJbN1w;&daXDyK=qAdk1~Wc9lt1@3TVelz(5A(-Oiz*?H0j zw^Y49oHk)gVB|*}Z+T`jMqubV28dR%c>U)qQOp6XI62A}H@9%0>+QzlbzRAMoh+?~ zbaTqVOV7Lk37?fu4C5R*fk7xkO;7F&rp`oN%XgZev|`_8*vlyh`40**S1^V-ISHWK zmeVq}pUCO*4yg3D?6}0~)&NkTHW$9+$$OS}+5+o8@7;}QGm1FQ%tZ|w;hvE#(ad>3 zTefsL@zh`-M_ukPrGG`PZ#GU@$6oAO<4IQA-f7JJdAF8RqqT@YX9XfEOB(^+Z>BpP zOr8g8_L=eRWYGZGZ@ZX0-oy;$bA?}?D)M=L+seXLOW;ic05By$wG`Uv7VkY^j|6jd zMy&gMLGjuVqJWEVet}NrBRU_H>UwxA$U z5&TDFcuOrjwpW+C!;2Z9v-(B3k-?RI5I*Dw` z_P%z0CdNrGSe=jG6gvukUs+8w*|b)$1cY1w!wL8rO>zG_5Fh__T%(~JvM<{SCE`N{ zR)!mbq5heZpx_0{p2h$shL|7`0H!@6s4b1`y@0HPu=?7fC2KR=4O;<+K^l%$Xg5FD z4YcUf7=WX}pZ~|GNvadW8xzn_!RDr>08ZpuTXyDL0r~$U@UhV++}!I#0g@#QYKM|K5G! zDe78DVXc!D;XX|=mj=O`{_!;$lC&-$5pCH93o;I!`u4e-&L14iwFLF*eFCrC1j0>- z)41Ug4bWqtFb(uLBLOQ2;z9^I7{TK;oqVW{0VPbxa9IE35sYMJ78DTVli_f<1=yA~ z3kz*V(x24?J8SEwYbB*M7q3!ERsDoZ9iUBs`wT;Td+_Td z(nN9;&vx;Xi`6znZ_PX=0BW(LE_AIf-J2aP;Qu}04oJv34q3gFwE)A*cY?ZTGLCNS zIvKe}GllI1Y#=}tE9XG;ClK5U!IJj;+{GJxQ9!I_fqeP0yvOGk3k|+5+70h^r1cwuG5pX8;42UlcVnqS10phy=Le7A# z5-4_p{+Hdbk?-Fh2xv2h#ea)h=B}sQ35vk^x2JPK!OjnKX+yL+8+lM5YgvCE+=<_>otxUR@&PzD!^W$ma+BEm)H#3~ z`i8>kgYPC)S{~uwS=&DFRSVh-6K$-<0}fiLY{}OC$w|=vowQ|b=U`4x*8p*-TYzjS z>%x6<-8+(H0%CI~xuJ|#F2K%})rV?FJ)fGIt; zQfJBuwV99Zon9j5nzs(LsVrS&zH*B#iLf)1!8(L6vaXm0vS?hSQ~_w`@N(Re9yKLP z2#=b-F*N}?_@c%FDP=r6a@7!-QZA3JXOx~>o@s7nheX8GC4<&I% z+eAnq>G~OjB+l2ym%wlJX0bbYvSCiu)hL$*)zM+U4&sWTY^b64!>5?h?;{fhqi#O^Beb-iwWmCxD0;Xu2pM%qcg$&g4U z`5-fMes%gRw_Mw;pTmLm)3}4(S_H4K7zWkmm$G5IF1F+_?V~l+g^MQOL7U0;nOzba zVn>U7k*7e-k3_U{Z~%;x5@1L(1XM&H;}<6&oHZgbd3I@Ze*?kK`iOIO!u_R94If5F zL(*cFKGyYax|c1UEeRfe2eK{#u5hr>FX{s;ilWVKQ=h(-^SFG7mxAt%Yaqv3!g*O= zXXFY@FO)lmD$dT`HzbT$<~Y2sM1c5GvUWM)>bb53H2KVh9Tw^F0LVkZ?5TOjF^Xn7QwHiPKSU0v$`v? z9Q!m4Ii|~cjMHlBVPQ|Sd*QC6qar}LX%=+P_F_mja;4L4p-53;a{t0qjH)8W%knFP zKcd7VsLHe9+&=gH%$tY;`{m~fEwi`j4Q`95xXP}mPx-STRdpREXU>1^BI_6&idoj^L^@ z=Bj-k_qT((oNRJO9aP_^e|ME8&ZJcOl&?YweazyHu5!|5T75gOwyQvgFk!&;MyY_= z@;TE4g&vgLiQXOp%Wd4qHbEF>hyaPOIG9D~>7E67-M4}wX%qQy>>Ok@SHQp#nVySJ zKSD4x{veUF(+q)pGESWR4&Zisi=;`h%EKPrGN56#Cvs$)DFNY#Nw2sfpNPKwceYS5 z8Hv;%21P>rtF3=~Zs>bXhKGDANkOnD_iv%_EBa)CZEQgW03HPosA;!>5k| zG9*+MhdQAk^4kT>X-Sy>H;=U!jDiiQ=#gpq)7;KzH?DZ2bFJL3-Eg7eabj%@18#tI zOMmK%4lTa)LYX!_DRzOL{-jpUi6LP-U^G^S_t-X3P1hwU)T;NM?5^=Hs*kBEss-6w$(s)>I7G<>zLrD*#5_Nq?2)P`vOLJj;d zUEPx9N^PP`*}Jpjw#2~fO#_M5XlH|OF)3~ zY5jH0Q6%Q>x?rrMrbm^$W6rIyq?^+T55=mShla@~?DUx;QKU_+WaidgFikDs$K2eC z!*zc2YWN4OZ+4uqR6}%nT6RA5Yb1JD;$hf$%>)9Wyzow0y2@s(^rhtk+$Dt5Q>L4X z|K54#1PfgI;ro)tC929(P8WrAdwP#NbE;45dKIP`+AcLUlI1O^tzix38SxM>r=1*0 zw;IH+^&34cW>_Aa>X*VXZ7r!XG}u*JbMh^wxU9Cxw89}?ug}FSK5dSyejml7q(2%y z(Yg@*C4ZiPY-|W3@%eSsF8!7MdtgWnXy|2bbtCOB9GbRW{gm{a`GP@2)d<0=K_k^M zsMo}}9dg$YjSLAi4oOJTnS~IdOXW9b{r-)-I^~VvB9_WyY{ylAEtvk{cp> zj@nBc7Ko&8dhugxqKhRU$Wzh|T`u0Ruwoa^-G^c!3daK)R^TV}j0{-6ChT))wkr1+nvTy=0&EF`8wr$TfcV(P4X(qiqf}b%_y;S;RcFpxieY zzaX>EC&&9OHFp{XUb~T1sBgXJpGnynXmG|ratF6#w0OB55on`9;V)P%@X6RSHqCNf z%ojoNt;D*PgI%h{F9QaJV&i)O)4&n~xR7IS4fn&L2~5IBSkrq!GgVWA@lJ*3xo*1& z3Dp5?xi~^XT&>(AP%|qi7e2)jzuB+&X&!i;N@p7=`oq&-4ZLvIi1*D4=mXAWxU3nc z?V5I_yQO((U{1AvSbvo_rlJV3KGduC(DYnaK57Po_$9mjt9PQr+@b^A=ujYF`Gm-~ zzYedRFw!03H|sc8y1k-oJW`cR+v;ZHQ1!w3j3Z*Zb_1KM(V*WBXIlyWe(AWHW}88(9w z*b;se#SbQy0f^8oeWdd$zPTDPirz_N^{=#&SHHXK4_EKA=VygC=mG_O)HRI>=cAdg z^U4~1%}#x4*iqY|c+}vmSjifw0$X_b{d=gzEeHb=7|<^QqS{&`*VJ~9$t@T|jWVwQ z;4cYS!ayMZ{y|ZNF;#%Gve!iKE1K36qQJ7z0VWE;vM%*q$ za)}`uu2=6(%#KZucIkwy#p|W{Gw5A3H8jjJfJ~{D2$>e&?|Bex8|oaPUHI}S@K^}B z`GB90MIWtlZ(m_W90R~}g4bVKZukw>6n*_W(b1T`LOxhma(P8(jBIV*ey>uYp;jrKTfW zl#)%{a#x{*by7Rw&Nl^jZbs=m7-%YmGuagW8;o%d@UBjuhrpJAW)A^&eBj6tTT4qz zqPn0rv4AI7jI;;x9S?fL@*u^Dd2}abF@`tlRL=_^P`{$z7_k%zQ<|zuf(4KvT1{ek>p=eOK#SOU0hg& zNG3&fo8M5G6_4Eo{Rbm4{HiP>ZN{0}jj0`miHB-L#h0y&$I$)UX?szH`3W`&7(Won zNDCRgh{aI$p$O<1or&RZ86pW`WNV?X&`2n70PV-ZquqsA>GV}Li>sfU zSNi@|H}695RfghVZ(_pl?Tj$3Q@i)H4aoB;Oyki3!&mh=wfq+u3Rr>b5 zvemARdONl2-h}i>4BAEzwlB$l0!`x{7Ds%~PWfB|ZDb)d%Rf0Is)8KzA$9TWomJ?x zBuvwKTR@%cp}c&zu_WNC3e&amq?A5Dbz>ux0T*rwiNaHC2?eC!ir}F>8$44ruKDrI z)g136n2o(snfV!3Ccq@4)Wt#OM+8e|jPGDJrRK|KAA`1ULqUhjOgE%LgMi3ByEbd5 zFK3F6*;n7P4w~&^c3EfS?m`NB&B9bhOHuV(oi1o=;bwRrH@{Uo0Jvwh>ru7X7tj8w z1(|I<=A1RR@9^O)=zT88pBi5QC-(b-)Sy{WDTC8&e3bb|1yfNW>2P8 z0a*mA|D|^=XV=4dhn>8ehoGMhRYQWU9?1&rdq9XoVZ;5X{$?;-`t}{#IkgM7brW@C znfs3PWOz&`vl}X-B|Fk)=97exb_apZA4!NZh7O3f(wwSLP7xu__gzV8iP3HgR7CW5 zHm5il)b;94`K$uC`;r%va*AFFppH^PocP`7J9mEm>}#mq`IsJHXvow8lrZcz(aEGv z?di_;v3N5CJG&I}b`~cop&&b}pJi7ZrA{2Mb%U22*gY;2&bAY_}=J2b+?Y%`4AinMPP zsKbIC0n~cbb8DJDly=`j6uEnA#Ysg41!$`+7em|nM{dkl_oMRJKmYMer-ullEg)+` z=rAXgcz2f$fzl2b{Ok>ZaNnv=n3b$%Yo+U-M!jIK%ols^6$BfZnr;MDzjR9MJ{A=% zzEU8woyrPD?v}xU1*}YYYhxfJq&*IbA_aD4qK7f>{Kzj}432L?3XU&5=Pu5Dh1_9K z3AUC4hMW8L$xn%TFZ}9#>_XNr8R%9gnN5D1=SZ>j*k>AyS(w4s`pk|s^$qh5bR2`!-rpfwJI(NP0l}1Xf0V4!zlX(j{A*T zyhA+8u+yqFt+0gjzEtrgV4ri(q<`rGYmXjw+>*GVwGx4X;7sl6TSG2iJ5pLw1pQFY z!D&PVH4e@sbSdSYy&;K^1dVfe=e=ps7K>T0ojK797P8YfBnspJbWZd%Ip$n0yUU2b zix_lEW&|(gdxzhR0td*c=-Cune`BXhP>w?5!Vw+IvJVZ=E+B6K4>c;-!OzlXuqvv` z=0l1x<{Y=6Ae*a_ht8r-LVUG&bowTn8l;Brl}Se_0nxOsK&-ax54imTEZcBUjWz$Y zNpC1Of#!Jui3q_R4uIq_@E-o%pu6`hrevTo`jvCm!~>%9D`!eU9eYn(_K^dGRsw1{ zue%_~!gEljcA0#=2z53!({+DeMYSmtr8IzkyYS6&(!o%0>{7Ib_PFw!&>Q#{2$xp_ zx%|o4LGa=z_Xy2zB5UEZRoJRL*<1dZ$H(6QQkYKvNXk&Zr0gRF2XQT(Ajo zY7rbNjg_S$j5I5xP0wo&4%4z3iu#<>>?=_5TdGsuEhzCi%_>`=HnpoBe5`=AzS4|f zOP)!I=O(1t6w;a3wq_k07?(tv=>j@|lP1E>?vrR2j&2IL7jP(Q4hXF&Ey>RzBaQK| zAp5}p(+btiJrPcaP!^9v+b%EP?a`9Ea6|dyoGRwsu$l>S_xcg(5b!;QgJhfd*c@+k zvzHn#4;@>#!SL6V(BCw`xyGQXA8**qv`O*BAb)Nc8YiXhLkU|l1ZcKe4@nkmq} ziUxVZLBz>tG`nLPkdrpYoFS6-#B5xzWGLZ;;GfeSnY~7EAA_aN#5Lupiy(0?s&-w< z3+R59dh#RODl$~u-pVfLbqT-kPJZgj4Sr4q75Ts=K{@YEw_LgFW06MRKJssCw^((z z{cSG+XU1W@5MJp*N?==gQ}0?nEN8-V=_|)L0H+BC;-HcoBT?C-;-3#uOAEY>F=g3j zoCG*(5$d>ipn0I*&l^NUjAhjt=kGI{) z@!63^D4Vmtq0RH@Tl)6lyW=U=!WH7tXg@r0?hz5<)DM|(WOJ4d`H!bAZhKkC*CWJ# zJ~>?k|I2S-tDKw@JhU}I9Z-NJsO*x_=3+SBLfOi_J9P+Zs%8O3S0%9#Un-K+qQP$3 z+O^`Mgz!(5G4FbKi?dc}eEa#u((d7RJ_h&k#?;``7|W(6FawyHx{Y9x$SZ({pUOV` zvIv+2Xs3|RKNyOBwAdyAF2$k7C1F80e{ILW*at&YLvGx#0Tyx1+3QtV!U<|IEVYO0 zzk=P*m`{k98R-}<1ogPZO(4P}#Zhmj7*VSZBCnO(h5!_c?RfwGj;AKF6?+8hQiKSk_n!dZSC#?n}L(M2y)U8^<9c!i%%}y zA@W~Ev38gc$BzFQk<7O*t3>@BRQaAXbhx3yv#bmpO++6{d+AGsQEz}WBU^13YCupm zjNCd-y#M9)7rq{bJ%NPRxE&#PGGJuzONccOz;1Eq=+GQGm=0eCCHFRV11RL8f z$}whx79^(IfZO1t19HZw(BHSOa093fBe462sz7IaK4ivkE(eavpWkhi6#r@G0Rn%~ zYQec;R5;yw`>|8wTG4z0w2f#0K53pt-li}Rdey$FH`<+n_J`iX*{V(?JpG4%JM z)Rp;R(5q~mWqBC&AwT+^J9qT6^f&zuE?vHEY|dn2;b7bDy=xsYNuFEIxexBWbx%w3 z{+YiIMcG6VgPhM9w!AUuiaox)qGPp4#2wynn#t>|Edt&x*W#2NDn!to(a!T7=5UxJ@sq3w7JKsUmMpQ9^_vVmsa;MQqR0uW zMRF<#;K&Mr0G1pIpTEj!uhp|$h#!y{G4%!0lL7n#J0H)337GJ^s`63;7__jLL7P!b zp74*o!7a&ETl7L-Z0c%iTDewM#Z+f93Q9^z+ta>C>a~F@yT5$$?Pe4SuO1A7f>jjw zke?OdyeGKE%13h9211Az360`rYi6ryVTvz{n!?8X(S#j>JpC1Yqh!IV8IF;8i&QDB z#_y?o0rpC_D0?q|85~I7PQC+qC#6y8f}+Xe-H5uo;D`j4RJgdni00IoEr-ok{|I6E zvt5>`RUJ`w;+&DXD2ALk{;DVI*al!fnc~R^J=!f;3T;cWMts$(Lu!-A^4w8gY&q7b8AXt_dUb zUm>=X#${P=MR$>CBSLv5s8!jv$noI$r+!g~PnmDKtd-8F?#MPs-g`-jOQVsC+w1K*6ckuiIo8_rAos zaY1?8MTlsyApw@#v_T?-@TUcPt^Q#uW={N1e#gG9sz$xSZ9AImOL2382L}*WSgDGc~Q0Vge)*g(y4g7WT%~>cMhj z$V|QV){h$JimOP|DXS1Kqp@6Zzgt=#mM=q#{M;~M-D#_!0|ErmeZ`F*kdd8P_##U~ z>b$DODIt8rerQBDZc(s3wHbd&P-wbS75t=9_FJQVz1x%d>hUliw*TTwN9q|Qu%gt~ zw#WDNYTAL9o(R_e3-8vzy^ws^+B?ci&rK8QmicD#$g8&|Oxe~FRW=`r)*)xc2?Tbv z8R)QfdsOdaaCli@T+{{hi1>R}E75dBMH22d!} zAe0w^T=faZ1t(vo+&|OPFC$lGPqYK|oe)@rkqm2`uXV_JX=T32OhaND%MSI+hbtDCG*3p=Po`r19woOYbC$Q@Th)YtHn z#R|=t+=1-L;!a1cCJ zAy3E9&Tjl4Q=d5xz!|htk&=}Qo7sG0>)5wT;J)V)hIQT7OR6i(jCI=qtZ}m;xOCDR z`HJ4?kyFr@)~N;9R-~Xbuj(H+s#EVG+mDeQK`JA_8$X{tGXUjg(H z&5Y7f=im7_YM=4q_lHpj^!4Ptp?*$Y<;dAAJp~;x^)0t=Q`;Qy`+$rkK&mJR^Rtk& zh+qSM(bmNG1&@^TEZWl~R|reC4Yi13A8$G2`y?^t(51f+%5#26>5;stV=xg>*D-aR zOM0~?0Fv3*-|eql+lL-rcnNJPD6)rqMP?LxyVFWoGLAL2CJ+iw*xyT5i@h~^<$P<+ zR${nb@Yjn%_}@sZxLy1;nwYzluZBprNnClXJYrB8D23d7AG<-_6_WT=a8xv_FAteX zDT&~J6L|Hgq46qO@pfjs*J&Y?w}2LFV5BbkG5i#W!hkqy304pMX%Jp9r!W83q+X?AY6kJEWk7Ke|HRh zlZ=xwvZ{!>hoGtepJOGqg5Nt6TcZ47$6Q6 zI@h9U5LlS+g-FB>#go7zXTatz6O|%2Kf@E#zJ95LQrn58k+X1j$)ifK#>U1d1IaQi9YSkFCf*luW)qX?_cIw5m3$sg;y-Oni@piUajj=UV+GHCkBz$IBrXcvIesP zSqIuzD9T`d1!ZX>@IT-?l-Ql>fmGEp3w@Zpwjhd-mT=_oE6ERiuWx0eZ$Uf**{tN{ z7yq5_D!g)rGB|u=xdJH(EwbEf!LoY^NHpO(h;&ip%C!VINhr0NF60TnqqQlq|C%_Q zicq`B@&c}gq&o0pp{xy*K!G0v7sBHTu8j9+0L#jaTGUGeuLpq7=q=!fRL9@*x(9$! z@mj(O!^iCk|7*`Bm*3Fq>G4}@FfC6{#y~xh^shpWYSRx56T-f5e9>=zZe?d_Hllvv zW(i&#*j23q$9o(WLLtFKgiMCyQnmr^QV_jP+EAJLg15+(KqQw}#H)D6$-5RUU~0jK z$w=k9RI1+L{bYM~*R0b#*YDEMhZt7oBHfQ6mG3m!HLbQ#@YpUCsCGo!E3a@qxSvb= z%213mYGs4Z_1x?l>MMgY;W`vp?OP6y;3Yu1KcQvz%zeOYE2;%p%~Xod{94bND^0BM z!cbN#FMu|EMJEtF9AW{V4ff3TR8EzLapxqiCm$)Dnw8jiCLv`nLb=#Uj);(w*AD*r z9BQ)11yqjB!zqemcI>VOpLtqq{RLyE7ef)Y?5;?rDdA+QE(XCHPn?Xfs7FEV3RIAm zc<)Jbj3^`t05<+wP)sj5a%CuMiqf(+FL=@1XW^i3Z=33cnW6D{GL%qHfBdMiO#{LH z1!q{Eb~3(;FM~G)n>whJmd|ur?E90s?#D{c;E1-_(8-L5vf*+IIqu@^2*|t{A2?Xn z1%t>Q?e;IPRXYsxUxCX_AT1Tai|KNNl_|sIRW{dMe1f~x@~)2kbGwPC2Wit?ai`6q zC6PzcfCi@l9T*VlfXtK%wnz6_G$Xl%M;r<2#>A%@o*DrYVN3g`NY#3VlPNo58s9y4>i{Ro}W@HfTY{uwTTE@ce z+}U`I{nIT>f>;4{s3l$cZ!vsYJNdGjBls4?SejHFg3TTYb~z2~XxQyli+wB!uN(Bd z9vFbetZ$VOzqnApqs-C!Qo^va)5Vf}BX(h#&QuC!+VC^0rYtP<*U#B|UgpQ>ELQ5i zc1kck<|&@Wo@(Yiz(^f9szG}7T^CCy?l^e{GHE^e5QP!4?`dk@LV$MFCAcC?R{ZX+ zJ6hUy74_T}lkCxAY_`lt;lNl6xg^I5H@CUJ*PA{}Xv7{wyfpYgDZHlc-21Glc7LAx z8-FXu$zsNs=Xlk8h#k9-)EXDm-PW_f8F3+yFA7}Dj|H+)HAU&~BS$uZiCNt{Oy7|tQM_0nhxe>Zv&!$z}iGFhBaQTdqORAbx7>BHErKbCFU9pS>)$KrU z{P2YaO6QKI@TXM_{Vv;{Nj5JesOjJ$*Td00tz(Fk-J1*Q|G_rZok_G)X1XI2eHsN8 zYC!z`k*G0kSlL=q<>i@{&}5r^9#+7+*l@N(XPvAazyV?ODPIfYYj$<9L!X+pC#%wG z*BwBQ1NonFl9J18D7Pj(D7w$O=4>d}0YtW|`-hYM$*tz$jwVGu}d{ZNbk!ouJOMzgK?D>sBf z=m0IDX$;|VyAoep8}zQLKl6j9DsFV|dSTYXVeDF+g|ofmrtE^wY1Pra<23_$nB2YQ znyR#SV^>FfF+Yu_DC$)zK?|O>>q)W0quVM}>yeJF`C^ShC|T;8b_V z$oWLz#B!S++bpD5?4wR2lniKnqFfUJb;gY!l=VI453hDkMQK` zotqXiF#uIf=~9vln!l3manUoV1@yc}YQiw;e4ku}7(IrQG$NVI=Gq{Xv%r#R8BN}j;ndRRZdx!1bJI>ql0btpI|v6NIDarA{ii}4&LP9Zbn6S z8zF0Nu4X^z?u$~Po)TmsinXVwONZ$p6{$C3;`d6cEuqjr>Ow95MEi%QC$rVz+$iwR z4R-fOy`tjPjA=EVuelOoR|%>?b7U_qjCp1goy#29W33zFgB+oE>5s34G3mdY-m`!I z0&LzcmjU4B01ZscC`sXK#M$oDVfl zmw2Ol{%wSf?z$tMmQqyzpbN5WmIrgME=D$Nzyej_Pg-Tpi{Ss5?wMU<`4jNg(wbCP zpDhFAf{ckFl8cjtT|a{zA=r6mh1cj~P6}by3lz@rNi5LwtMz;xw{;4cExi(#U(Jok zlI^WtU@;PA7kKqxFovgk$#V~w86(o#;+MDTcJf%} z?m04tpdc@@(o!#8NzsW@KRlZmpn$0Vd@JH3ajw14SrpL{_z&to$LP*lW!s+Y31nCD z0|q5^g3|dbCgb_XTxPq!O-kige*33bxrYl%UU@N`A*F98S_jo5Ox-#Ks?K-_z1B*# zw)_ow52F6TEbdt^*6Nir4r6pzPK^<>`26AVTLd{%ox+Ma>mKb>)~k{BSQP8C&c@by z?+=$m3jPQOc5(0m>u$;MA|8@oup{g~R7@G?bKhXD6=szDHIkz+17_V)s&1xgj3_nD zITrt?YiU%wtVdRq7_y!jDQCYXEZwi&*6hs%Z|?wu96M!dXZ7q}D;*K!<^UR$f%i(= zaSc?7q(Cr_Klrgdq-L_$O6tvAKvUVcozxs^#rHNaBcw0tiN|zLw)Y7m+OOnuRm@LA zCg3HJkbZ+OV*{Mz0h_5n)9FP@6T=4x&gjkIuJlaegcb46013Qm&HQU3@UH3jc?NwR zx@GHByyXN-H}#aSN?g-aCo_IP0?|HUYu?y2JKG5?$Mr65jh7>g2ZH*eA?hkH{`jg0 zsInvMHhUn}xBiElcPc#dVyd!PTr=a(pbAjC}q9>ARMB+$s6gs9pz zTMnIF$N@FIc{$i|z6j~nL|N^?92l`II7z4~lTHNo{E(Rulw1)5iplyrVKgFb4#~uv z3^es7glGTf_5t4}`aFUGy+;JIEcu;75qS%BAPZW(Mqi;I2zv<5tKjEOA%9ICD57A& z|9)&CZ@8-nA}T*9kO(7@|G5caqMh$*@clu_?-Yp0+b$@1YU=X+%cK8UD#OiG-H;hz z*M38c+l74n4&Ou+C};+02nMk`Ay2;ZCL`)XXoAq#6lRj4k!<+#rB*yjOeDgX#HCPn zVfmrNFF}au5j^z;Sd{XGkSBu77rM}o)B*nh zt)fwUqWS;v8v`_FEr_U_`E;X{6&=$RPAZ6G6WhFPL*R2=09+nFu-_b!?&7c4t5?^x z2*_oAwov_O9$obvqHg39ynwl}^-L^8WHV!`h@>~*i9MWUYGdTokY;o&ID)<21?`}= zh;#})ztgHcEnMS=m6X`|XQG>tNNwNG2ATqC)~a;00sdg*(GN}Judk<+W4dhl*)4o- z9O)&h%5Q8{8jZ-vjL$^8axLVvBOqC$=k9R0`fIml%d}xzOEvL`a5FziO<7Ss^=GQ8 zQ`Jzf6hi8O;LB%~cKaubJrQ3|isH8&@_U99gu6T2SZ!;0yKbqKeEk4wRL8&}5~j4C zzBCt%J}?JnDX{Lqmp$WIPQN&F7oNNu*bsYv{RjeL(`Cg z^0AtkJip#(IW^yo4^OE=D-kWLV-=}fdm*gxRaLD&-9+^^vAk(gcDy@A)6m+y^NSr zB3(q~2AA4`Kr>v`}s|XGMv}jOc=s@+Kd?h4? z;12vlTrdiNOpNiueo@3rb=Dqf@9?ggd<&;m29gEVeg9JQBag1~<&2**KPxC?w+9l_ zU@q|srZA-u9dUqyMoSBIAlI%bia=gY@Pl0mF}4q_0!=dj)O92B1K(>wf7!kbJwB%; z4B_8cx_c&Y8T6${ATZNrth*9rW(4n1a1jK1>l8M}+c}7zgZh573L*u`Hl3g%1TXO5 zNG&E3$j#Tf803@8VEe7g>NOYqR>^#Ws8yhT+r8tp7+djcP!Sp;)v+352=voKL@;}7mGB*K4_g2VWq}dfNJwh+bhuUKtzy+m{zzzTT zq9MILs`eugmTSQ0%Ny0VX6q3PV_2^CpydSBEkmgyl-^wk>-L=-7r%9{P!Xu>i_YQ& zBHte$fkPKbv)F8iPyY3LDybRWl#Qg}q=|NsA2p?|++`S1T<|L5ONa{2H7AOAPVh9(IQV)QP`mri#5eUu!a zAo_nEH0;?d9gUu?^BpncLe2UwwBlPJ>aIXl*k*^E{nFi`Y;v}O55y6(>6>gCJJo-(XO zP&7?ZL5w!2s(@Al$w}TIkeq^_!2d(%=2eHxZa@?w`3ed>PLxD6rEDA{r|*b7`p=ps z@-xb)9fpr>tg`-?3)2mrPFdKtKH7CyC+; zvyQqeuUOIEHKyw4!~Q*5RvUwM!YH!=CKjE=A&Q^;PJfI@J_FSR<79*>R|SLg{-ZmR z(7{hAjTP}kAfz+_5+J~EppPj1uBSSLnxY#*)FMDX23+&|0%yDnZ*c)w;UPDTpZq?R z1<9d9lZ(I;z|ITWiM_N(OAzcK^s2eBGjWwy#`9nv!;l#!P7*))eUbuXU|{Sl-=C7! zUw(s|Bj{G80Ixiv?)ZN;%MxuUL`VA&@1=uJ+|MuLpihWkkD~=5(D7(#0B!L^?ku~% z*w-NBf+t`0qP<{_+P_a8x|AM7+D`ypiI}kn9xRCf%CmsRj}7=Xj@G&k|3QEnL+@Cy z2QJEYm+ma5#j)ln6@Lp9qcqIfai#`q3W6!(!LW6IpsO1pU|~+{1VdVt@sau^Pz5$9Qlb& z32ZRXLcC;AAi_TFJSk zuh0vuMlVpw2ss7W*G)EH2i1?D>IJ{mZStgA{4H5tB8cYZp8 zQ#<9viS>dLAirlXa2YvuL=&PEwP2T)K6@e#rv;Lk@G`EfRsLuZsNPV9>;hgntQyq4RH9c%R* zVX)}<&#ePoG0I!oY>K_P&?6OeDvH1l6D(%Ijtk9n3MALxcXRXh%vO)%ZUol&AYL^V z+qkbB4oV>kuR&#?N(FTsmC!LY_)-2bSD%!fr%sFcL%vGKcX()Ou-r`|n^PlHpy?OZ zu&@$e7fV6G4D4zCQ4;iTuW!Oa*paf9H`(90pELnW!}_v`)gz*Qm~S${p_1N;YCFm= z)QH^0g@|C_PfzOS$dN6&oj7}X529*kE#(3X-pbAZqkU%$s>iW^xfhEfN#lj%x>n!# zf7J7onX(5lx(SwbZ88?o2iFjDChAcNi67E1g+%XYA)+-qzJ;cI}U^ zpVkjAZ|)vFXpdR^Qsz@L&&9PlIrR9CY*r?z_NxlLGy$PbPxn6#faXBRbC##CifL$S za`JW;r}!?Z*pahNFIX8EAws9!U7e3sXNL~vS5|~hiX6obc2wC@_`=g| zFiB6>ACRC%L`ai*5QSwZGvC1a!Vl%{#|@I)uU2GpJvPwWPude7GB~5^>`p7Sutl{v zR(?vwd>3p?*5eS@!c`Ulq`;Art(>{+EBTL{2JQthKIB*N1`hIG8BpUWjt-N!;Zm!d zxe#Vx#DrRtRb1_`i}zS|Be|?DklVn@+k0^Sc9tK*GjGt7(@7|vfwNAWZKe8iS6exx z_zc7m5+lcMJ5p1^YG6+5)~g*!)9&7p??Tu8%_=3+NgR>P6kFT7@pdtFbKg30aLx9F z#?fR()1vg3;)>*x5sP2_4X1-isJfS+mK32j8BH(oqf5yp38$=?{?OrRr5lem+plA8 zrL=h`w)8jgD%ilX+>Z16Uy{OK;cOOHlmuhU4ds=m>$L_I=>JGu=`L}_kX8`&7BhSE zgin?pTcs5TcBynB_f(H}!uvTo1gko=ma0a5B0UarXicYA zFxd-Zx4nx-QbU^p45{J9O>MfM?9!Wp{N9job}8QU&y7ih%#Lum#r|5m7#0baf+N8i zsI83{Or~ks9&Q@)2=8Df$-JwdsbP(_WD|Hbv)%1&f!3Z$1J-qZid*72LX_R7HdDbg zTVURysX7?4#*a1lfqB}I?4N#B+Pur?;|6^B=v;Pv!G#W>n73<_jQRDLUc>^o%xZi{ z;pkuaTZ3m4)AFOPuWKLLFm;Aeq++G7s#`n~FbEzt9f zFR`tH>A!x?E+)96ZB@*#jxC;az4^2xZl_pBZ--a|zSTD0=Vt8@UP@C>4k5pTP)$&! zZ%b`6Cesg4lbh-b*CJ1!fR~S^2$v7Be3?T8o5++6o*yT=+ll6dAJ^I(_vh*AV@Gg6 zAdpnDXbAk~B^V_`X+8u&iUv+Vi(aM+`;kIpjZW}|Yx32zl?RoeyZ}Rk*XMKRx zsN-~ixN@$dL1{`@&xVxx8czvrSKeh$&ZGXPskmm--MFCe&Am(;a`>yuxVVoV-6VH< z;ikj2A((d}_;Y+a;nw(g4&$iCZoi7ty$CGg($X9&yTRQ})y(#TmhZ{s_T@d@(~MM7 z)_cD&RJB>>y+251YIX$r9G*JGp10zS4i7uAd(VU85qI=U3SIK*uH9cbH!_JcQJi_8 zHVO_1AM-mq)TwD_+B)5Mq^;Iq!69!COXuCC`z{{d^x%#bEe-BqMRtbmY!Gvw%{bMB zU1(0uSd$W9*-PaX$$FOhzIUTh<2VChhX)e5`=)ya%gvhh)3xdHTs8WWa^88)iwKWj zm8m40t=w!fSM4RWk;4dU2ojNzZ%M{?uUlOvPQVFPmB{4=eSNRQ#D!ZrXv4y??3VKPgPJx@rOnvSdzFaT>^H z#XgXltoCwmW4>kKI<#ZzLQNZkjjCt<^dP&@nX`?~TGQ{UlBVkdB2?LhEjk;>>_41* zR^P*YwXZ8uoWM}E-svQbY8R)7s|Aj+2DBW)y3|&ln6~#1Us$n92%FvQM@6&ZbbiVW zE3q&TK&9q}#Ji9WPeCORV*|JM{Rq*r$Y_uk?I1%uey_i8SXt@0i<{ z_OQ`_>S}l~yJNdLl_t|ZWM5-ijs{jL(H-mmDdq|g(JGuo@C7_sM*W?^{0_HN)u+2v>l35o5_SfJw3;+P zdbXr+#7o(l#kR@1-&#f5`4t0$N0iP5NStPH+IWqxU49hTNYBNRsH=JBiE|^aKj5Ur zY?>J4+`PT7=Wk>6Jj9?)gH_@{`8$vxVhD_ zE(KR&|6KD_g|-bms@|X=`RoqI=ayEv!#fSMWWlXh%>FY7FcRA9yZ^eJSs#?NYEJ-u z$II*(-q7%{T6v&mwxTN~%tGWsxk9?%Z($2{!6tq;ehDOyhJOi|C<|)BiM>0oOt&V9 z+6R{odP}tD)2E8bLG)D2RpxZM&Ke?_Dv*QXRAWVMvSviJ0vgC17J8{&@+IGEh&nm z3;68bGqMeqn*Af-G!h9(GLP%=7sB0&c3M32Jv(!0CV!&+O9hkKo2RtVWg^&YCP0j7 z&QNG`Yul}0Y!|yzs-!05>&(ZPstAi?FC#5UUfFxw^Shd47n`-MEP^+=RJR5w3!Abu z*_{om`xxA*ondZ{F%3N#NPz?R+z=MxxA8O5?I#v2rY>f0&=_0SJcx0)Ht8lezU9HD zf=;56Aq;-$JiqT;u|@BuR?Y_&x5*Y$oT#-f=QO5)G-0juEWNtFgBKQ*II6u$D8rGJ z7wZDQk+mt$(#|YH_Yr(Ob2jm`k*Eds2)`%J^qezPGMq>q_j; zF#R0>3sgwjQTe61e>eVl>KWNcQnLF4p$5rsjuGrn4Cco~1rdtpYT?Yd3ry~I>K2(N zMz^2Ljc0k>ds*(7d$?GyL8!0fPBz_*mOGZ=SQ(5j_Qw{qVXX-))O+rlP|Mi6_F*fP zz#gR$6!Y`#rfBP8gT8hZ(?a!+gp4xS#*Fl6QlDLumRAcYCcbS}4xBKSThQYC5R=Hx z*UywOxHxvZa@L{J17ED?`WJ3<=C9c5#{t}|;9i#q7rR-yNfyrTYb7tlJr=x-N4>WZd3K6-<@pi-=El4*cf2V+G1;19n%}oU>d)T`@BZsvZJClZ|}rg zTP@|-l>P|iyJe4^uk@5ml|MeYHekrHTc?Z_^vO@xzT@N6pLw~Qsd?>)AVv^*%Msg@ zlft~@sZe5;7WtOffhT3MIMxGC*Mm6f3x}hhN#mEQXM*yEl4tv>$a))++TNUsdq1$? z()_tE@S^PPXCuOXS8_zDMpPt$6pbehN)vLJen52A|8%QWivOJW9CM?%nqX z=?O(%2GaF@B2-|xdW{Mbu9ms3+gYDs?;#&JCBJb4+qGUh}!vA7xg4q=@rX7tG|@O&T5Lx z@o$WG{UOb<%qz7K_bH>TFgYyQKdjkVJKtE7&@|5~E4_))o(=O43DCA0b|#H>{>GAo z-Wht{HBSd}kr%5x+(o8%YZA2wZm+GUo$uDbvaVjk4o%`dcWhl@5}bUiJ1?74_5R?v zPo;_L<{x4U2_*ljnwgo8IxGFRQukGK;8f_xjLLd_*&hRVe_)!JwxQEW5ot^vKUI%P ztuhDC;=4{SJY9lw^_=KRF)j%eH&Sorj)ok|4|~WmYLP!W-S99r#Jr66;xF$VoqN)r z&vmxV?~P;9$5g9pH209Vv z2AUNFcosbwGS)?G>%s4G+;K`_bZny?An`=5hmq=6BxcKmB%Te3CE(Lpzrg}&E zCb(Vy?e&&Z#*Y|kdK^}6Uv!3KjSELC>(~d?!BR(H&c4T#TWoI7Rc>bZB-Gk+4;_6x zE?ZsAtXg+{I6(4366z|(<$hAtu{8Wqp4zck=E>C?DB6p9_6J>l3_AQNz?juLJT0I4 zfZ;6>!``2|g`UtPUpphubR^`q74Ewe*86F^(7}mZv2`QPtiwgw_PtBO`m1)ViuTcp z2hJ~2S;M zR5w2_zhpq(4HK@Ek^A3BA^ta?IPAyifd=SldL6%Dv2J9;kJLilQ+6dsLfcIZG`JzY{T4HWZ`08~N z)Jq?Hc6%wm62Z~hJ1=XyR&0HZ8|TSIOsk5D#ra>Yewj8fOK86{(~0XOMlv3sW@J6C z46uGNnnKGt+tJ#o+4g0{1r751+2V0>;aqh&&Wu)&hmTW=HV$$MGk(uWO)a)x@JMpY zquT&CT5fm=uFUHxd2xYmXn+ zbY^O{nCFq#(lS4Lu37vt-f%QovWUI^B3ma-{V)4jTCv@v5PASHzNGrSp8~!^$FX=M)#;*7cV!RDD%|v4V6LTCGV3wN zb+h-^7mYR49n&EbYHAiz*M@z4r>|c0RBclYC5%3FR>-d&BW4W;Yo0p0!2^er;OeH* z9pk4jDtB5Bq_hOCwq2usC4Vf=)0keK$_=q&kR7L~)z$kyJ>(3O4``A`Z85zOZD>mR z5{>DW!k(=;TBEJnG+yqJkkod6&av4zZy!$RKLCXN+)8aMv)egH7ZSdblt+1kHh4)@-u3?P*j~+v=o7TFe~{^_FNH2oQfot91@r zAc}+}7gaavws7a9IyMGiy^9az7YBSR3TmkHV{XZLy_Zv+y71WZY-?Do(#f-)5@}yI z|D0uK7%2AKuWO*^Q&2;X%3=ML<-G@*;!F;^`8m$eNfo19<7y+%}^oQnJJ_%^{ekgL`@S>keF2-G})}@T+ zqdpk)a5^-X_XO&4$2+ajW-H>cKmf17w(-jy*v*$oC;nu}|808wtjrfLu~l=cwM#HE zLy~d*d6{QpQur7;q20SV;CquFn*H%!fv;+5x z@UP`2v$k*N5^F3a-afvC<`OXZtPQ`I(i-Ng_v>`e=QweyPgd+T&D#Uh3^A{ZMuK1{ zLDsaR`tQM8=Q_Nfu9MAnNo7{Ho|p+*B#f`a(MuNB3cXE#WqCK+D~9B(MDJ@`{M^)9 z70(Wl1uyO8X+&BSfB^iOCJ7q0z%8K5r$tN7UA7c`*TlwK@7K{fWtaeCX>do=FM3if ztZIief9TBp)n1Hw6Bo;lF46K@6P!=0$?F~Vi5%2?6i+!lEj^f1|H+;?k{zym_ZQ|I z>4w#?)@|c}Lqsd?Q}!^aX)3|9bcH9K5D-!{uR3kQY9D!B+PQfRarDJS?JKSR^ChE6 zjjFo+x21{gnf*$nd*%|MFsNauD-Xwg-p1ZG zdzbWC@$v)K3@d`!SM*agG4$8Q?9#^EC=0PHUyG*wOsbzYnL&y))k_-Rt0ws~lJ~kO z=u@tczf*&bU-XdTH;mPOD%UT$?lPyw&;3`%%kcbtHW`fihcQl$8I0jO%5At@T<=3q z6M}y=U9~Os=-v2eS-WJvoPoj~mnJRai%Kz!;Tr1EssX(*uRnTRV{v+eGamJR0U;69 zy2JfDj>MFOU)GLEtlf*;As#^!QBcycT?(5;jwPRqspvc__E%@8)l9bP#0TY__l2Ih zx}^Q)&1xx0VUN1W+T~h3d5)8>^q13|wF;NpX+WAOs%~cc;4r#c$Mc2WmtHptz+{ll z8mCa7diy7irnJovf7!3pX?uG{kK{DyDf`q*m(?Yi>>~TL#zz~Xoa(%_N`Du$d|50= zh?~{g6xek5MF+Fba>c~Kkjm99t}5J}7|*Ud{mzeLRw99C-N6v~-2Kev@ zY*I<-cEcZb_Z7PSem=H-y?OrDoYCLxxMViTq;rinF3kDqwX)JGhkIMb zZz>5pZp=y>yn)r0+#>S!=<7#2cr$sfJ#d9_7j<|B<4zVxyjyq)^X?X&7`;<#r zM>0}ZVVc>7nN*Xw6`_53NyKT-u51kLZEmMEIoN@$a4wnNn%fo}ZTA_5&tZ369F`}@ov zJs!=xyk6&Z?!D)pd(QJb54E(Eg#Xx^_~GQWR^D6`GRr)W@pxK?QC@T!-P#_OUl?f- zOD;rP(6zST$jH@a%#?x{cqwUH+42Ki{fUMJcc<{-u^0Lv>qCK8o4qRY z!$Yy~U;zfQg%>SmlPu@6e1ME`;(j|rZ%z-`4l*c?nFd0OfMVbc$f@wI4a30Qr@qYSLmc!`Fc;iAo(wpoE zl(-i9>-&y?HXld14G1D>(ogB}NTfZtq=%9ry=l^=G@wRq)ni&liP`RRk+l zHwQvtXCFx>W2^cxbt+2GF-E-BAAt5jkI8z2KB@vjKOi=f*? z`vmd)m}IYtkFWPY;TUpP8Vco~VAC+e+$wskXrgnr*XZK)_ESs*!A<~u7)NqIp{c1C zHOWD@O`K(I)Z~IS85qaebR;Ks#H(yF^B0`ZT=K=^an!^#E@?Q6gz|4ExRvY2h|@x^ zpy2?2^H~S`1eBxxvv(yOlYepv`SO+unrqKl9}vF4VRs!fedb^bC64+Z1fGVEmi>xD zYHD9rR7IK?UZN5>ULMM$6@{vy6c`9T` zZ=rsR7I#B(=6LfL9g^mXTdFzkjBl4?oH+c<(`@pCs9)<;U8mit)fZ20^t9%DinaN8 zwDtnl9mLQ*5*kHq84`zaP5qSb|r$mI^o8v0Wv1v5d7{WnczYlJBo~Q?h z59fx|7DSZKTp`P5cGfm(AyIf|;WuBDb0t(v z?Y0e9m_cMI02Xf&WnKM{*N7jyYvgD+^vs_)3sCR2dCgSOQib*|jvy(U-ZxZGKMWH);k|YAak6EQzrO zF_gZmQIQ7mBFx@SQvR>x%Ca+e;(pf%-A&*qE@)yLi)n7B*LDVWMxl2<-)}8=iKNGn z#YV-MH|U>gc*C((32%q|VWjufo`0*4qg^-UkR0RK-CC!_!3m#S3odx1k4y>ihu1JM zl{wA-Lw+*=Y;&xkL;P~%yKWp^vcl#Bwr+S{8c?TU8h#wq-s*(=XuR%dH623MVg-&qk2y6VTDq7&*7-Sp+i zFDrrwQ}zsoqLq z%+-%Z3nn4TfP3<66sk3ouj)%m!O_IXWo)>M$FLc3uOq|U2S_HFFO z?>cxHCl9B&$defE0%OKA0iq5zY9be8b7_Re=l-cP#AFJbugl@o(xTFq2ESei11S)c zj0y;-C<7Q2z(NQSA;TOcSx{V$hb}viTPaIi2@zLgvKvj>r$_q?Jg-U(Qg9xpmi=nJ z6fB*m&Ae(Hx*qM|l5YFuon{|G1K;ZNc$VXXNJhPp99Ny0C$;U%)HJ*!A9=M&(T{d} zZ?gtiL5&86hX869u(UywZ!Oh4Y{`Z9oBcT=mU8vlad=H)%&Vc9C&?3uPB*1rswr$v zLMyi3kR1_M>R&=3U=f#^NTH2DsUn)}06dX#-{zb(z+pJ9b^YylyBl^@=1r5O5jOhR z=cj{AtlrpCzg+H?;`bg!jlow~hvu2Q_&d!AjsP!7+EMY+|fUiW(fUTgYaIesY4pWxzllRUaTKg7CQ=@RUd{UlP)@EH5 z_%`hg2Pdntvv9lxTz>8*On&VX7as2U`J8!~{Y`S^DqYa8+(Gcuz(>za4qv4Vf6~ZS z7!n04XZS(k%nKD?5&%xI@-CL4IKcj^X(L_r#TEiYsm;-!!GCV-;JNF#ub2U9tBPUUbGBe&AztM9XZlVs88W~gYW#Kpl4N$wqfK6VJg zSHWPyQn8@^=t;7I+eHAr)>$KUFpbq1@U2+7z(%^%^wwa@4GB&}lFtav5bnQ)tsu2t zWeJ2zlj9gS>=+UH@Ei`iq(%7=pN)WroxV-3>)OUTzzYQaCma2z2uM4~t=Uu*96V?^ zt+Z@|_+B;uumK(!R4J|i=p@-JC;x%sgj~Mk_yP;DKHmZ*_KBHWIsw)S0#Zi)fzDf6 zcP|adA)>@A+-k6uU=kb5w>g(hPmBSp>q%)kjy~jEx-#UtOjSE%t?m0h1mC{J6ccc9 zxq?C+26!&~K*CJ8cdk=*!XUw~5r^!3HeCXAC`r&S-|DU7)~#e=*qlC3F>zb|eiu+M zzbbHIJ{7eAx$&8a!T#&yD!-N5Q}B_ai`!mjH2zcyoR91`qvdp2pwNS*J*sBtN&{U~ z{X|C&y0(k3bYqXHXmnvYfcRqo$vrBQawPoX_f%%1W+Uh)V{_f1%pN8 zG{LU7x*Gk=dX-Ls8CSCaKn|+ieygeGIX%eG%EZLfRH-j=t1~4Nze9VBb4wj&!{*y2 z=Y9^$#048o`|b{)06HHr$Q!F!0>B`O&sgp16a#+wBwHpFlZL;=l59Q}knpH#+XTZb zKU)IKcqOIT5t-XRT}(2WiBg{PqJ1?HmWuQCd@zuxNXyrcEn%!=pGRMJewWF=0T`kW zGPC2pkNwjJ6{k$1e3ZJzGim|-cezvXw+Eu?vTACl0F3L3IUKBi;P$c~Gn&HdbP=mmQ7Ty3Wp8kPXJbZI4W5txttNP_cOV?3I`; zH_Q~chFEpXS z4b6-}cj4)V_e_$ho$a_>Nm?dDwTYR8Rz1r6td}Q&KG&ML2cXX|LE;i?9WV~A@zv$^ zy_V@nmRr6d6ZFEvpxY$&bm9#*5j1`G_RC_0M0zw^-v~6N27T`7LWYSa(08g!x5FPF*N>?bq>WhVFE5yeuJk=?HkiB`&je0# z(SWaO^T+bzH}g4S6E(1kRfyiy0)8$znP9YcV6`?{;?U-nPJx07!GRLeH6vsUVSrhn z6TsXdh#TO;sN?4;j*FCScJy<)AbUTNL^Cr}U0gY6SSGo+3ewVXvWS+ zHtSg8^#k;j8g$X$igOwU`BJbZPPXR81D~9wtDRw03;zNkVnYGM>maKagF21_bw;W) zTYxjfCCagvAZ#!rSDdrc=yS0L*tvO4cf7$e2F)%V0F%E45k>H?y}d4 zc}HZo<}%&3TOfR(T;y&xM{eTX*CdpQISs+)C|%U8HsK9mSV+VBZM%7K8?5I8I`HhK zStLYK?eN-4SWE3Q7Cco1ha)#lsN!Ud7ewk#G$d<$WD>eXLoK&f8Gk) zYG{TLhu1+Z2uQK892S(YYJ0Q>X+WKpswSZU69P;f5#LrSA@7C$H`kfPJENGff4rKh))VlaB> z*)8ezW3(Rz*RLM?*yw;Oi5*a_R~57t^h6v*E182v)Ax63!-{r7xhUGL% zkh_jRrjupCf)1VB@NQFRt^PeL`WSn#mRGSwIaY66pKFqIzxD=oI4N+^v}7)r4_lv_ zy?1_ib*Z8tfj4mKOPHv2r(LkYt&yA~?W9Wmkpg5!J}C^e82X*TZu|nEX(94PP!h>< zlJ-c#51%P+cYMnWl8DrP+4I@oCjp9aF>%m!@x6t~fwM2L0nIOw^qVt_D`^ml9eHZ?$}J&{M#?70 zK<7oCK^>B&TCF2ej07SdZY2(8Gm9QhJ6@VGWWE98n=N1RF>$uFZZ=Az+J^}b6uccU z;{)(9CC=`xWjnx5Vg=kiz`-Gy0}eNNr~jx)Ba{TYAp9`;lCbF)y*pxei$Y8k#Z<)u zn}}5P9%wJLyL{~CW6;Nu(#}@fG%vifAVs5!nVV>`i}Hx$g2#wx|F<3Gg;2z_XF~`Lkq=0Kz*zBF8{L|Ez$qjD14C^MgXwCBQ}58sd}4 zG;9bk*9*#F$okS`oW&tnFi&${No= z4w|Jz3S3I2ldQ6uD*GpCTb|f-LvUoLZ4tKSoq-TUUS;sQpcQ(uc=CAY`Rb_3>Cj$N z!yxB$R{_#}z9fdo;Q1z+?asXH5en5K!o9x4v|*8>&T>llNlzJ~YkKBaE)RA$ru%IznEsq;Al>c3o(8CeJkL%W^FDDat~ zt#aGWx4RXRRxifB0vFxJFOzs03Zwh%aTx9>YJnpc9P`XgnOWRk!Vj`1XE z4tN8pqb`8qs336Zws+!nx^#+9ynT5p;AEf>T2l8x#m2*GvVl(4*y@6JkN{6AXk^Nq`ai+1Dd1h~A z`NFU=fG1f@Pn&@{wiHEm4?r8hUbVn#v?28^pl@|L(jgHM+c>mCaJAJrZguX5E2ADb zY;A6;*|9N~OlT@|aeUajq^-McuMzeOeY7;Yw=X$Hjb1I6KvS^@_&yP2cF6MA3yV#` zYL&m+9++IA-b~mkd|dtMblt>RF_X4tPm)-D!P$2P`>UGH*aR5Y<{0+S_OH~tb@nvM z>OzTqo*1zqQEP8AK$NuA$%8b|U}t-+RDfgbIarg<54KjdPksuh_ZcV|p1t{Trj8Ww zbG)*&theA)Vl6VI*`=5oysYqe!7Rlb{YBAcv7*23D7D#QnT*5DQUk+@O)U#vO{ZJE zb%}7U1_e6*iRLOV{l{efq|Q}^tfZUuIjE-2q9A@Af#Hzfd2!>AbcYqtGp=j>kUY%qhg`+0m3kFWV&n(N(n4O=r+s@&N3 zCeD1(@$+(mRiU(UH2P^@#{PSsECZ{Pk}Kt+CI?oU{bJu)pTfV{8joq7uSlrvxmZM0 zTpZ|CC1bHV?HN7Jn1?Uj4h8s(m>$~Mz`SJRe>3YNQH<7268e%NSLUG^q%t^m$igRw z((8!|?s$IZ#^7{xk8A3cIS>4n_>zHvMN_pH0XGoI-8BgUY!hy-Doy1jhTPL6o7;;4 ztqKJ52IERYP~3I#9aN|ZszxC0Sb)`k6R_Fx4y%^!GubH^4Nj1@)X8l28z}CZRCpwP zfM#rbWC88lrh^Pmw4SQILL$Sb-} zdmvE@M~dM{_2%#FRXWh#bW*EF5n0Q!5Y3n>6if>K%4O#6+v~cqI%d#G`e8{jy(vN5 z0_i2JaaZ{UMkMM|vkEe(B^DcBIkVZqxtvfrQJ)JxoMyTQN+KH$dfDZbXK&AUMjUHS z7G~h_I-cT&L4J-e$M(((#A88oY_Z;i#DXY&nKO&U$YMF33rznml<`m>Zr-dYH*9yeJ90Fj+8E6vL#Lm? zJj2`$2-tLd&(yD+E!6$R;f}2VCPGdzHw`9Du9EMrkaij}f+lR4BxMHA1j>OS87*b> zILmzS`s7^r+sBhDV!3Ey4yth3_;C~zn;+_GSn(P5T{T*MfsR!Nb)iFA*A<$Xz_W+} znYo7a0KG#~{vbk^FuM#B8WZy~WBdD=Gj)_HMq06XsGl=- z>|^KglJ@q6Zh`2}9snZ7|0u zs0@YzCH{;ASw#{Zt_Sf#vX5@?4CRk4pO32CpJ(kQOJ{diGVz#sI^Y*$uL8gF#{P}j4&#t-7~oG9GQvh- z5E?^8rpvz+;dT2GZ5wJs_HDlXN^y87X!$ZczeugY+XIxr7#=3u`gZBwKAAM#{uH7I zS?U7t?tZ|n3HV$A6D2Qf6U$qMo)sq?5_6~aM1w0~dwW^gNw#N1`Ua6t6QBD64VzR7 zjXr3SwcHNkood<}@_Y8@nk=<6?Ai6Yem+siSRmT!w1l`mW*7tu4G$G6jIkW(4|;SX z5p{@qpS|iUt18&C!GF@((C`~5^-}Adp60>`e~nge@E})e*W~LOXPV!ABkn0+Jeo}- zRf7I~whfg4IJ^hmkfB)C#XUbS(y)AP?4?SziKN;BB#fas_{ z<3IAb)Q}@*Vg-tvJBAu)F-=F!M~>~;ibC*~_Fe5Yy?$GQN%PY==a6dQh22u&z9NyD zbPZ7{A?-^XWH_QnE)}9#(b63iGszCAHvX#fDNL-EUXI@5xSK_xpH3$ZA(Mf(8HqoP zdr>-%%zCjSxTvg43zR!BL0MK|K}X;`0mt~BdEZ9O)&D>ZDH%j=lmP83m=kLpl0$Z| z1=g89?|jkyL%R`LmruTW6rz?0e<-V8eUR$`2154Y&sj&kFxzf4;{A?^`+BKR(XblM z=m$P`3~?{c360jZ5(LIi>o$Zt}hYS0geSF}^^5%+FG&SJ% z11~xqB~mp_s|@s=URd%MLyp_5aIMOmM|E!Z7T;W#`?Fq zAwFNTqszTS@uoi@G6n4f0fXp6M?itGB|0PH7;JeBAbpqu&&O^!iV3pDS@-N%>7N%h zZ994MxjFEEFVIKpFLI3v*R}4Ky%L{kd_OzeIAa(T;h7V2J4G<+$22NJDt7DsT6$sN z=fE9&tm?~FeE;-(1b+Jy&1y_?13g=sBMy_pIP}Hro?fH6`HbD6H=o97yk{8VP*nr2 zugpzt+U>=lJc`>G9cPMiEk}_I8ok?a$wT0D>!e}_#f1^cnB6;JOtZBg zD;%>p^#1lH#EoeD{vTJTUz2{r+10vZPZD}TA&zl7@ti}aNBLu=0MWsV1S@f#w}AiI zGMIm`rTu{G-XC2*$yJZq2= z)_xuf8mR7{N5)PbN}Vr^&UQ2it-GOvLsp5r%{S=xskN&xFVEFKXW7{PqFYpsxX~5N z_8~0`%-5`W%; zQodOs(dJOiZzF4D4~SY~tqN(xOjTXCCRI?!oJ(k+cyP~cv#JCF=aCfEV|hWbqU~BO zc2$Va@|d!zecb2sW>nEv(Hhw8=vad~t-4?JG}G(LFZwTEL-Dg;YG_y>h zPLm?Un`;Eiq^3rUoS%&-rikd2gP>)ghvfM~Pkq4NgGRTCUAwe5$4kg$Z<+QJ1Ygu5JX?GfmQ8T_4xUdW)H(zt>0;e zn3vZ{$v5+5)AJ-Qp_3v?uXm(8E6|ghT)Ok3bpRxUxSy~&=7Z`-QCMCqARTc9@gD4! z1lWG{)iu5ZSVs_mivgMjJO--IhUM5mp?6mO<>x;gQiHZ#esdVF!`5+tRkuBv)N2F? ztT5DksPtZatr;dL$NviHqhjuE5=7~(Upd}Bk(ugi#uh{z?k*(*3Xd2-(FNh>*wz2^ zi-gJk1PetH1gt?#Ldj^4C%R-(WJHYZXD^6{+|U6%aen?&Fe*|ELfbg(4_KbUmm_na zWDrQO6TpmE-NzOtr2*kJnE5)Ef-oRf zlby**8BhJGwEr`@AoVp{U0Ip_nQ=ZU^%RK3WEyu#6Ssf*Fl|1 zgIth10E(*p{|do-{*V)NuzU8LHB*7sBIuL@VphDofWZTF5rk2E;b3!Med2e@b=^5H zqw{cU5OJMXVNZIl;@VM({wJIb?(*pR<~l$_^`Gmk{v4gfB_+2+vrVm2Fvp+xLyDm8 z%nD&Z539PsdUL3&0c>iW*+F|R$S1B@ca})n&kclYu}^hlqa zZ*YI>OF3v2vp$yJyP$3X20?$m1hjg<{DXWzNtXp1NXmEp~x5h3g+tD8f-J%5Hj|f~!c!?li*g*7zK;;Ub>< z^j8gBY{w2S4Zc7l?Kd@FA9YAkE<{+$3Ck27yeW10X5Wt3oju=zF@>CY8_W`;-(~V_ zzkckrgYcLMTwBhzDy;w+6k+*!#*OamHq`bJiLphP2UI?c(2q$NY@96{w9amuCkz^! zm}tR@Sh~?J@?5G{y9!eeG^6Al1>+S#xtJae%ExaMcJ4j0!Jc(kinkZ^V&oPrN3`yS zeHLt~m{>0LJO=dS@&xzfko+AXwri|s;JvZpAw7%`p&GvTP$33*#Pq)k5w18 z9`ErD02|gmIl!8b-RYP~1T2^KXLrMxEDugQ@=fCI5%EFu*&z;?d)7F%^$VZs-1cv$ zfg|SFFB^OH>Q%0wNORB!2$p%ds5^`yDCj4tQtT(Mf_a^QNL{_j4hxX1ZXX=Ae;w%G zeoA0;YYY4>e7>RiEu4pM9nf zgv{QRkzQR`j@)EHU^KK&FNUx(gP)y2zr)*ri730p{xuMvqsoVH8T`!p>~R38Z_X&0 zq{buKo2u4I!Jr?iE|h+~#~n5TsE)w9{ zvs;!Y#6>|wVz8r@z`%7Z3_`>K^pAm>xS zf%NxH$Dtk*jyw3}pTOo6{#p&7aE+G%+LdusHUA|T=!qEQ^JffU7pnjRME*4Br2<|> z=u=Zurfp$Ddwcth^p?x8wPvD_h}w@pgXmkOMMckH@_#MtEqtbTH1c_bHvIiqQD6Xg z8H9jo99Phub(4{u2NwG+8~}4QHjJ=8BLOX=s-h(C`C<-8Dt3Uj3&RJ3jzTclCTMAZBNA8|U+8ob_IO*? zsTBGNrhn|76=46+eBx+hbBs-Utv3twgXIuQr5iSH7Y@4pm$3v{#+khg&{dE_s_`#H zO9KKP)>5uI`@dzt{~s9;f5~wCKQgTKSz*Z_%9>4UGOUw6X=kjNjS?f~LV4k`3s?+SGQ za5OYD1Y^F=_m)WqELmZA-q@)7&N>s)nqttA#T!e5L3UE`Sm+y{LHK2f^e8MgMv6Uw?fErtsQ+5^w$QT!HaG!>~?N`|^nB6PW{%ZH}_7pphSYyutdF zGN3Qo->(=6Syf=9rT%)Z!;dZ2pa(0L<&E`SqzzC`#&7!TK9C&HY#06mi~)Ll(*lef zz|{bCpR=xdzfOo{fB2vIePR6$5W_n z=fb-95hOLB5k{jWz)N#TU0EmSvbO@8ctM9Uw`}nNFw|{?d-+I|0BuCbxD2-v7FcVvn&NveeMP=F2J13U=G$=%zjXX zW+BoBoccSqJYjprqO*nV!*bp;3z+Z!Rn2{BSo8vKv%>1qWVn9_@YX=PV+e5U=sTY- z|EE(DY%3MgQ|ylMD17oc>;A#_s`G*lkcoXDLCymm<#l1t)=b7eELf6IThzyg3mm+} z#~?PdJe~oZDe3lfsJhZk}K?)I?GkD zy!t%&rh;}XR}}QL^b@NG?~4Kzt(iuWw#~m9F>}>u5YO-6yPgTT&=EN(9m#4DSbRZd0)mA6fEM}Llo+^Ky;y=>~cHAm^ zU(g6FVdayg0~JG5W(EL61xgw)ZPkg+F{T;No59r#OmuYzda9vWMzsDoVnJ7g)IW4AjCxi=P6}j8!RKNF%3_f|#a;QEn@gkx zjP|%OhNJGth>u$tF8L|^UPk)ipk`u!qjE)x57|#e{E0L z&H3asB5k2VFL0q1@;bmL0{E*nc79SD!JyYZpkVn){8{a6G%P z@r>mc=YOm{y98eE2x*c zGL;4Xs8vU99$KeTF<>_XgsNhIr{bab=6cwNNJwc?vESb@9n||s45zDxrY@R50+b|QNh+TYuEkez!U%nlxn zQhxeV+@f}ua6@kS41KC(AH+sfG#Fn0a;+M=LnDbM4Tnppar*dI+VX z6!&SWVHPySxkNO&n~R`!oSHnqAO_ujwqb1>GG(P~Vg7rmwb;X3@bIkj#08@w%`%|5 zJ=$rUnz7)p05Gy|&LQx+z)ng1f@afQR?PeJrG9lykB`UVghh~M%zDlzZ(oOPu?ET> zpe%x&5Qx|UVkppw<^!u+YSnuR;PIo~S0tes2m@`dakg2N*{)|5fS@hKZ+M%t*LjB4 zb?UgsXjg({<>u)Tx1-tZ5TZH0maw!$SP9iWVpX;193>XZ$;P2qYtzSt$!eN{Y9xm zU79Yx^RVDxB_*I`A&*u9I*`e$Onp9|L`lKkDhE@g$SEgY&zy`nqRqD$Ta*9(G`wc+ zbZ^0g0{6I=GGRCom3y+C?3Y~I1tvh{H6wb3!pi`7h;sQ&im zhfb+lM!}46N6c!M%|Zm{Dfg_i7ADJdyJdFM&T-&xrP%DMs%Iyck_QQ-=iirJHjA;TL@>0u-_W*aq?_@?}w!snTkQ4Mpef=|pe-LBfv7U9%HR=9%g?e`)P1L5+8Bt8qT|!WMOWPqjMa+~92e(o;>_Ali}N&x zKlz9!_asty+;9PTYMkUQ*Q*Sfx+4a6S~U5H$@*mGt|Om!HIK}g-lzOFGUgDk=Sla0 z%XyTBY319x_`7fEy>CSJaw=grfQRVb(xD%+s(YKsxsSM8BN3NN!Q%42)0JZ|N<1#f zy3NzObFQk@B-R}8q&%2xjy7+XPw~Z$tvV6zdMO7uk$lR{b0_87=M=8hio^Fzn9ivN)r8j0M&UU#L#&F6`Pnetz2>CkHeuAKxbxp@O=SuhV z^WfO>j$ukNR`+l==SHeo`nW*&&=t(!@ees9^${`!dpr64WGUC+dzm}fZ}}cH_3q{@ zepzg(YP*_F6T!EG`ss)xGzc<(qr8VNuqa*lh4bkIU|$9 zcMcMKRNfOmM?EiKS&@?O zkLXKb^H6(0B3m-&T?5URnG}KiSZ|f@yeLz%Ud~MSJs;|Gv>+4GBt_sjo_!Vyu^E5Whn7(-OVBPi-rMX&!di_ z_y#_jl&>AxPKlcA7_AJ=m&LGEc4N8n^Q06fBI&Vf$N(zWiltb(T`skUYW4+R;Y@FfG~C|$UiPuv%Hcx(>PM(as0sBh^DVpR+1zO{gvL2l_T%Ga zuGIWF#n7{%Cqw0!ZSZrV1eb@YZ22?!x!$AQo4jx7n83Gg#h_e(KeTN!%*`x!j%QFb zcU6A2A9PklX#RYY?A6WrfrxqqsK{ilmCN?U_btX*1bCZ0IUDNSfZFXA8D!HXI9d5C zGpW6td@EPs2Lj=*hnbWUi598ot$F7UE2sI9+Q=)uA7Y>#9vHJ7!?>GKbHB?Z{k}9>Y>f@(Huj4* zpVjaU*_vC6>Fbtx>H#%_lGi zoi+1F;q?vb5QYx#%GNiGegl7kb*jJ*(1sSo4ei3Fabf;L=9Q!TG4BNNq8?PUB!CB| zK3M!t7s!-vLVU|d4Bfk25(W=xah1u`c!Vf|=FTBz1DGOtv>|#9*0-RdyRr%s3G6Q)Fv?=kLpM zu8U1WOyU-8O`cST(uRqJHlh-W=2Yn9Z>7W7I6zn7)d-`E`&{RXw}&TSorXOI#RkmL z-G&0`qb3bJHQ4VaQM`}vZDvS(Q;kyZd=trP+j(5yr(2^`rdVzbr%QjHog>4nav71V z_u?6XW@i?IbE0oZ(JG}=5CtRjXCM0dVK!OwHO(bU(!z#z!+XQRndJw*M982pO-H;8 z`6^IZRE2%-F)hs=)k!jSeIo;v*TKSi_8LUxLB6nvLNN)cbA7qh(v%ej)6X!&l>|Jd zoqU3r{pMrFU5z;6ejUAsHqy25O7dqr>R_eoSFDo?ltS9O#-twMwhjc^C8@g8ihcaE`BMq&+SXLHn@=9kshn8Au} zRLw$dU;i>Cd(>TYFvn-xdz9gZmm_VBr`QWz1Y8%X=F~`qVMO-w*`~2zYBNNqAmBDCfeVyq_}HsLX}!XLc4ONEuZQNRMj78I{^6 zcQj7#*T_aXm&=%W_s5*azdU>8W}4#la&da{#h@H&bA>MIWg7M8ms}9)`x?Ru-b5ag zddvzmbg*M?K6>NVcJ+yMAy4MQt5G6jqq6KCsfVI=c#|BoZJGJ(FO0w5v|kZ756J;d z=^UqLla9U!`KlXIE8zVjI*o2*P!;C!aKz-(htroHiqGtSpo>x-4$_@I@Nha@meB13 zTsyA`Cg>JTCixhbT`a3irZT~n+)|FZpnO&= zCvW3W^-jnS=Ckl~Ij|WsNg$Prg;(j8o*tshkq#0@^Ih#TbW5wQbz7BhLJU<0;_qo- zB;=XxZ@T5dSPdMI*nlL~L|oG|4)Jm|I`sjIx@k0fz$LIu*Vwc+iEDA|QDL51qwU8& zB-ZAC@G&6Vl_wPw;1692Z=r|I`QS~y4Aj;S<)#LN{nne7fzHHPjc-tmrGSKd+ub7F z!xV~a?nqCoixX}vTP(NuNMs@6)#pYI?ZxcTa;0DJX)vfmKS(i9laDSIseRtYBi8!U zQ>}VrzkuyS7ldx%-mxEM7ok467Q$(=QE}*;aKELl*&NO(^+~V_umm@Zhg(Ho-`w&0ZPfcO(!L;)iyuHmOc=5&XO2^j!&mX}Wvmru3p-JffFD(Z zj|V(L@_T+FD_6?b?y?T!jl)CHCdQ>)fx6B|D)-K68U3)1x$YV$LMcJ#OAcHM@x(HB z-7NxMLAUo{@XGh>-U)C&Ja+2O@sdVv&K{#I0fHszWnuIf@qqY2&u&hIQEyr!2fI{7Pq;DYt(#0gQXCjt#Hxm&v zF!Z<(>(o6IFwjawS9!U$A z6s?w@Uz8ufIs~0yzwM%my_-9yO1>FEd*pzi_Cl&Tw8A9!(Ic2=?#bgngki)4 zQOauPf~bQvW#79#lub_So0SGUoaC=Hthm@${9Q3z=5g&re%t*!n|KuKnMT_iYZldV z8_QNoh_)D=TfG5fY7Vw^(|2a^rd!>&%-p64LzEhC@n(Y4a$i@i41aTEz0JJhh;Z)s zQ1iPb>Eof7tzJ;_Kj8gsR`$bZd%%oDnM%(OUajm^>JC$AyRWv{KN18{BCx=0gZ%88 z)#hheZO3R%#s#mm162>l81UzhzN(u|V{eB4hB(E-e_u4l)f*UpQmeiNFCm&ucOW;O zE7*ridZ`!k`A7sc`;Y~09&_}ZAhqPfEHyT$pO%d<4lg01$b1BsZVU}wto9*H-BjjN z%kAaw{O#R*H(xoNIQGGrdZ5s$+cksuCfAV1Dy}w4-Hdq4M6Jy%edO`caI0T37m5?S z;qR`E^sJsvJ?LC^dfWLF3mGALHjnj-+6hX{JYhxoA*#Zs+c4M`^W7&b&z=}&3QpU% zwx|D=E;Pib2^LJ4Y#S8(yCBOrMxQ z3%_HC>Hgm3A$^ey=jc;>w9||%vAb`{Q)%YSuL1uS(QAt07>!S+8XziYc>tuABUo9# zAPL@Y4-gtD;sW7=Tcy>%;5oV9mtjY^8(cBi#j(oDzNJ#LD6dBI;639@`%0a*pB~t- zL-`uV8ChJej(KVQWpCXm<%!FTmzU?4$lOgA?q_=GDwE;%Vlz>+Fe~(IV*d8i1Wsm= zqTOl43z`Qrc5rVqu1TdLYEtDV)}&ts9Jx!TzuV^G8tV@icu43sfSqSHpv=6?jsdr7 zVDdKOxC6mrsWmY}c{ehvqL@g@uO?Kq5gzJ2@*Jh=~EclRzPUNEUL6R8w>4hLk z2t~d7Sm~*VM|P?HvTU#_;cyR+mfcXzSq*L1frn2%%5N09wbEiPF(5ekc-~od?uJIlSDKf%;~X-2)Ym{g3JZ4#QY>YkWBw1DyJpZ#Gc3M7%LbOooC&9m+-TLd8I5(_nYL}4JfeXxfknHA(sY~%4E#Z)oWceK+@b>$HjQXy_ z;cxgsL^y4WnT#@T;LTG|L|?nKtNP46Ymlg8i{NHuqka}HFU}HBOumt6)#YD$$Ahk1 zo1Jvr$0Gpk>2IOvD-Zw#G4k0VLw6=Cu6N}i+IG!P=BEJE>5@DN?0!s`#8`i})+sscHg6;-Pr>U#}Cj7G>v<8v096 zki^?dwk#0(}*v%1TZ{ARf!->*>2GOx_?Fv=6|O|40Md z@MMEED@kv#u7|C4td(+P!^N0?Rah0YJ~6`nx+OF*7yyFco0RcHc{|Mc)#;uqoom zU$0ybuZxY1-Cgx&9oDe#<)4+-wvIvAdTsj?;2r(gZ|3{QOkbb_yHU!;3|PcL_KpE?GkxV4{p5gp*o`1Pwgu;p@G z$|QvYjBJh^1ZOd1DUJ0g`O%hDLW1hdX9u5ewW}&@pG)mlS8Bd{e07e>RFj|c{6+Ry zeJTVkm*+oVokK$o?E8*EFId&KSXr@P(Hdy7V$S|jt-^me!+p{a(HfnurYq>fVByq2 z7S+4H^# zT~fQ+D@Nx4ae=C$@GaK6j)1IP%{EJewUaV)$MzL7cFW)EazocKD*SyRGrp!SsThz% z>@EL&16+4RdFJb@K`2?MUHax6@7i2o8OY2pcPpp)0$}f^XrX~mzy+3ckTwIE@U@Z1 zWT})*N*VZbQ4k14yAU1Tg*w@J_CVw<-2G!x zTI&<&fhAxj>5!C=I2Lq$-E2y605g`pj{LFYJfhA^<^-)=*wWV_5SEug>q2vu&i?vS zVO9nZ!stb?uh09#AS|u{24PKWKt32~cg@0=u-;(ZVp?AQ$57legvKREK*k(n;#@HXiv=2w<9WU90%9RVlpc>bv*tzO{s{??hZATyB`R?WvaL$zf8ra-=B24|(_9&Z0N#(UJ#4ouO)IpCmDN&YIMY?=}oD=zP ze274tFD-n5Pd{`+I8KK$2mcsiFJ;j(Z54noTOKSXsvTHGaRLD>_W@773sP~{m)({) z)_yD1l6J;G2|}NqHJXl8wigaxC;0ty6i z@!~gGtkMn1{<<6lr@*OBj`D}FPDo)bN2Wi%L+!aaP|y1->9Uud5Vj2Bo;8r#Cc988@E{Qy<3Q(0k zemH#aIW9r-;JY$CgVW)(uv$Y0(&Ra5xzEHp`z^Y-+-V5Yg| zG_DI<06?p9%AjTL>d;3*U-8l32FFE-<$Z=LwYQL}73PDylaGieqW0)UO!kJ2WQSE* zbYa0uCD4To0|GutC{Zyx`iYHJu^>CqW!>+#(@oTZ`*dothpbX+0R<)TzQg5{3#z{? zxv;kR+rq4lH+qto(m#%!%vI96o;?z-CvbcE%fa@u&yn{bWj;%9kslws1Z}RY`Dl4g z(<_5saps;R*?<1vL3ZozqWlP{YYIE}+)d~hoK?&7edk5ZpE2|D*!B3f;jiJS9jb;Q zHa=WCYsRVfoeCzbT!yJI}(XK~;$wf|OcW5{zXGldYEGuriDAwJ;>5ygX=0vO`TU-Q=8 zrYM&~KX_lj0eaUl-t)yO=ZS#%qkL#Dy_&@%)o!QFrzwQTZbyU-1%<+=-}qLr_DF3J zXuC_h4-m{%A=n3$a~!s=JedU!_pwfzUo%EA*ycx;*Rq84Ib4psEKKlryl9ZP(7^1a zo>u0)zgpu5<^r*HGA=BQRed83#@uaSu|CiZEKx3dhH(*>c~FAdGeGhh{7iW1O) zA$1@>#PS^VzPsz4oZR~s#BR&uEh2eIzm0HDbSEzz#Ci9>)RlXirSFpVl*T|=96zUq zAMofSq?UeLS)HmZPnR`%j=X@V4a>W3u#2gggZV^WP~0IY*tt_6+zN-BjKB0LD>u^C zY#OKWkY>+Qja4PkP(ryil&{A8p13gQam8+qRavF6H>L==X2p>n0uZn22MZR$G7~tk zF-u*rOD2|RZK?9|nd!9KQ-!!qtvExzv-JI0s2s-w3fGtCnma|EyoVh+G9>Hv-b{fO z?D#^9>uIurc&Nko3j6aNI?M{MM!D`K%kk;BmF^EiHuL#=2+LnS5}57Bu_`f0jfJ+n;# zsK|ECx#PO7?^WNssk;0A^g{gDA6c&LU80oy#;}5&ONB_}eENo;{}@?6w^nu2zGwdY ziX@OGZetUrB4L7L$Hrs(TqERT>|P6(qL+IFx$CBPw~w4`@~imA;Tx^T_Sb*bSk>kb zjYqkN_s>K&mo4}s74lg!t7y?e`oN3^eTx>PqM$;;f!fo~U5_ygw%`rYJJWEhFe6S| zH$-fFW2pX)7Kh=;9}pSNbx-BLrTpt=ic&n#3bl00*0sU!jSz9Os|LZonjk=`7cOut zPbGMCQig&%xeif2txwmU zX2k2}xtQ?#XwnyAO0(FCQ^0duBMauEzmFzg6~M)G2WFi#Dm_2_;|r;&8U610Q@2|E zg3{|}2^(r9jPor8^ZUiUPUBVTT|`gg}UV(sE#{c9&!V` zprS%YNyw}P9q+#{SA!%(KRNzdl`b3X?ADyRfZU>VUWAPDg{P@>ll`T}bFb*8JsEnk z9nuZXL=awl;Z42ZV0@;ad4=~6AvfzQQr|Gwn}H}h|1a>(+rCW zflYh8WjIrZChyvfMYBM2JyDFsNT>`pV}Expl>9KB>}%=lAhfW!FV72Sn&$1Y3a;XP z%`$&UIvsHbN7%E`QN`+NuW-~aE9LvjkLv^c0tp&K`gj=Pll!X(8v9EN!VA?aO>IJg z)Jr35{Kf!z9diB*dH9N2ZiDm4(K8x(h~QtAW@ti-Y$#s8JYXgpsL48?CaW#)_J{d& zw5qA7VnW8`8t=OSO+C;A)@{hv^_bnKTaCKIKPzalAcEMW5z3@7LA3ZbhlpFzdYwt*kfdI%y6a6B_m!c+tYEt?ysovc=}* z03~oN93)uzDRjC3S>;kalzXq~(i822EkeebbR%BIjh3 zYLLM2dEtE`!wB8BCqoV9gqhBkU~=G0V^U# zm4L*of6%eP!iS~R2>JGZu&fJ@_}3$pwevX{Z`3ge!L{@@b7=AImvCMXDB$fuoGxnK zpl>1fdx@*~y0vT4d}D3)%OQN5tZDU6I9}>hTJ_pI&(y>v{ai?{gVk0+_jWUH57KEP zXB(;~ycY%Vw5cgDK-805EmUs7t~9D6lAc-U_}g1m%=8Z%3>>x1`H013g)=hZEbHlZ zzm^FH51cIsoOh5KWo9JK=6pspDImla4K^62k`0<_oZZ-$(4&(@nfp;1f-9(rv}*(0 z`p+iLHVc1itouvZF!-HifX=r(I9Gc`cD{|zJf3!lsr0SMN8w!G zC{=FP3&X7tMrry8@H>Pw2j6X~u6M~T!YSgG;B4mOu)!y2LM!ieJ!WwMjJ8IJ2%LPI zt=&Ed-gviHb~$}RzUW7#bj{A_>4%@=H?m(o;`NMa(p5^i{a8qm&+)<+tk&d1s+ zpY9IF(Q;jzJog~sVI2x-ZQpim3XO$ZMdS*x^4&uaKZq|(T+fWyh)75NXsQY#72ayD zxPEMFyM6H$IIJAJVZ1N|pn#WP5iBGmlmWt;`#(2;l$1WiZf1x#ni$ISCmW+d!T)6J zJRpUsaBA6_P6=F&8%l7glkuOk;?gkAuA(KHcyxXtlig<0&NRuq{fvx-X~PfnBEt%eN)bP81;iB4)=oEH zWQ+Ly;ovvEUbk2r=(g;G*k1~i*|#Q?n4F$Jn?IT)8=CWgpba%ot@HXZ7qF38h4@^y z%h`AC>gl=DJk1Yfb&fk@-JJ(sT`Yx4JcDLHyBBo<)eM476)8(mDZ7dR zp?N1?tM=sW)@)w7rg9D8HUYxEuFE@BuMMSn^AHq`5UBIVi7SfutsZt(on{{IaPG_X zL*gnX8$j7N6WvUK{&|~&D9v*|Y|Zmbo8Wq1Y1jY^Hp(f>nd#66dyBjrT|$X0aGo}# z@@~-dyZbg7q|-5|4hp;q!-Ej<%2%=7PfpFa(KHvpd zq)##)4~k~|nCt$cknSa6@33M1113CA5e`(&q9g?Z8PtV- z8xf`0r&S3joNhhTP?2-enpCJKJ^Rd7=i;2+b1C|eP`r84u%vD86HyhrgyZ6>bopaT zb$jL%zlgL;*8D{Q^f|LX%NG;}*JP*N__)j8TZI#I@<~2&c0|NwjDUV^o=?b>JRvs2 z_^QC;=B>ql4x7~K1(3@tG>wwtoSTxtt5Z+f6)XdR5hMuau(*MKtS?oI?Lt?IH5nUb z7chSYv-OT*yrRwbw~3%kd-#r?XPAKFkf=cNAyLk{<(Ps`e79SQFPRtQI9<3Y^7P$k z3l5%Rta^P$uUVc#a0*1y`0;Wh`4>R;JHeUKH`P5cDuW5Lqwla^lXa6fWM8DJR6)gN zmin^&?yy3AetH_rwAYC@q#UcI(N5%c*dB0Qb=el8X5C2V9{O~SaNJ}M%p1D?suf3f zv}JpM!^Pxm@8J?+j>{es5tW;oupR@3C7NVwr?tD20Ln#?)FOYY$sxOx`=VW4vgACo z$5#(q=`;&s3J19MdbyHz6;&QqdpYekTu;_LZ+ zz2|?uA@a17UQN}pdyY7+l`pf@b_BAt)(-6rWA&3g$_v;W7`mgxmai9ql6pt9)4S;r zGHdB%y>VeFue=vXMP+cvd3~y0tg9Z3t=-y4zT+4{m!WD?jLK#CSehA`hi2>_o+E#C z$Wz>Z9vhQl$=TZ_UPj*&b^=kWx0n+~lwTUoWHxV&cVqi1=9OFX>qj z!K+|mp~Bdd!7$H8WCu}@&{o#!5V$+VWiGA8ZkM=I#cfzU{rEw-%WqF7Sh@>1SE(S+ncj(9;cTND>OjY z+z2IxZha*Wbn+k8p}!+|Y!W{&c(b%hpC))Y9^tl2&$+#^PnjSpGVKsvU{r!XS_*># zWd{x`J)%fUBsgyc2FJkqKmoRy+`iXJ2r+!enB#Fxfa$*leQvGZOWx?Qsmi2vjKkHg z+j4I5V&$_eZh?6>5qVl7-69y6=~x|)IUDIjP=ognC`SI)V{j(=zJ{EJh374FHE>{) z#y!R(At+f{Ha7RXJpkmkKx_{m=xx40IU?!^f{#|ItRVugqVW$i+#T9SDI$TUXLjDT zV*cFUV*YM&A1)vgLpU0dGs!$USf9_UKRR(=qrp9%|FD}_eY}?5pa7=97r-fR=r@ww^mDM9&KCC|f)c`b~ zP;?NvPa<gtB^#gWM(o=c&7V~I*+)8;4QgMt(uM9 zRUi<2^VP0a)v?`y^ObeT1`rH|wP;XGatYHHVNIzoE;X=({HrO`5Yr@GBy`Mw(uY3z z+o1%y52`OR>|=7>Rg0mHjGbHOuB><8z_K@d`;o|g$AM5L`t`a@jc&3&P4e)39H|iD zQ}52!^*Z-a%!nRxpEupa6w59gHidE3rl`sfTo$k-ozu&+J41}QsVBC9x@oBJ61DZd zTqYR79KO9xtgrhxXTG^4qQy72KheH1y|FoD+ zr`hN$%hHpr3Ki!s@28n>HLFqCA}ycVc8}9b zAGYKXx$m3Z-i;uU8$!AHP9|(y^38rzo0j*Dlh`aksfpp4D3Vt!tVvtn4Q}jDc{Kx2 zS2UQYpc#?}oy7$p9-_39%&qQ(LiRL)V@=CbBv*s66@2$;khV6b9|$efD#h!UVkc5H zrje~tv-YGN-V2N#5yqAFNEx~dhf;SjUNjMS#;?2vdX!7@b;8F$5nTCTCVN^AQt%{6 z5?-Jy`R+o2VSWwM6fjvw$;o~Bnf_~mth}EPiYa0NiPZ*g-Cc7`-sk5Fm~5HE3HBod zoe)HL3;xEJWmY|7&z4QnUQTxu(E&Y5Sj#tyBc+;@D%k1C=SC9L(N27h2A~T4UQzxM zXEyD{r%Sr}p>rhWx;Pl^qn(SHz#p2uHG(%E*|(edjZuO!hz+awl+Y6L_+e2Lzyjf+ zbR49Awa-kvyD3*mGL!kF8xX)0rT_ezPo7CHT>WQDnQ7spb2&o7>F#=?lg!g}PfG2N z7@pCHiNdlU##m*S9X?GG{SKNvFiw)E7$26}C$ugH*bvT{RL)zCvCDmPVqd9ni*&zK zP;q&3sRopW-^lV>-Xv@hv4&#?g#GMnxrOVmwmx)V70I9yF<=C%?^7p6tn#;tn5AS2 z{xWc0xk$cH&JGXXb4S#7%I?e+Gl+iRy|o^;v)}F65>pSxNrJ);j{|3;Jr^Q}`n?H{ zn*DXqb0CVy#16^BiLK&@hk?S!v_Utq=w;ro51d)(>uQ7tJ%t z$Ts;1ztx=W9%3ulJW6gfqCX!t0;TyoBy%iPL0Q(Co=( zxN3Bw#VCfS#fAqO$H}{Ggg(dnrcg0~iwwR`$SeAX;g0aBf;0mZ=WBknme{ zSITYRuU@CcJT#P1M&>IrJxz8`XtU34q2r3VsYLiQRQQ#hmz$~l^>atkmRgX2iWUgl z=9=e#tuZVxU}$VCw}sBrPrl7h0F*ROKWA+s>*{q2bK!vK?>fdxM86v^bjNDXow4ei z!}obHy_@DfN$r*#D`G!aE|B${FS35jopnGyHw2(FfqtcvsCt~kfZhQG&r-oMzxA*Zu{ZA$xb75=zOmlf8 zf!NAAMzS+XpD0)-ax7|c$DQqa&((gUY25Nsi+t{nFE#5Vw<(q!@`a68N|!uz9dIU+kF#p+QR| zXG%?5@YgDYGM$X2VvTPn2U510O)TrPXc4EmNW?3z%J5$*YlMAHPkN^z9?J7aij=p) zqYNpsoM)Ft<26bMr^L*XHRNNXu(EM;pr;c)ppa~A+8qV_N{$iv7Qc4#`6&`Qlqjgb z7gn@v!pqJWMXE=JT7{7L1>QDqkXm`;V{dv?;tkEpy(0#t8}$fQA;k8l#olAyohGHi zi*Zr4^4;gK>N`AL2$?)4qI!qIFE$S+X0!0E@~U6>X&F!B+A}N)P^l77fLWMs_@u{i z(D0hu0q7Z?i7m|)cVA|)vr9rFWfAn;zzFaTx=;9TIC-1V_~a1c(j=`aWev_Sm2TUV zu>W%cVCmoAU|m=?Efuv zt~V%lCzLvOspjrkj>#hOF>8^j$XuV;g5tcfH4q_=OJV!+g`GCnjBhM)+MsVRQ+Kc+ za^LyO!0ZK1-<3s$Xc$XAKV>Wd6bbLmghwmUaLb>%#cRE=FRrE-19!Ifx!|tNV%pY^A za_p8^i`169K|#$V+6Y01S>kQbJ0teOUb1AcUe;@rYiA8twloT?;D7+v?OiQZ@IiO) z9KurTUXeL^lwA$%+gx|LTtdM7fY7?&tJZCvUl+-`{+-wBmz0d=OXRxUROfT3Lk3F{|=1j3Ek2|N?kx?^| zP3yh9^ujDktUIGGfMky%;a#^%%a^aeblcqWzEZ3KD57iqKIX7EeG+J(lDwL=c94jG4S&Xn56sH#>bF)&%wQ`@c5kx7|Yp(!oSeEB+}nodL

xT3vuY@3ww<7P!Qin)Fg0*2b?`9{*|>GCvv+_fq;!#LF73gHARO`lyLZ;2TZmCo)Rpy5gkL>F92pWnk7tl)1zhqqZ9SA z2zNRT0W{^O`U7y9sP--sY8KzYw44CLw~vd3F@9m9FsM5M2#U!*FA3~uIg$iHp&u8z0 zk-=^WygzL=*e_v$lxc*lQeuomKO_YIc*qAzS2^oC01QrF0g&IPK6S$gm&!lizXN@v z9AY#8M~j&<5bV9W@K^&$w$qPJ0G=wTf;nvTwz)fG6nkNZcEJ|Cre_K@+qBD%Cm;Qw zPd|anKe}nev%?REc^%OqsCJtj5R2R6wnQ352uXtW38#J3?0%(7my~kl;R^%Za2} zn?e=6tIKsz25k%bu`4e`tTh;Xuepq_&eW=I?VwYe` zSSWTvf2@oJw;g*W*$0f@03C~Kn_-oGmAW3AmS@9}P`zhsv8^vpI_#Cnhr>Ym4DfO` zq;x;s_MG>Gt)jj&Fw*`Ty(Oq&G~YGEIWzhzSQl>#Xquz1op$~@JRq75E}2=64Z;e? z_;nx+VF?NybkgDx-<<8(x0Z{kEKg(3pi*3tSRsI1g&L4MWKimfOFY1X+y)l9Iof37 z2vM88Ol0B)-LkYTht_+g@0wBzlEoB(I2Cz4qzF9Xc$V9XS0M-=W<=bmM$bSN(E}r& zxNRX4%(RCn`E>crpH4b1@uF=HU_xC$fr5GF80Wr~$q-%Ja#@qp@YuqgEriEjR>b)Q zKmae6(7VFF1j-r8*iqWwx;uCIcT(eiO+A3>28>@Q#gfs!kN?*y|7Ck-TNzpCs9Ou)MlcKB2VVllW&a+~d2@rm#pc7+7wD)iVU{LY9!Kza}ua1_MrV1vhi-e)a6#mEV;$H_j>c(uBdy@}66cVPb*js)R&rsdN470N<;%zH#MNX14*i|UR&X}Zc>IYaAu_)6<6RM; zK%e+)z*gW-i&u?~@~j2-XNe zR$4%#mE%4Va6{3Ua8JQtiU5$zE_*z4+ZsqK*p5#~Wf&jx%Su%U7KLj54;6Fs0LmxWxUR9@~SGG>v`OU)unO z@qDP>&2xaZr~!Z6NZep$U!|SpFU#I9WqSfE1GdOM7BY)7)cZQ14F|c|10o$PDb^nGA6;<qqfl@3wE+Fk5aQT2?>o=mS*q{$CQXl zU!QZm-U{49OVEH{q%;8#d8qQP3Um4v2JW)()2D z%MKXe*#T24Zg?Mux})^_oVyi7uW*@WszSLxET8mcI{?DVXO6u$(}BsD6YBNX%b{)P}jvbSN`oR~= zd$8{ly}k50BdJW*%6LBCii6WS75!A#@7gA;Zi%QUejP`hEzX|J+U6jsjbvaX>-l>Q zA=bFvG8Tu(-qQAjqNBa;R|xaUEn&7(0(0E#=*c_3zI&toFo2HZ(yuXUmw}9EuE4Xn zbCXPlQ~IG8-Ad8UsiEJy7)5t_*pXt96JS}ul~;zar1g8eHrCplGTFeSmIt6*65s_9 z{v~&mt-Wm8K4i&D5j)#@9*q;9Unul1II5>Pw6%7yjvTA zjklu4l_+k{URu7|Q@I0E7~yFQP{~l0vO15B2r3_f)uBKGm2}dm)u+=}}8GoK0~Ee!ufA ziY|OIzBV;J8(D;JxDy(sEbX!Mg8cHp_XA#0>r3UK0eb7l)@CkmJioo(s@jmWv3UN= zOn}r}t7B(%290u-Ha_x2x7%owQ+`4fpv3ysm77@eJ6ETh1_xS}+|jfAO8~z*>a$;;^>9a5K3}rCmBa6qCEp$GpQYENRpX+)h9+b1 z0pjaU!@q>O!ve(DavW!e?UYkDzSA^bOkfF$>5_Xg+HbdHb6Su?FVnx&qNl)rDsLo3 zNmI{Wo?$l}jk+Rou>WDKK^x0+pOm6E#sl|a%9(-HfIjy%zb714q6x??I5+ylYF`D( z$YYo4UI z^6C7c=#?#JiWW`}mfsDl9*)L)&XnGn0Od`WC^bT@k1m%IeggrewAVMg2t z@E>Ve#1-ZL-tR{%!PvnqgNO^0tCyPpQWT2n9w`I6@;BPZ>i}5y)LiyET zV4DC9cC1CKvGB{t$`aLZH+xS$KLfN=fmu$_u*jjxgjSsEkdstHsl*_Q+8zdG0IyrF zX_P*PC7+l=5vQP`MbDK>2_20cOC69=Dw+xUbb0-&Q{r!QQ7)LGR{Df-cc(D@T?Q+z9J`kI1 zZRQDxC>$b4sHm6YDdbd@vsjttd)Q(`QpKQ``I6I@)*tXKkhq*)EneW|5qJDqOAnGx zb;XNX%P`N;5|JsAVr83|S_m({LQW(i=mhu%QyXK5F%sN-43mcsC0_av2hd10hy==v zP!us?=oef9mFn-#92gJ`y<5cpO+=Xu}n{(r|E3$?Dha&9)g(tMFii z<@*(~sn6 z?$!w&t#|`^nJ-HTg$Nw2+TIyagkXg>kvbuaoVxX#MAfsNRgB8Y2oHR zt6$$#F`3@28c1ppm(6Zs{_L$!Mfux4D^Q6}aNL}z^R)JZ17zC;@K9zTGqlMES#$sE z!(VwLG{6kTruX3c5AfAOt|HwXw#>DA>w*Sy%4^j5dk|v>KRFqPmPdEnsvN5v$$T;; z$xvpOC#=pk`$hhFtZhGowY!S;De!xcYBAmW(;*u@3kgcAU8~p^-BfNyP$a&D7IHZq ztO!v6e$wC234bJ?iYCVmiMAf$4+s2FE6gW?e=ZQu4e8K!30Sf{ZIR#kOMp5_HX%BX z|A&0`cupT*X4!D>G08{_3)ko>&AH)~w#w`trU85b$3x^Bqn!GkNf z2v+N>GM_#Y_8nuElvIo7h^rdH#?&%9s8;cBBP#?`{2k_^m&UamD{m`&#$oJzoMR@E z+v|xRf<)G*%S8RTs@3cem=4=XtJd~iA7 z^FBk(8gfL`s?K4jy_;xtxGKe^oAV1RHp0)>GUjuD(rPsb3qk%QwgsZ6q&`+yd{3Dp z^%95%FE+$#^Xk7MDLj25yi2pa?d$s{St6u~xyB>{wqj9P#YwlX?NEd-Cy^UHx1FW5 z@l%6{d%R@g0407m70r=TB7Z&4uyr9}h4Q;W>Vyj0lK<#*A{+>eAdvYmYGO|%PoAIM zItult#pZiK%a1_{Bfta-o@s)&@6L~nPU&UhZ$mcWUP}@doDez%xTnTS6p5y zy;o*f_sfOTe8HYt>Q^;yqP`l0lG?i;o74KP|F|diNraS6zj1~&EjBbcR^ldX_hL2+ zPUtyoe^m6n9$&-Xr^_vaceM15OTi||^Lf*o!>+#=C#LD`;xDSpPLYIKtut9*OD0xz z0s2+;?Bs?6E7Y>-+E_x~wlB$Pk90N0o|R|-lPDT?FIJo;O$AgqlCOHM#1b%vP`)=I zP0tnSe6+Ge8cg+7{VIzVm3z0Ix9InJq$2umBv?ZQdq?wnQyrH~qD zbmR|x6482hOvnBdI$qaJi`FCh6zwx-+x=gQGtSPMT(_dEb%UXrQ32)(y!;VTDERZq zgyPq9b>&%?I+-U@B4m;7N%AR{^y1U^udL}^nSb}!QHlEv^ zu%R@xPawNoB!0h<04qVJe~UK%fXulN4Ekq5)Wss~0S#tAr5QijOF4eMK@B<}4FJd% zWP&=WTEcRFVaA&2rD0d)et&kgaG~qg5)?f7w~o!Fh=*m)m*mf@Hoe<2oG`^S67gp( zn8_`}AMZpxNw3kXL5`DN;K#{xBFSqCZ^@n)1wNj)=d(zrJ8PlfWPValw_a!@FInAs zSb=BlE^B=ZUgEyU_lJK82p%ZBR(E=_C4vY!)1&F9mE|uOx@LSoe)8+}-4vbqisoLJ zXaIFF4jEMa7^A}9fb%^EQE!p}l^poNevdQb$mh@ny9q~$DX^^~Ns(nyrUKGNXC1X| zezN$l_Dm~%x&Y%(4^fGB-GAx(;`f)b;Ij{L@e;<(wcMbs$V+1F1owH|e$CQaj|cp4 zhjX&P?ujk0z(OdA@DwZf*l74CTY-Z8MeFKTLSDg~VrLN69yK*vTX&>leG|StKZqXt zWuIV)PBXvD8WSV&qsCuNzW)Y0|0nu5R)mEz+YYYG7tfz?v$X>4?Q8!^?8r=Lb!jYR zT_G*!GzxA1K+uhyXzcYCMd^rl?a4_M7w_cSMwgSOgtH?%R4SSoal}w4k;OmE)n>{; z87=&Z<5<~&WdbP;CAvmQ9^{M&+j}uI5Hs-6VUH-+dnE5!(HLOdu}>j+K9F~Q=rkPY zzvj>AFqyuF7*iVdlN4qATm>Po;lVeeQD%M6cnckd?(5Ym9HH7Vm*@ z%npy3ARP6S z$*(MiWf9uTIb$<>!Phy8gfy>PhdtEQ8Z?W-Uw-st#?akQOU5f%hpC){CT0@ByV%qF zJYQZvu**#dBViJ-+3ypnD&8|Kw{5$(8x67qo`6326FXr$3(F{F%VG4Ibuc=>gSM^v zSef0&tJy56b(DMfuo$d`%c;_HPm$j9C8Vh(#y1$*^J+_^#@%L}Trq8la#AySB$vWR zG>)q3&dD1a@h?1p7jWGRP^5~vaD;0jv4MRrQOJ;lj(Lg}dC;xKOGfW{LS&((J@@WD5J!3X2`?I~l6p?YUS% zh|B=8wgHI}8d0FRLtwfsa;y>+g|&Oon5R+vsM;nfqx)Fy@2 zAx~&PQfYqg>@}TNB7Aim{I9(YKMbG9Mlz@N&?qq%zRWdH_{pM?r5ttd{dnW&%F7nR z+h=mQQj+k44n8y4g_@;WX3`&FFGJyTG2-tM zg|hwk(x$#BuXPi(c{y->1xQ~|bA~!Qyi_ERs`&ybwA}AB>V91eGI-}NDc|Ynz;=Fv znZGjN0E8{ZAP-uh&T}@Li#0Gcky;Ml`~kN$!}Mp#y21&b)$esgCLdOt#n&NozZ;Sp zg$==5m=|{$BFo8~9fOLe=87F-!^QV&#Ox{1+QC5f8Hae$XBF`cWshvQ*6yZ*OMz4) zoH?2RZZZfY25L;)0B$pvEnKIp!O_&KqLw6dzFBQ9li-u__L)w%MpmBgVRCW?uJOSY zH&1EH(8C?EN91VKK-YDxS|m?`2`y4K^yjDAGo*N;VhuiUbU9c&qAIVsoLN4j)R3S0 zgKwMwX-5UXaJsNx_SDQAKqFKhb6FOMX~06wb࿻$#HSI;;5vQO--(!2t5zl(2( z1Bc64zVOFu(|FD+_rR}5B_lPJfs%_9HA87STw=^o5~L;$H6_SUm2cbS#j167p=v&R zQ_LP`sa;y_>&+4M!ia z2}5#~v<;UnX7-I6@d-+to%esm+eg?#)aw9@m+QcMCf4SLvgP+i}$>w`Nvs=dW6j+=cc|~_g(K}*6*bL5ErIFFyTi}Yrdq+#%zoy*+f(US&lMrCGDn3>7Q}-J%ZakQ;yw`p& zZn=ar^1{=2Z-|g47b;GldCB7JEXGeqsPNIu1CS6#V^=AwpKG{b@osS+?^UVyKkjCL z0W;vl2-0537`BvIcOCYuVy@ZM;li8Z5M=@7fR*ozGRmIsdLH2~sRk3&O_s5a>&g{1>18o<0ZDbC@9?G{?hOZiLmDFjlbxr`!Ffrz;`Vag98oU8V0cHDwn{gD zc*CO^Q+s|!t~IASJ`k2?0tbc{b;Qyq-%ybm-*ey%;ws;Q|33EfKW~3cf(zGi$l@Ik#+Zvw5q|2jZ$HG(4F{X%ls+!v8kjqySpyD=GSpE z!%Gpj*D`#~9fX9ohi_;F3Eq{l+h}BLkqYjUUuFKOL2qC0+KO-~NyCBZ*~K12te3&*0k~jK-?%w}C>W>TH~`g(!fAGDdX* zMqvF%TCF4`*8}Wa94Mq=uPM!WX45@zir~`a&<$U#)v79DB_|z zn`*1O=bzk~ehcHIDAqGl5FkdJ;ltn+Q>e?!Np@l+U8V)7BHL&iR%(;P`KawRWXEy)7%%0~A$9AZ|@4@bUBGymHOIdBd!SkIl#;bypA-!py$CLAkT5nHcW{#Iz zx1iXETY={vN})o_5+b=5>*F1jA9Ca8ZhV{lyfGJ$+L1FlP3f=C@@KIv!arW(MYo)V zun6dy>w?lW=0QIjj=SL6mAH7V$skYsxwQVuYf<9T43Qp73Xw{ab2Wz*epei-{ zW8=*LB5CsmZwG3FAwM&e9hLq!3o`-c0D(K}+40u6iMaH?z6dy< z7v#nVuAJw^=@<$}D8n?;k*Hu41A3os$FV(WtX3FhZJ(p9?o>H3BJg5!iK}&FGwqjc zt?_X!(!p3Sl0d*AIkU#^@R~kin=hUi^|G|%i9WSQSo$V5i?S>=sM-)_G)lPmx7-jZ zU~9O5GyStUGw~M{j`1s(RT5uMtRn4|DNf@C!dHh3n8qye?PkE8WAPi$5G{4KP~r`))d@_b ziM9OvVV>6WZ-4j=09C%*K_5|0+KeoZpS;|&HTt32~pDAhE^U6M)h@lCs#s=M0i&oT|~OS1Mbd@;2Ka9BG8dm2drc zW_`!E7GPFSNl8@w@Mcw2uc8Pg1HR!|nVa^}G&t>F`Vh!?s`KbG?-=7pv^n%XI+cog zu-G;uwBQ`hA$30Q$Uy=lwv}-^K?Xyu9IhYBc+`%VokbAW8Oq`~-2E{?Zz8)o?v#@S zCT??c6H%EdCmx@{foQPNxJAr4ny_cSPu?FjY9waUs*_(xrle-=c@8dI`N|Qn?0%(L zMl{Dau;T>|NlK~Xt*>$kfb#hV?k5rmybCI?S`RkMH;3qHoKTrQ2H9D?D}kcnM+dmj z@k-u+Uz397Y!cPZEaX`9ZY&harnl&Y7K*SV#d>+YL=!=>@~0lNVIG>alD53<%NIm! zTd|B#4uQWcY#l!c%Fb^t%nR4D4lHN^RSbS!{AXK@Gmq#e0*hD(}`@bxQKv;3`Y z0pmR+Eyy}7$Uo@E+qE;N9%VgQuC5N*c5tEWsJ{7L@rWaz_}kN!1j_#C;dk{P3`N_Ug9c8vD4@t{cCmp@41)W1w@)8B z2I>66swQnBO5J#QEsm1c-ixbw3;7CDFY;3dBO|Y*Ldx9~=2Nbhl#Q$1%#KWP24UBZ zNo)z6*&g0I3hlSYIf>r1I6IXrmZgBs3x#riUJaVN9myM;Qt3F0nl{yM?WcEX)UqvNijY5gWA}zW-dNiE}&am-Ucy~dk1y|Wq+1E_YmVOU* zHrfAN7Y5I@g@*Ayts685SUIXwRrU!}Tu5^Ot;L5&Px|-@wL;apJ+Rh8 zy+HkO?0%q(1{dhFTuaZ4BuT z4q0qh<2`>?q-m)um@U#4Rcf$IiG+r1ABa2$*!|A?X~9&}TF8a(&HGN zD~`6P-S@T1P_}kS^v@Fz2^o-?KyrVQ)x!9jw80IhX@b@$nhzzzDur~-tR8E);!OKF zpj|Y!fdQpLnh!z6gW6=BC9^4n?=hQOIzq0vr%D+Kd0%ztOG~+>gBPD^ffmDrQTF2x z`6oI+$2cU74e8)|j*^7)kVAE0=;P5kmM0m?7uoVY)+gkeDvv;g*g$V->h{=l0+Orp z=V8Y;d(EzWu%4*-LVw#Ppvvir^J(Von8aj^<45O9v6m|i!n}7!uR}@!)LoLp#gB59 z6+vG%|7z~8eB4IUT_M29T$Q}8A^K29@^$5uDqF8g1F288|fQ&fPyRG=Z0 zPwnl@wm%~Tvht_a=cAwV3m+WwP(Ud95KiJ6)dVnmi}MNKeEKbfqfc6yQaEspb+>Ck zZlm(T{H6oJyWlRMs}Z)?uD4L-b5u0zG$34i7@Sia@<8FS-~LG{U7mk?r8L`xIMRaV zeZ$L#pj25EW}{$uxj4p#D=t-e7nW4_6w@a_ZyZt@kag5-7Hn!3^_eQ)sd>?psUSsG z`b$F+oWK4QK9YGOhd7ydtY0#k(?KEAZoYJQhbFh|`_|{wZv-c?JJjQW2Eru1JX0uV zj7%OE!mStKD(kX-kGO?6TITbVnYA(K&OMt6eDY&^S-Q2SJy0IX_>54u(PkRP2MJ&O zW6PAesC_~mTg>S&eoG$8iP1eLP!d#bctL$~@9^|M(q{QL`!cx@9Sbws0rRacpZEfy ziqPMP!Ib0vg^YcfIbvCoYFR$(%R_??s~t_ry(Qm{_k~<(Y74+cTI=i`;o0GAqh4z1 z9qYvNS|Ga^0)idz?|s+{Y8j?+YwqLf7W+T17&Y$BCH1PIw?^;Bzl(El`)*D-Ktjf; zZ9x?+;tQ}MZsW(neW%hj8l4$SG4$5Ncou*m8_k2!xs0GZj z1MVqqm@4flDpS^G8)X&%5Wh7|cx|ooKo@5_Z&*DE<$y!x03naJyeN$`3Uqdncvfzm9mk0D- z;U@=t4kQ;^(B9^X3-$I?)-B}c`mG=U*%j|#OwO%bVoX#4I+#B;K}i3IK)-JX!&S{&&#Jf{Pk8lxL8pip-i!4+vY5zPbk~_p|G&-pA-0S94mbg(g|-+vN+UkmMdWA zudIitU$0*%Ok#u1Ge$66ndqL|V-^r;q--CK5?Aa>jAark;zo5%8Nw5G+N5TBb+5Bc zexh#5LCfDNsN3Bggz`>6(Bw&}Z{&o7=6|=BPW|bv&%M>#Sw~(!4sT-hZ(C;_sa1BO zoc|6!Ozs%3jP3hEeUpK*%o#LIhqOgexxYc3HF@`b`zz(Un)xmFc|KS-9ZpL=t8WeJ zIe5|=_eKQH#{Vf~oP{(ES{qpgI^1GZk*+r@eQN+$eE;k;7uHCY%i(QoznOIQv3}Nk zo#hJE-{=$Z8wTL}G1BQF>4sr~gT?sx$k(QabokS@>(4F;ra{@H!mp-%@RQ zxSC6rR9%~B;=8j_$}_4Ep*kQ0`x8h27Hb$WwH{cN9*xzMjsu)}7}8aATai z;gkyUBqsXJ^DaPir@L|jnGx}`GmbvNh4=XUElIm;pRB3pf;=7QDEXH4e$@y6_guGo z>~$!%)p@Bb0`h)D79|-#Y(w0s`((Hmpj_Jp-(|xW2%GH&Yv97PVlJcV<2v(o1cQRu zm|}4koZdazU!aW6HqERs##8Hr)`G@o=$r?-f}gAVUl*+=p65324fSbJ|LuK?Ye-$(!)0skmhS?M@> z5blbe3@GX2#B}__mR@dB_|UIZ!WIIhDifor!F+#+o$qUTNs1RBiRom{VY)g(G9Q%8 zZLzoD^%#0|neR}!UMHMeNY9sr4x%yQ?7YHZS~ciSt?(F*TRonpVe|JHgQT0tL z37&nkMcf#@`VOCg~m#z7X@3saUm561leEiz9t(qABlpEojK*Z3u19Du|;`!)w zlU(po;_}WEVNF#^X#^GKfP$}Y#o(DHzr}>8Y=knArAH)tQO1~Ss!w{qKA9HZIpgLk zf^`&v5koWWVf;mNq^_A@+-jt#sMHZu#^U*I2rI(6BqxKg;pk*9P%;9px|e%*s#6wL zwq&TU_TDSXrB#$vwJ!T>juJzDROIfdvp03JgJ%cXXJ3V9B^Mt&FskW=k?k+$P&sBW zoY}#9bzk^Iq&KtGONqOCq%G^c>0AjT+IWDrMif4M&14(L;cCKq*sY_EmrCt(Z_LE~ zDqIc~IO78#ItKF*1*ss;5h-@t*K>UU`acVbHq9)z2twHGCh_--&xR1zHvT0to$}Hx zquvOp%cl5xI9NY3-VVmQ_PI4q*~ts{B8)#td7TA#+q1|B{f4nlMA+(6ev6WIh?N2~ zIW`B=`tsF-VblfvJ&w=8%Ilk2Ak_`OJB*6Sy}M_LR74;nyqQ;HDEE}*x}%lC0DvT- z#O_UMroV|%FtK``NZVzXliD}QL|oZ{AGCC@Jp)H;l9N|<+-1P3D)3@eQn<*qAya}- z+57e7LL)Vg&N%HgP?Z<98Tia5SD?J3j_?F;0C%t@XG9^jU%eA9y;oC;Zk_$MyayAA z+U^UMCIL)5NWm|Jv^l^N|QXGfIht@ue&VfUzsij{{)tf7g5I_uItkXJa5 z`3N3Oo!L=YyR5pNYcF;Pn;bkod-v6rn6yGvXAId z5LH9NlYaTUci5C(E{UD?dPG>lAxv6oLk>F|q+Y9=9;O>>+A`O>vVG(J5}bU!q|=E% zy+1uZjK_fVHf}94F~o1K9;;w=UQi6zO@3Td*wVS<>~@u_=Y{E?X#kFQV2S9jY-SbK zJjY!5^7jo|nIP*Bg+0_~mbFRv}-*A zL3(r=%`7 z)%aoS?hEF|FIZvSAD2heMoOmS*|3t^dXl;4iTVm%gEJ8|{(slA5b;F#T1#XKI}Q;# z`KbPi`Md5l7My5n>y+^JL=d`o))$8B!vu9YV=6S)Qqn zGCitK+F3HR``$fUD-XvJO^)U}4}FS5Ry%_`C4*MB#6Kjy*s1VPn^a=j#$of>Fnd{W zz9^*M$90km+uyj1FEIt+C#bE#s3X_tAB_Js^3*NZv)1mu`2Jh)$MeY=3`UqACPj}p z8-4qpY5=K2MF==Ub)~`^p{pCor41s2+B&jA8CO{MOUHNzF_3M3JW?{N3It3b~oFtugk zs@l&4SkbfWVG*ukXoB z^i4jV-k$H(9RbhH{jJ>%ek8SaRb)5oce^HA*><#Qhmh zE>l@Tw)PZ9Q48jW#X(_J1Z@jLNqLs`_&gfbfvEEgnwYlL=3-Q(2<%(~^Ny;%B!bgF${n4DKIn`o!=wZaR8#|yl zE_GB>`#qb>-dE4(UVYhiyIQpd$Ad7clB6@6G7wtW1Izm> zsrON$f?LlJ4~F{VMoL2;3&28q)BhT?n@L7k99^?Iy@~NvC`zlVPe$l)kK$%1+fOMH z0hX)9k=b0tVQo13KO173HV|&+Iso06XRk0~vxM`CT+-Z8wyV<%4SR~&5RLd*^~7D>q!e|60%i*N}?ozIE&HvRYrE#2zzGu|H* zCrXQf*bNu;=S>_TTaU*~-eawg!v~E{cj8$!YR1re3dU#nb%n}`cs{-TXM!ZmuU~;U zYn9dV=rCOb*=N1X>o#*KW+ti{=OK<$SXuUdnnxwZw8eET?BjdL1GhmZu^fvF@~%#> zr-g~gQ-U%^!``f9v0lWS_x|;Jt;#~`NeDe?jv9d&ue%moz?G%Q+xi=|lYsN$C_z>4 zZpk$>cr@WUYzX+_nTeeO4TKKalD4;-Y$Dmx9{W(|rA1xn9wW-XR7jqucyWTC%WwJKTv36X@A8S7G$F3e39niN?~(_-ONw2aBW#ghyY~t>aj{>n<4G5b@wcK zX6cyR2dgdUZy^^Q6u8;Q%~Q^-OBUxR@5{WtwrcV$srzZQ#B0d+8=Ht<2UctxA~736 z+h1;c%ComXumiZB(hrR@sX=tdJN!{RD1VTTkOsBGjwsd~=P${4 z-IHZfBy@mQdWgP@KIo_5FJy~U)d3NR` zg}T*_cZdomSnNqXICCn++KAiW!i+0nr{dJRe_l@YD+6F9)fb%#`aEBU+%BgN{9M$P zFWSWTP~*ezw>Y9Kt%jWfVW)`GbMzLAVN`5-jdC(9{&TExMcyYbmb`Jr3UHWZm`$aw zeg0jap}X<8(h0brjG0Eh>F9b|S4g!>gO}L$u$&3p#;{A=N)3;8GZ0Lqi;rQj#Y_@@ z7)CX$Z;8m^bz)&K3scqn8X+2UR2gQZW~`5Z=e~@5Dyr6({|stg7lTz$Wg%TCJ9R0c zPC3nFDhGwO<4d00psE$3N99}v;~v|1(Ylj-tW=FrXECAHs|1aGb!FkOn9lI7gvlj8z*EMm3K9N`^%L8SaOQ2zZuQ;M9gTAcXht{^cJ)i!Y zc7X7(Q6AIGSgNN z-da;P>SVO%OF2%7h|oPp_D)(>CB`yCMC5`njaa|V)`QiVt-_74h7XmY(6DK@2K=RM zjZ9Ypg?CcnLID&{8HE!Dq92>K_O0iRf`d>&W$4H87bAZ*Nr>6qhN~V$p_z1C|K_M~ zsL6-N|B{&<+VrkM%-x?rP`sUF zc!eQj6;lxov`^EYJ^kcdD zsR%p5hGc@4n+fcT$cJT#YrS-j->`c(nI3EZ;Qb>wHdscHg|*x~rH zr#j*eatgDQj(dATxGzjZ*D9|hm|f^|MT0p@kFe5gmj#qW;?tY|TwNM4Y(nBn=xYup zwssudANJ62N=8W1TwsoHVOw{0P~iV0hl<;AJJ=|Eny&n61R8#iJCxnFioWE2i&Lq8 zvkBBSpP}v#6u|`;>qkYp??NauRU}si`+aj(@`rJ95>{6~X)M~B5*hs5EwS4IYiEHy zSKVzf+MWK_bL2?z__dgICNu7}JXJY7O0OUpp(h$+T<$G6rQLPizT#v#737mtHew)ufM=KcM`Y6iD_;;(dMZKfvDAj2WzC)h(0clEUiFmjE|s%>~z72 zjh&MPDs1?gvn(xTZ^{BzeQ~n!9}emfTfy&pd7Jk?(Mn{R0azPyd9?jF-MuSKzuRYS zLGF~(i67WS=u>^@pj6eBf6U-n?_}ap6 zg6e65z>V}i@M!%V@W7;kN)FV&Z$6?F54N~YR8YMp&2J9g@&KJWFkwi%6jLE5Nv|+e z+y;ma=z@1pN1w~u2kQ772BuqwL(bQC&&1$>P;ZED3K+VNueKmaLp z?REvP?|sFvnO54i-F?!x_Y31kn>qLD0{b>kS1uh5 zRIX#W5B@JF{gXGq0Oht1EK7xa+bx+8n`T16n*z!?o6^Fyu|$2~dVKN~v>FRX?GarD zzcI?W-x3GzxkX1%NZnw6nej-P@|)lUdI`TloXiGlfOAifW>t=t07>(&f6JKp;^anM zpOZ!&3PQ5u+y;$ZcFWRHHy17}-GV@kU#lb3l zyAjXwglg$=w*JR)_s?RVR|fOZ-wO2Q{~1HCsSLqCLtXEGyhMOB_#bNzka_?21L_m{ z?<1i;h5s%F>Qngt<6__;Y*oNR|2aoO6tP^>zoS9y1YJ%QySZj|JEks9iZ#5M^)9_20GH zX&ikohu-V?W4{9!IQ&DV_bpHYAOqA7xFYNk927*=QiIXo3W~&po`Fn&_NhDz)r0yQ zlm8wN`wR6!L9U~0Itb6)pJGaR+3>@?&E=}T!W_!7)4$)GW`3P7^hwZ0#!RLjD-A>u zgsKPp5f8Ck6p`iW^~(SlQ^EJgvT?+-*1ApHOAKr&2ZEe>Q zcEaBDsDu`xha-k-i7IVVdnCO3K%>eD&|3(kd83{=K1h_q=`)9!Kdk2!F$V=HqOg8q zzCtv_C4T+I4;Agp%-^pQ*q^g7e+ao_a-8|v2amqTe2psjuhB$k=wV2;j>Fz;ZqG&I zgK#=gZud>_(yU_7d|}HlL_WJhj?~|Dppkh+*`XHQNW;=jKw1sP9{Jok43T#=R%iIr zN8=*<^1}Cejzn?P(wZ}`Y(=aYJiZL6z6YkHbuj!}u9kO;|N4_7QS%1Yc><8ux_Q&% zklRC$<6hZpa3JhnSzOeddV9RcPqe^h?=)1{-0>d2Ga`XK*7nSAo<5!pWX3}Hq7~8Us3Qm94 z+?l8H3ery>ZNXijd<|$O;*~2@3ef#mPfCT;q=|C->%&JNPG`-*3JLU{uyU+FcSeRQ zXm1!F9;W|l;xHyG$VAB2&}02oc(QWs_g<)#bVDKJDt-O)fOu}AB+5>$E|CyMp6Aw} zbU(10+uk|3JD}~l>L1XI045w@@hpWH4;?-%e(7dCXqi6wuaLv$YDiA3cOUh7saYuqCPoYHYU3sO9u5HwCr1e4J~f z2H18PtZXO=0}K3k4*|K38ssi~{RVnND5tkvE{y&&I!aFCInLHyl#dXM@x6^Kpr8c9 zTi^w7p*yOCvU@NQ#f~m2#&Vbx1c796a(@CXi2l|Cd%6DsDkJVTR4bRyzuwvGNzKsS za9>gk&nB)@jUs0r^uWjR-nbedNgFF;RhU%>L|zDNqoTm~e8h&Lp9@_|zFGCBKaCQn z^2*qLx>-aIj+erkn-NGcNK_(or>x(Rd_rH{HVwDX!qA`3>4;7#9kad(V# zIaRH(y4ZDb%g8G_b_nKrzRGaQ&lFAyz}HMSMDBf%iXkI5l7$dWb-JGg-vrY0ahlhH zq|>X>o6!$#(rBZZy$H=)?G^=ZCnC495Bw3;Q`I&3Q(IM+&y<<>sG|wWQ*V3P8n$x> z&E`~4(BAODdqGf|L8rRAk70VqMB%kDdLBi>8D19c>vhm%IQmv2!`pFgUFWt!0WvuM zCi0sp%cEY(782{-hAyNCVH{y9C>gMVOcxwN36lu^6n4(quqcpxHFZk?DJHm|IyH5< z?t%&-|DhB?OF9d?no||OGwj9?q|5EE>@*gw7PDz})vL_$wrqdwJy%T)3SMH(Qn6Hf zQ;Cj5XnL!Y8jU_|gxFI$hL6u(WbvJts{e8JMtde@XA>%7cPePE<^kKMk8>&(6vyVh zi}{pPzx}pCts#|@?9n)#!ESi3bARxiAETVlse5BuzFiF>4YMCFhI%FAHruoBt3&Ck z&k`!6R(h6ah?)TtpCu3U2~zmTgSpdIoZ+wmRaQt*vb5V#$hr_>3Lo0!JS!L)h7h3Oy}JGAa@B?ZAM3D zQ@Y9Tf*lruE1xzU;ooZ5mLyKT>kHBNA|PkDoqv|MUF%_2QVj}b^-1|(v-vIM?LkD_ zo~g!PI7vJ4l&IATjH9xWJj@XHME~RW*r-%%gP&oq*6a=EXYHdKkUYF-)>yBv;vp}C z?^x`3_R_C`ZCs*wu17$$td8bc`;vC|aZ|w{SzTl-vQBYVcv$rk#R7BrkB%HsK>|F0 zb`nzk^??Qp80*f=+(jB~x`yg97vMjSBo40FCUh<}F2GC%r( z{q-ewd)%QpR!2<4UMOAr-u$aE^g}yIwGOsVDh?lp5c1g&bcK#GIs1x)tV=vH%HR5w zsy|?%>qi>gx@?pGHF7qg2H7qPcks1xbYXwwySH%y~Xt6+9Nq6 z-j?97;yue9)v^=R+Wgk~0y?t(A{3>a7KNcXhOJcX4dlE=v_x+|d6vWm={qXfB7$qMIo`Lw0tX2XG49Di0?yM5_ge?M~KDR-Tv_%M#6cmARUR zEC!+9CO(k!p{Z~?V|tT)+1cKWG^fqI<#S&v-zXhWcPd5TPs_KEz; z84|yn#7VV-#!b%qK_5X;uOl#b8=H9cYFHq{FF}0xBfajF$I(wE3QF2;6CbZQe{C54 zP1nkJz_d!F-|Q79%0Iz|{bDA(B~|b>tSmf)A$$IzSOkZKC+>jp^L6p{~ z=4y&VRNKV4r^?>5dV|&pPYCOd7%8@%s1kU(^%Lrgwbf5pP*JB+a0Ro8Njqa*-yb{f zMjkvI;yqNEehN2?2Hqb;bdKvI7mW+;EkiSq%HDrMQ9Gv(L&DcV;9cb2S&Vocp*ep6 zp=-6_7xD=)e{A#p!JEAo#F*HijUb$4Z9GZqeUNOy&Z_ZQS$kr!FZ(^5&UbK1H`o|K zYH4sCaL7$%l2M2!-5_g!G zVAbKbu_nN!V|7WIYwJ_WR0}JQ40#QU3N$L*bG5NV3l^SGcOz(_;{g3lX5;(QfefPa z_#xkc^;Q`X2Xl02i`?q!v(4oljotBHDb=Zj&o0cJc{DudU~^yKC z!}m=$9Ez}6V#8-wU~(3(8m#a^pF@bbLv;Sbx$>`1P6bKCXJTlqcA;7B6Y zASlDcTjR$9N3IjDE~jCGI+Yu#PE+A}iixT3cGtny{idhuRSw0KUsn{bHP_aFL8}b* zzn?@c=TZEywwFaAMp{#J;)Gbt!e+?j#`H8wc_i<5{I)pnNo2Bf%T}S|15JA3+oMyN zod=r@aM|trP63VW{4eNCc~utws{Q=(WbWMTULR&MW<_-ZJ>YpukHa2s4o zv+FFrFYwiDE_ke6vF%z<2!+8e`!YbOK@F9m7;Q4QZFnk z_Cvs@(1DaD{Wc)k_5$4eoJ9p>Y zo|@x+FZTEwu2M0`lIMx9Xg*t=&Xcszus)xAS*+pNse(AK6m-iUuUzN6`-6qWP;J#- zvK_`lJ8zmF6LS@;RPd(XLV{0UAf3RquQ61-C*8vfAQ0KXQaJo9B#L7lsTES^Tod+2 zLnMG8qir7D#M)nbVb5`(e0p~z^0MjY+~h^~SJjJ|bfXIPZz~>M*%ZHy&Z}GC7?t}O zL^ZOtvPTDWg>gLNQ0eG*xObM`Vi~de7NZ1mr*q8QNyp@P^$s)Zs`CKHGPwS$rLq6h z#OVK2K7RXC{gy*Nq&j4Gf(=rt|10~nu*3YfiKgG)(TFM*r;YK48_eA1pZs-vKZVv) zvfS{F#c8$~WZ1{|mkvR0#{*l+dJf`jKRVhNN`zFeErH|-B;8k-+o`Pa+d@~4T3~oJ zxJgvh(g-BGxy(DtkNRv^8z@I{`%{ooyNPTN%fJ6IjXy$v9u?1JW;Z?YTA@`e(;wa( z-L>*P(1Vk&@>Cy&SUv@}Tbr^c6b5aUXglpL8oCQWF8xa6tzZ-5_6Lu*u415D0zGqq zwjlSmR$;;yAzlNeMlq;;Ie>A5zrFR}K{~GcLL;_kV^D963w;1QLIAeIy#7siB*cq8 z*As(s@6CJgubbB`*`E^%G)N#xFZ{P4m8eO{_|fK_g$e-OtY?Ekf;Y5RM{B|GAU_oU zf1xu`li1z0uA`2|`Km+z_V+4WAQFixj&W$+Eo+N`e zsu}SW(UBzNRS>rJWt)L#;P9ly@y1%4BpL~GvOswWx7^}3)e_!gTTXET)88|j={R|P zvhTBiK#JD|B7nLojDg8l-)6_udygtaMuK(g$yRjdfn~}!3Z0t@- zxFJzPuYnV9RS>g>#pSankyyOHaz^5oc*{(D40Tui2Q9J#$Nf5@3#TFv_G0`pNfoG; zvC29wRvsufSgh8u-@zo?M`0zEXcs*$~FpWnZ_O&Al1{Ca_B;sMEWr8MBsf zOma=yDyltLE)Z`@w`f44_p7xcATY9_d#_j$j&ByxIty z!s^!!YBy;~YD?DD&?QV{#O}}4?ZSK&_}utJ;8gQ;a3_q?n@_n@px)?l8%nt9Ej^0> zE0CXivp|kOx>^8MdvpBj>s!3ug{v6Mm&QT=78SSfrf$kRzuA%PV4}7EBosanQIMeA zk;O5`tFat&QZvZMEbgSplHkERzXY{qQ-an)Qyld4C$p^zRPN`I1(z9hMWpQOdO~A; zhy0h8RC_RO(urWRO&=OM@2e*5b@Tfm;U zIpdRs)as|*ZzT!ZJnUwDOim(l%Ni>vZwCuG)OQpT-3Ko(^x+KWVB%`@Wt-n*h}C`Q97LPHz^F=V=?Rg{Ud0r4Xn8Nr@IjO^)r=w|LtH~dPH|sgt+Mk&(QRh7(%jdF038MM+KFQ=5gWtb{ zppBu6xV1-78ekxy()^*gjX#sJVG5M&xDX$OgV!_GG&o_O-)rye5c%_7h-}wS66}_O z)&}2#*9FF0;}~PMA-iz1(G9=pWjcp#$yl@A%R#%22luY$ASqIG8~mLObV{Qv(K-7- z)>w0U#z{6_-Ej_GKoTVPy}AKB0Phb$@ZWuFuP5e@>P3+i&t5hkx|Y|U<|V`2eYCwl%cNo`I^DYDFV=9cWHj=aOgcKi)w>rB4ai+4 zabsu(#7&hMzJRXS9m>bvSIDLwzXi!DXWE%d*{QfCtk=qz>zy%+uNz(_$#@NSeq;U{ z;YR7h{JKz@U3)57`J!arU6qOBW&KsjHiW>voq7U%kbsStjF`msHz8lh;sv&C32MxB zD`r|y3HCjd&0bRS?(a5fg_sJ>F2sfof_7~7W86t)_<09K-j_1v-Ics0xg5(m>EhQ7`Q(TEybbTAVC2q^hS`C6}ieTQQ`O3Iz8(p96d@2{pD zL~wFW553Th zqVmL}f#SZ}@CX{*E3Tl#LZ>P`$#n|Z2e<#rVGzctFlnoC@H%aiW41ws$kj0C73wfH z27gI?!)fkSVx+IJuR53%`xdQwZdr9X0iAG>o*C8TB@4V!{q+93ZO>$7mnu}^n%i}& ziuk6oejh?gaA(zGs$G4V9UcOyRtJZD)F1CYma{t2Rw{OzgrqZoTj#2mO1kLm8PbkK zNQ{fQ7_)gyC}LQNDwzpazxkC3R)rqQtZrQDvQJVpVCScr1oZFSvzM3(Xj0{?63c>Z z8Oj>qC%H=9afo;JgYYYzb;uYb9VueSw%q_1aeukW$&o`3n)BXTn&6m|?(inS96HRo z<;13sLYyvxW4rZCT3-#T3(eqj$X8Tsn`;Vk^N{;j&b-q}XkJ?9YOQZlCPdDnl=fuW z@`jDH?0KkUsA9Y7N>5h*v2$b?=Gb3mEOv1vqweGPLZA#c*#*QAG$QJnXG%R1hLuEk z1;b94XQ=$)%T;%yKSAZO`gEZ@7ZO|xeu zz6GmAErq;(W#|yM>6Jn(+jHw~7#4y+65KDVfyLz?mO^u0CvjlWLysMW?hi9=b5#eF zCBx^svuhZlX_GDCjOqFZ1Ep*(hJ4l{wX8V-vk%!tsI9e9KMbdB3KY3tZ(P3c?U9Bt zKQ%+f?g*nq?B;ebbi&^1?Oeo!rHg$Klwb&@+2c#aBz)rvXu5mPoAmoM#sarR+iGS)!G@y1Si-A{gGT?fS zKHQ^cViotq^)gabu<8aHd-Zw}bcb8~hn)|%t5E-?7d=^<%TrqKhji4s$7@_VoTNwP zhVlbGTNk_*a>E>6Q;>7T=NTsz!VH$@HtFb?-Z01V%vxBxS#%HAi;P0oJ6&)D8NYGunP=TnGcm>`m^It?o_Fa{YMcVy#7S6m(Vm>D_07g$ zb(&ebf+zd6u+5GIwdj2AWyFy1teMb zlOW1kWKAbbwsm3LM^@dg8tMarUz=*9CBhTe_8{Z`y(5c zxm{MMi1I~}P5*4}RVm!9^zvbWSYwU!RZk}cY_PVxAZN%2{^4wAlU2DG z_(~h9g125!spr)fKs(|HFr7|tw#Wr7q1BUzt{ps}1)lFzk)>>xhW7w`_60MCI3d^U zQ!8PcB02qF;~N`*Jg)=D6Qa~K@YYzaVQT2oAqZsWHi(E@pAwgh1xfWVFF^HTnwnX_ zt+G|(V;V3E5RbO%g#0V?9|r6=xc<$==zqu?ua8YOhvpjPDqW4^6!HYg1&Zl>*2h4+ zuK6=$VXzE@>N^IYW8nHXbN3=5I3|?-*KY>-z|WA)#nFF#^K}$#vMG4Q0(9cvP2n7H zAgn&>CVhU>i^_$&8yvt1nCD1kmy|bXXH+=nOHy{$4QNK6F3b` zR05etz6T@rZybW=xzs>TYTE)QTM9FW{P86A?+1!0twHwIVPJ~fz>yP!I1{?L$iK(| zfe8Fc8XfdTLP3@?x7H;m68ou)XKBGO@5ME(E@JH8qdZ{cRhgM&^VfgkSP5yLxWCl` zm(Fm39tixCgcaQyd;IVNxj*I_i+`~6V}Dx z2j|1&@n3>=TAv4fAI;1qXzs?$EmD1zAof{+iG9>2T}9>c41sdG4`0kV@JAWa)vy{_iHp%qGT}(U(&XbSEi3hlFG# z&gDeb14f3|;Q^2w+y{Ah1(Of0J`Nz&VA&B|&{y&=>ij#w!OJ?77&#ptFC5BL6HtdR z%R3|E0ch8iJs1(@3xL>{bB0qprpzX^nN8gMwJKZqpR2uvVt+9aN2SVURuqhVRjY1! z)lH*Q;G~$S?9%v#Hq&$}{`_^1vIBgyudw9#RT~|{{_++&V zXoZJ=brJY>_7KDgVcpFMaeAhSp~qZ|j9~Ko0&M^cR`Ph<*HWVZ8W7+pk2}p3$<$te zQKR31MYN6vK=AU2DOYvG><^9;(z?;#rJXW|FAv!x=fLJHnQVd+JCo&T_o+=8 zGp@$=-%y=e2ZpR#->-`E4nbPF>!O_ z6pdKwfj`23mtp%^aCXdQDQFCfM1O`;^56oci^R?J*8~DS#SFRe9pxBQ0vEu|j?@tvt&_i1W`wRjkGVChGc(XR#49K&jeoZZFV?GRIh{E)$K^ z=W|nG-#Z)omWp3>L;_sxsEeI}Nfw=YjQD($;J*}Lk|$61D#RicgQ_|*gquE=BRhmB z_0---BaF_6hJJNcdJnFT!?0gfW#-*&5u zuF=1uo530u<$hWOGOR&#mR;G8N-0U4&aA%Z$XN`ZLwzu?E0@Dvg5uitANqzK4h_rn zFio}Qyx)EAQKj^9bb>OuVCGtx<=)hO4_?q*Ud|%80QsVU^Ryy`-KC=^cvPtaDnp)= zBsao|KiK%<^yM|bpfkKQ=!y$QPwN%7K1ST)xcoR2jh(IP+hob2QAYI&F~*RICSd@|z5bM~#LiL^;$ z2lS8CTfq9=qpWS!20IMiWtC5td=9(cv_XHPispO<_kqF-`~ZValf#zVSm_kFyi^u_ zKVgP?Y#$G0f%k@5ccrKkL|G2}J0>h@)E5)eP4Os@kTJ=DR@SHJe2K?4A#KBL)1r@U z><$seJ8P&W6;jo=Ju-PXgZJwUBxvkokKw|Gj3|9g|3b=cUrWc)sYt4$9P+2MmfFF% zZ+GO>Rjv6OjAi_uB}bfj(4{;WZtr3yArWxCE6Uu(0Ra!k48p#*22v=hgjCDlXi@UX zuE$f4JZ67)H_={;;4ZS?Rt!@NMCbkDI@{>PAriZd6}CsavEA&C+=9R7Me>L(1t0CL zQrCNYsMuj2z{S$x;~T0scYaR}6;fCdTxJsV5#!k??2ZPxH{kA}?_snZjS7NPM;WE4 z;sQSiLYPO+CHYF)&6M)~SQA=xopcL2O|m^UeZDbw5(kp~A;d#o{*Uf0tLvyk^%$Pt z>WLTk#J3cm%)*8YEt>QD_ zn*c1Hm0i4+TD^Rhv_=SFi>RShI`OJsNZZV#EzH1t;aJfi$Pm2C!lXvld@ zbr(fwzVg%d1E~qIfzBYl0ul*GMAp>yS|Cpol0R@PWssiFNLJq)hPxf7DLZ2)U(AP8 zR-FC6NA;y+B+f!jR!ri~$Ot!ArppjPYjK2qVocll5T4Zs3gaq0PtnvrMI72M65;`hIDe>j=C zy|?QqEP|V;79Jp{O-`ci4+4gQ%M(cpMb{`62L%&BJ;S;k!wBG`oC}gm9R;b`9 z9SFD2PhMRI#TWP1M(#!`muYJw8@czWn1x0WQzxRw?QbLmnvT>}wv|Jn1toR2In7?Y~0u4S}qPrs;8 z#s|cMUVfrn8XEfUVy1#IrHQYFL+A5!kFh}bjKC2n-=0X6M&{hsC0Bb+j)}&HRy}=i zBpl89`~kmw;Dy9RX3*3?L4Ju5yyAjMvMHs`@z!0O2JGp*>ijcBK?ujr9s zvw(2(1E61&Ejw5>(ErPha5tKUWc%K}A>q^20s@ucLw;r6M%QzvH9E>-$YwU`20X!r zBQ|kEjfJ}3+R^cPBYWHUu!`iRZ3JKLQF2*DA|2~q*~N|`+FE55HZr!R_P?^B6SFdq zo2L)H$50|W7L;Vlp8W!|S;NtOkA2JadE>Hpha-xm_yoA`z3snx-fh!RK4f!4 z*|PHe22&(a+@X1p*`daJlDdE%&+zk&pnSrsE^Vtw(=(zJrZV^Y0>r5Ao|8Y@r*$rw zmFy1FIeoQFZ4uz;k?r_HMyyw^I?(OZpKf7gw;h5sd6jG5`P}U=ML=o&P zO&n~~^Y?kesS7=!%i<{8Gq!Hyyk~R7?5h|{cgAuqzPdM|RiDAb6X5DYY#`*@H7z%s zYl5;fM&k;7S`I!%HcnHZG7<_M0q64j>MBZZ5)b(g*qZxPov6656mroO{ifumLyZ4k z`c$0Kd9ZFXA>3yQulW@RFV=L-Ejz+{uJ-gOPc?bq?!LWqLWgvr4df*bI1ijlN77X$ ze!IIjwKp*ZLoAJtN&_}V?Fd0&Ya-CG)_1DKO=bvQ4fq}HzfK(G6y$5F`MvRyKMdk+ z4#bStBqBhl-@cUjuw2OL)0Ht}(bh`dAj9hH6{Vr{vVz-ucD9OjGH>52G(Vws_Tztk zq#jM~sB%1*+(n?*lVvRWO0bO;6qpRf^ynuWQ916;;DT{hvS1cF30Cry)flHX;HD)MkE^|I06c>_5A$;pe;r?8LzA4mQ zD)YBQ@P5gr*PWFDdPSksuM?A4I`*-n zxm4;8R4_4s2P7y=iYP}|zhMmd(+D!o=@hTJBE$oEtJ5gsyzoEbq!15&}FWLkxIOa#-^zx=TV;2pS3Y;?7dXhZe)_XZ6noq=V`NS zvX!KAqK4*!u!)uM|g?Rbgae3r79YPTa6dsKN z&y~$wHkyhY?GkooSJAyzkc&6_pxc+YJr|-!O)RnWGqyv1b$*O+kOHABW`4X8S#n(o z8QTi-_#AxU!bHr`lf6>}tq6>-jO6odr)thC^LK}8JOo7z`E4_cb7y-2w~EOv#k+p*obgRZ!MEYbaQ z#~S0BVDqs0-J{PHLSz3Gx;x5ipIkB zt^BQOuj0?>y0pCo>ZN72*+zJCU2+P&+iQxOFL^8?4;k(dNqkI_-peJV!OjftMF^*E_k7^jANu? zA7aY6YQfqJy0KoCJeul0ijumSz)Iye+^Y>i!V^OF{TEr$AEWo{Sy8l=;C{au6>4L7 zAzo@kBydv2A;Mv+skffI*@aFcx}@b9w_A!_Om-Lc%q}ePp2{FLdre6Y|HKuGWXWtV z-(!z6v>wk?I@7w7BDqAMHQhUP>TTvDtK2N{0;mHH@Y|YHR}*{8A0BZZY1PPym&MI1 z&=y}`BB$Bu_vnMK_T@2Dwgv2=Pqt;Pl@(~NN0e24&CbGUWGrI)XhgY+VgV}xMysyj zlpgqAEb+=EG@-GOjPA%TY7%-k{8*>mY}1Qi6fr4m2;fy0x7_aF?VoVFxyUrZitwdT zEte|C*CB$9js#c!NJe5@x3R|4ot)1t!N|=B6(6BH?s!#E0(_WgCS2iITc4mt$W~Td z*zOD)M%@jE-A8r>SkK`N%NG?hbqnO^wggy`-;cgmN3x7)^`}4FbCP*;qP=i;KWXl( zv7^=xF{q*MceJ-5IJYZ8h#?oa`p_|BvEC1)Qc$$WMK@gReK9>l-3PWDgD~-hRyoED z)kt~biU>BnsA^oU4L&bI66^RP<6;D{5lsMSZHiw#L7XwfF`Eu}vEB`>pQNFF+Nmp) z{8QED4(>i?-)ukhnqSI1)hv|3Nrl<;UUoDLjmBbC-v4lfwoUkmne7}2EKANLzV6m{ zcpdva8-oaa8M5nooVF^SAank~yj8x-OpYfWQ4;Vczax$qP$1`PdtW^miTII2>4azO zy*I)sC=;|QtsdeIRSSE~sB}@yt;EG+eK5K2wqO+>7wiRQ13Jeu`ldeoCC})bTIJ#b zMgXaJRT15w=lfmyJ;43@H{KUt)3(4JYltnOK^g$Rb|{m!FnstQj`QSxO`Zt~*Cll1_Q? zeos*4&cxZ*%`fb#phD0jV#EYo3k98fBlM!azxQqcMrS^$=P!VYy{5JmlC=K#HZ(oLY8$X1w;EvdM>HCF2QuyW%HW z4LYMk$|ROvB2MG!v&-HiY-}tLOItaq=dYoE*z43Irbuh1+lZ)9z$fKlb>5pl&2Xw8 zf65jtRpJmUd)~vR|W_An2l$hfx=C#3X^kC zn?Z`hJ7W^QM;@WEpK~Q?n}XSI$2&C}U30M;U1r{6b!9BWZ+G{CNQ8YTJ;`_ve>N(8 zjfq}r`c30{hB9H<9Eoyg)iab%A5h_P?0>O9WKDcyzN=OmtPFADI|-z=d+e#C^Uk>M z*SJbK?Yh0tu8U=ms?fQNX;sS0nhZ;J(=Aq6XZS>Bqo8{lm%{L?g*S{(4JIwrAo7@h zotEr`<<3Tid^7Y9q9x(0!~m8P+lz~k=AKEd6XkdP#fTcl$4dth3u*;xH88r&1S=eumT9>R|HvUiNdBW8b&Dt8!0Wl9cg z>O8D?CC!uhLN>jb-$m%B+wBKzFII;&!VKBGg)-a<{o>~XXt|B=Nc@9iEZ^Vhj$!NP z%SWna(MIoV=iUI`j#FQ8jU?n{)7p7A-Ozv|XM zHx2uqnTgm@5SMl+@6S1GEix8AI_pc7I{1Dt31C_I+z^2?w+`X^)dI~&vP`O3KhgRs z>K;lg@NjHIy9a!3vQ>B*`$_EYUacRe6EJqLG@j!D@^)MKGT1T2<9DqjFdx_b80an9 zF$ww(S_CE4bZdTrPC`EI+2iV~mx@1KdMAVv3e#AiN)c>&&vzJkIATtG2~hSv|Dkx& zT}5Hi|3unKlhF0R=C6HXs2dFQXX;>tqMJRH(`m2b#`mz|>COYn$+MuNZg=H)Yo(rU z7*rkZ+l=;s1x#BV&EC*VW%Y)Fcu@uJwy4Qo0u&;e4J9bAQ19)d;o;p}9>eJ$#lk8N z6Ocu6kKKcN%@c+@{)!#gPm~PolP^_22X5cJb=y8YNE;BNME2IVutMe!_d#_21Srx;``JM5%=TR1FX|hCv z#{FC9{C5l$461RHDri`?dt{gwlCa#;s%} zNF~y7cfN|L=Um5b+#F;X`%enBDmEO-{e_)=>4bvUY(r{Y^PZ(=-q|$G}1J~O#szQcHr~q%4u=aRD=Y-`ZqK+w z;T+vhccGU0fG(`QywOq|R3iGYN9*1HlR`bXF2|9Yz9+jbvNgMVuZ%L{APO6dpe@WH zN&WsX(FSOVH=y&cArd7GAeul z+eTz<2SibBQC2{ibcQ*TGvQm_TW-8x{Gvf7z2jfLcEA5M(Ayr;oSt>c#3l$M$?S!#E(*ozmiP~Gq%+WH%o~=jx$X7FE8VX-rV|lpq`>jLY zSlg#ss4gHQmLoH3>*(Or0gTNLae)3n=v;i0Krrk^SCbPSKcMpVVnp?(gXYy_h0R~r zi#-Yhxj*lsqg>JE-?ef*ut3CDcu6@L8BvvNWtf#+KC;P2XxVR{{^^?cF>P;{JmTpp zJfPLU7g*_?d%4b)#KlESMkfpL*E0D6JQEGO0{De~bpXA~9y{ra^I@Db-;L3V^gMN; zv=ByOmSmhcLnPflK*}o-t}X?^0++DG+scIv{<3{d@A#SZw}R5$!`yEi&;fL;kX_1l z#B|2VWc52U*ESA4QpseB-P>H4rMiF+zO)MJIieg4_3|uP1N7`?R(G40yl^@Fp<9pb z5D6zt?y@zFxfm6xC)Uhfb7Tw#xFp}LuOi*L`QiH}ZIK!;6?^(mA}Zhm6}tvF`3R=O zbKTmO`_0QB`)k?tr`f%TIyzsmCvJWnl6=KV3J`dj{-S0my`x8=J!7STdQGK$yX8ZC zMFFlH)1hydU@lEGqXAMH&y+RvxnLq^lvSmk4Y-~E<{aHwA*VL=-r4%os=>lB6U9R@mQ>{!=_zluF=Cc7;h=}&3_9IT1aue&#}dVQRGgB2U3D)G-sPp> z`8#%3odwP_R^eJXuLaDWSx2&N{^1KU>b;b`kzV#O;?OW>1UXFLsbW59n0$wA$-q)O zFtJLA3o``IoNmzAf^v+9`%t*!qqoh2#U+f-kLjk*ajXV^_LZ;|$ zK;FVPii4)2<52$f#tO$(37Wy&=gVBD?3y?sQA)t@uN$L7CwQRgRjIq)VviE#ESdtu zmL;TyBwtr>safElG@(l=2H#jeZm77YwE6dzB>NqjjBPI~65h&r^n^BQtOwT8h$+jk zJ`6B{=BIaObNBmz$_4PG2j$Bvb15Xm6WT}HK%j*11>Oh};%VihbW+?(T-(nJA)_pC{4V-wn5mvJpQz7pg!Lov%&+YNc`$R-N%nI z)|vn@&vNs)4al8yD*IXSQZV0!o%g*ks_w97nFKOf*|X4m(ruC50xq0te%%pBb)Qdj z@Y~Ic*&q6bFnq?^G4E_&p2#Z_`sf|BAQCP00hHRy{F)3&XLp^?s(kx^_B^pOTThZ= zQnli?H<1-$H#WQWOU9HQ&o&n}c@|HrXoS}KrkK=)SJwX`1zh$4E(FA~&SNfyCgTOE z2{JA}KPX2wc#s4Y0$N$`ZlEdX{mOuGleu(k$|(BvN>i&dAQgTU!fDIjNU^a_JO zm<1<4YFOXQ!%3DQQ>A#{=uYXsl=Srk*D`rKRY*=lVd|AtDi!5}te%!1%>=n?i<<*oN3K}0V(Dg67gSJ`CK37ptN4S> z`xfuYEQJD_VtXG*U4ABB5MH*u_|aI-^P~Ljdc!CZcafX7%49>$2|Gb#M$z6dD$-Zq zg%P9Ws9s+OvFY@?aNh6GoGI2J&a_Pd0@-60xBqA~Z!gXC)fq%;wy=v*kB#^}6_V5T z82dx?C-iwNM#b&Jth2MHFaaLnG36*@>w%PcyjWc`yIPVY(?&OI&&c2>Ck9Rx z36pic{{;23a@Q<3^RKKNg%mj|<>wBxCXCcQ7?`f;;&IvwCW>f`RbOVFi`r8YvAImn z1f1xExWWfdOysZ`CW_o&QU&E6&(@XOz`NHEWXQKa(fyN}&0M-JE4DmHkm+~^oCR)r z!}@t$(PwM3#_smDc&%nMu-79RVDnl#jX8feh)q?QyOM-6bFnk-cxK8C4){|{zXz`W z5A8_&C{VGb)S~dCVL6o&)Ia0mNE^S;#ZMth6ls4xT<>3)C&3Y=;^1)iFza%Nq7Bl5&}+ z{o&Tm(1TD}erD-V77zGVdG5>>^7lEWgaFj{On{9i@Gq?m{NqsO)xf14CvcO#US$sW zI;O9bw>UYz=A7;Wk(vQdLij1b0jd1Uh%$FY{-P3W?8?uD_iWY&ObyO2i3)H?4qlK3 z`FWOuA(B@Acu*qSA|xvPyo6gR=$M=FAu_P92dDV782zJ9H^?Ffky;P-R|g}#CnGk* z&q_@0=K3o69#9U7N7t7ey{zp>lcDn}9Mgvlbv?@e0?`So{6_XoWJ;>Iv1|jPzy7AwG0>F0?qx+?-OG^M zOJBqb1?w^-sxC2WGk$1u7^dqk8Q||#fv4XnT>=krfC!k_8i9Hm_$jWpLw~Q5JD&V4 zZ8!Pw^VPJ6*NB6gG}2KbczldN955cI!}_1xcRK4yY-v#^Wn(SRfcJBUPj{cG5{Ec7 zV25t zg>htSi$;2jO|>uupXbEe$2`zY@#2C+je@coMO|RoL58+Q_a9@|Ru2H8UVg#N=nc27 ztn=*O5Q`&UnG8(~11y3F z>m!>c&f%BLe%69h*zGw`IIfSTT)cIpE!XmNWuUDo1kx?coE)ls^tZa45nuy&7dj`)K) z4Gq$t-*K}+T6cf0D>zT`}6ES z0A;T9{_s)G<2rTyp2gBfjtC?&$4X6RDmlHR+g5-Olfw9&;$>^}+q2Pkb;TeK0fCa@ zT`-f?U-JMfI0{A_R`D$CxK{JCS8`)j5=`DtP(286%%Fwq{}^g@3NX}pt-Q|8uw94i ze~JK5+uOv#(biN{`i=+Cmo9+uL!8n}ns5GNs5`$8xU)Z@tGz(EYTsfSpk?Osdv4Ex zp}0+?8f&U}$e7w1z42Grvwcke)hB^y@Np4eluhXn>+L_86F>BpDTp~{2)dTmGU56C zmlBar=bw-MV}fk%f|=&7>D0@z$8kR~2m|ipHB*-TM08w|J7qgZ^4|H&(nitW%^pXB z{iz+W3KkyVC#8l=ir;ni^%{G_Jy1^;ll-+JI(AlH%r7GD{fbKj9v|rUome1StF?MDhfLj0jSd+P_uNc z0X#NizpZ1YHb_W5oM~^j3;$y$S^JuOX$V|)sJi5oSZ8NC6KJCK5E3=}%Sw*3KOOMA zke;xunVx8C)Vg5!f3f%8K}~Ic-{_|4L9l@oMZki92m(jxO|YOKphDsW8do$xW!%p_zYkhnB ztUG=y6RoPUrkwvO0yYEDTQBUr$7O}wo!4hKx}%xHY{--G?O(2B^#;J0#C`;;XR&&!IZ9$j{=~sh( z0^aJ{INLQ)ns`LwlYKT=<5xO(3q;`-w=BSZ5041OuCwhw%y+UhsmGSyN>^>H=8LXR z1kuhXk^x%c%o9pD4{T>)Z?El{zJ}<~S)e9!!?~qjt(_1*T?8-E3Er-rWGAU2&{|7* zC2P)fvv7IB(*#~b6Y!g^EGkhZVuA~&!MtDaI$QRcep`C!i||`=e)S1@^A2#?SHN^@ zd%BF64c^P=JR$^?^^ALL&u~1YKQkSluYb&&0bofu?cENDFW3{F()gQ94~(^-9X%ig zm%EZ~0(KksI}G)3{oa@#YTv;SU?&(ll-cwLPI%|1;QffB46D82X99%S_7vJ$7?6l` zu9fJO(nR6Y%V4^J@e1zDy!2&wk5Dj9lyr?pPT0NEhV&MJiD0!?`2TQZhDJq9)W7P+ zS|^qLyn5ofk!*Xv`E5ND^pyX`!b6UYVeG+Fquh&#Cy*w>cwoX#KoJYuJYYDk0d0$H zkONik^uB9qWB2HU1@^^V2v7$Tdvd#;0_MP)9lT25Sye0(axgFFVV9BZnDT2J1%qYp~yQX%UQpv4A&K@i)E6n#&FL zKJewu-)76fJts`bcbyeGU`uZ(2Cso9G(F)BSbMrnz?#CRC9o-aM;YwiIbL@Gq< z$#@BTlXRV?oN-Ru`>ZD55TyQJdrCBr)~4nTmk|LQ0co8AQe*@5^T2Ov^Z{^4;X;{- zmZ|T284v}vhsfIAaC#^K*rhk;fg0F42b4qpNjcYloXx??`{9*$902hb=;7_*hdV0L zZ7WI>WxbY=Zt&JYC9r}+a>B#wxeAY@irk1@OnHJKjkw%d+v|6I@)@99uHW~T#*`-B z^m3gWEc&zqjKMp12$+GH5w~uo)jVM!3yNM>Ss`Q|%(nfZ;0rVJkbQL5wlKsw0|?rqAg3=zoayN3M;{?XfC%NFP&$iWLG1BTeMk35=c(XikG z-D>kMjxk*vRu=s%zZO*J0xW@YPztt@SsnNpS>fE%86l0?!pbgVjX(THB`)-V=j6ba z9NCxQbMdXS+QU4-KrG4jdXe1#EOvp2j9=OtiRnYQ8Rag33b-RAE1u`o`)_9HyQN1g zzgvX$uWvao3>H}&VH4Vx^;ye$#hw1|9oYc>cKWnz(XOC02)eMhoGc)TIbnKi#67c3 zkE~d+f<*;q+xttrarR$q>3`J2RLSK8mL3-_`*^x1NS_^cR2W2Idx^7!wVf`)SPKmA z@&@ne!oR}6fa>tRzzp!}(K3LqfZe$DdxuCS*&gi!aPz;~K|~|;HAd&rhYTHH!{d}^ zb`C+#Z(%=0*9Gt=<$z3{Af$goIVE~_ETkdha1H#p%dqFo^6xsEkH9s+H0jOUFrgz4XUk}U4ag=h|T>7bskc3K9VF|bgYloD3q58&J z6MGH{oz)9?As@i2^{^jS-b9vwo+Q;f5*pdcnteud`109QgA5X%`wclj%BblxRnz zd7j^`qUh3r{5ODsNu=69qw+dliDV#_7ej?85`+`{*2V($HlB5|j+f%}8O)?V#eVq<+ySVM^!*xAgK!7dkL9!Kk*vx-$*(Fx7y-th|7QZ# zB~MtxITy_hNyBmA8gAxm=6>^bl}%mVIFVwQ=Rrhb1m2PF#6bAFnopZx{^T+!NC5L1 z$S3S@ZsjmdFBJA%m!ijkkO1eEdQ&g9}QFh=`#m7Z^JJfN=62k+jW)@6luPVfCp z)dh<&(fSa6JBJy1m&0+LvIvI+zh2;qvi8ZDTgksAgp}F@t(j1@8n-vbfTRYuXd=P3 zm^wLuc%WeC@U~xsEdP1`l~$qsPjDd9U77+RkFaKDqG(CcyB8<)M{V?g?dBoC*boUv zlcXI^__+@ir`BvubtkFQXhl;60W17T#I9{Kx2>`O8)U8bj`^;c+k%kOp&iOcPg+`H zxfx9-v3z$uqZ6;PJi1T6z0V}yW;CF+f2ujJIuVZ{d|T62tGN0sQ4o>jeN~;iMMOcu z)7NIf!*;>LqtPS2`@6PuVb?^=U8AgxsiY*U?o@yq`72yBa!U20RCihjEVXnFRTgsh zSGyJLE_SMo_-XDgSnv&@!LLbi?O?Cl_$X6 z)GF~@q<|lPj;u7Hblg%laixsR51XR`It#FM+XfpC%rudH1QamuET{IYr;!eJsX^rt zcBQ`Ipgkk=D|Ugp`GJd$(w*OtV&_D=9IjRqQ$KnyecIPQoIT}t&V@Pb!PGg?(eeU1 zZj%=JY_>47p(Dtm<=Nla@yv?uEQL5v2J1@X)$11Mbu}zKXU;f zmG)fr9QksYx-RBtKak%_PXG`Oqbly`M;KV_U&5(w!?WhHG~Bxu@HPQ>qerIHVfWqf zcP)H!6vNfVc6AZ_|3CXqo$< zry3W+LqK@W3n*Rqlx+bElmnMy+^J2>{OPL~Ne`X?JWu?s@W1$NGgry{b7s@RvO{+jrwSt*u z!$UrK`Y&NB)0B#l_-XRUg=JeC}PQYU%=De^Y?*ceSm{yC*H)ecm+^jF@t*bbvQ!9}fl=8EWnm7l>4Bq3v+?G6cHA;KQVA2GhR8{VGy_nOFO; zXP%Lb1;fIQh5_R~Uutf0r$|(MUq9oj!qIBh;nfhH zjZE)hhG;S0{F%lY>Ah~y(xF_L#O1#V}(Ua*)QqP(m@qCHnJf`4r+am%X^=uo`kI4r{hBPt9 z;`>^%g}Zhlm>@$LSR}gyMrv2X@y{^Mn?PXg?@GCt)>7eyr#1f&=@$Oc~c& z9rVAX)K&H+tn*sp?`R zEG~B5mN~*rGl4hL+}FFu{|ur{+=Fot%DtMsRy*@?^@&3??NgMJv_C;p+-%#>K+45s zYi+EB?KVp|9n$Oz$o%%d4Jl+ua!60LRP?!)y!F%pB1aI*4e;h3+7*)4rv7Nhqx+`5)C{E^KaR7QGvVYb1tLj z3wVC2`M&bH+GT~#_VVojE>Y8sJaZ-5PSozRPtsBTYdEA!jj_Rnh_9Tb3<+Y)j*E2x zEAzT?ru*ALfv{n$Gbw3m>1Pu)a}s167C?A}4ECChtuy+za=JEjcfHw3CM}8OsV+yd z{!#N3mMH$gWeO8MBW+nXFML8BJ=z?9e|w=4w;7O%->u`(JDzDP+v4}5!QAk2B(Y1G z07+Qz`8lsZGN)a9cm~)`SxeTGmVBPyTyE-X9P&K>_A*pxy9O_6qC{s`Wgh}Jug?bT z#qjnm4%UHQwRdLB+p%1He0tpFInUja3(PiDA{DiRW-~<7_v7S?frsZT6N?)cFvQM$ zY*Jm~^^6z=$thYRRCo~t=79$V9xU#u6#(@|4&|q{zH{CH&JcEDTtI*poVj2NCEV3~ zq#E7!!>l?^KhzIsY6dF^X$EX9ZeL8(6x+&24KKqU>THqNkg>x$d-M{>FwFw7Ti;J z%B0hG5L>Mx+Q{{ug^AnvqsWK9{Kzs8bQq$53XbcnLCjzBk9 zH4Ijk&>=a#`?)dTCjLw9GOlc8mPf_OpR^n$Q-Gzcf{AuA4Lp&eBjcYrB+-@XkT{Qr zU2e;a`NmezU=*W!m$i@)7KGHTv*o8MP)kcpQGjbOhOD^a-$fEQFi??&)`Izf zF9|;XcM&fB|69Pj`2VE<>%X0t@nhF!(b3{u&Z5GnT|9dN-Q3TI1!1ojAO|MyMfCn(S%f3WpA_ae)c1ANYxEncarCQ4jMq$3{u zE;$;af5XwHki+@)zFyw*hSKqphtCc&`_{Rm65VuC&?fF_q||Qps!B6NW)~&xS(Pkd zHHVNE`8!s8{O9nGap0E0AKpKQf5-yI6nw6Rk*>?8+ z8On9zzQ2!8)dnAgM8TCa=&>n1;d`}be2Ri+FJ|AKxW}!PrlRI`>hl^R9#^b>|7*H1)|h?+adY1#J5~DVyM@QKQ;+y{0!b zXx5u5Lp!m>KT-Umh5GhGk2U1xcZ6ciOl0EP%bhMlqLIHJ*Z)HO5U-ed%|`3rJI#2e zz$UpJ2BBCUW$%0(H}WWB(`}I0feQIdcX1nSywp^2NHh;*_34m)Fmw;@Ko#YSHAu3w z3j|oWT{;xF3{x5zI^y}mSsW;4%7m5aFcp&EgBc#Ct~ICYLY z(Bnamw4_wC{Cdd$CT`m?g&9_`H}aB>SiJqvAaw&?yZFuonI2*R$x48V*%+s#J2}A~ zx#&x4;;*|{lPHl{yZga)Yrl5lYVvBuy1*&y?y3TSmWz$@C*vpkWC+UZrg-8?7(Z5# z=u$sbWkB%DIVu0Q88i*|yg-FE%$&R1<;q{SjMcJVI0e*5$mLfc-70^^5WBnqP>uYZ z1qJ7dLJ!lD)j7E+_66cdMsdZ8dL8Dy_*%eX0YQbR6lLc7C3yg4MOOs_@Is=)`FTnE zuyM9cF*>r2+OcPQ7UDF^W3$jUJ%RcG?F4bcC@qkvh8lWJl~?d*d^&fW{??6vP0#i9`Y}B= ze)$y*y1-EQP#Ne&?_A9A4Jxyta%bxdggg!F2HZQ-VoM7G=xE+GrSyinF-$PFQPA5G zj~?9J9o_--ch6;$d*Ehl_#ucAeN}y*zOZ#=ME2~e{7$pUWKZm=+jN}tSieBMJgpR~=O%Lj znQP<}zvZDNZX>bb9-rTuOHPgikiQv#4?` z=u#vvbxs|x6{gzaA~~S0+q{&$+N(U)C|CP5jfS6D`Cdn(j$#sL2dvrVvn)^FH zSB3g*q`4YDJUDPwzJ02;Q!;|*7;&f5ePZywAN6N{(Z#9l3GU0~v36n)9tj5KB*`!T zNKTjiv?dmxO7P5+o=k)Vm{X6ETDXzl?ZEGqf03X}RUnea@cd~dQO6D32(z;VmT5Vs zD&|M}&G!eo)MNs|5WRDWH4^=SedJ17?;sq3h!An-^%{t9n7li?Pi6x%J7JBaeu18)I90nuUZ@= zYG@*%&y?zb(oO1%TD?^`$a!O+j43V_)7Hc2KK}HHLtleJ*EU5iN_Ph@$WfrxX{l>w zeAs|pQP4C++%Qk`vz)D=qWO|?n%*>qceNe6e9io3^5i}Tryfe>?kfL<_X*;2T`u+} zg7{AIQ2)L&d6G9~s?Fllv0~5jjg2pFT$;a4n;!6p;JwL!bL|B^DUjhS%z>X<1Eab| zBdjxQ$GQhHD=t1U_D8$RG!QYTB;J}1@BfGT)u{$GXSYLsCExwB_flnVu5hFtQ>KCI zIR|K1udfARm>^lsTVBODq%}jOCI$EvmfMe}?L1oWtsjecgl+9a2dcDN*2-x!-&B|6 zd1&6Vu}wbg=QOdp+T@!yC#C>+S=8GRyQ+)eli4_ndiGu{T{~ck5HL41T737rY?gr} zQ8v=YVFK031F)`Tru6k?@R|QqAGdQiODnN|Q4kFG8&{+?so(E0K5#W!j)l zN$5&mm(C?;t#j+bM4@(8$e&CtH%o8H*^{&!`Qh4i<}F)we`)3Icxo zD7wXfuOEGY3MHT1hdj53)GqqD5W^nKci4v@!Ayr&8^gGlA$ScE-`c44yKFV~=RA8) z8^wtHtQ_A)9=|;0HTEnZV2$RK_vR)qEP5(1Bq3rlGL)kHnfx`HKzcJ0y;d_Zqw7R_9!X zmA3l>DECP#4ysPg(hHR182-*B?0M=?u3tAfK5dAvsbI}N^P%rIQu+)qUtbIyQg>*d z78hpoR7jmVk+Debz%m%X4~odko4=-3jfrDs-kR+mVCcWgdSFR=7Z--=&d6)Xs`ao) zZXGKQC^xOjE2_=zPnT_cjIMdOM2@Wc~z~>huaX5IhE<3d=D|g z*%Rq{@`gijVegF;MJK79S$^d&GDi24k_W!36jsWde=w}3=xVcuCYdr+#`Uhh7n?57 zGzs6HPwdE|#fz%WL7#k@ zY|FYBv)tl$kmf&WVJgdX)aMYvQYET*D0t9gR(o(f3->roxzopC@LOXFUSAL zIvN>pr_7a}P_0~E>j!LTBu9_+&dg4--I=s#Q}%`~t>%?po%?nvj7K2-VAz&$YoecU z*^K}Qg5lvqif)g6=2|VehQ% zW!ZK6`qrP})Qx%f6nC8zS$zG6s^%QGSh7Ci8*@lqz}NidO>I%A5OmrA2PgVmp}oJQ z%BO#f53|NxDLH=e;&p166+kL0%&U_6`;k$}Ggpy+T9|Y2^)o(c;~}p=C5kb#LD0z# z?}5IX6f-qxn31Q7q9i{nMcGsVZ^KefkBSIQ?L?O4nO$u*RhcRw5=Z*cW4HIk@8q_- zOWFM0K>5d{6DdtB$ji89!V~i#hOQqqUqakK2$Y){EY|26W4@TXe;1@%lVzWy!>R2V z)(Bzqw@$;>5Hf?OUq~QAwv=g33enE+6*QLEDn< z<;mZ~5IAoLcG%mgze3%V*v??FW>{3&&C@B>XOH^&9w1e$Ne8zlcT-jBEV|qrh(Z>1 zz@m{9R61YDws^li?NUk(=;C%8x#yjFU~&5MBu0TXXmzi0g)w_$-c#;IETdr+= z#Z=ei@DzsV>bF9}xI?&maQhn+@ga}j>4WX#W6a+Y5n0Upx$ypD#ZHP}4Dym8SL~6k zVkf_H(Z<4M>8ry-sbvM_+B6w^CFY0hZ8CWG%MIT?>fMVnX0GOzTW}B)cI|MB%t=>h zx4hliMl@A-80f7g1yuVjk1|*m6-R~67gp)tU!&*lUY$jkT`Or*uyM-C z?ea)!5wyt9fBS%)haUG3o?3%$^h$@B)#u-3D8n4+L}xBN6!@SytzfHy7v0DS*lY|E z^0Ui{%&+UeoK^1$tjDv6<-GgUpu6~kdY@(s-7l7*nXc=KJSNe3Ivx=aZ|HH|dyYr+yT zDV@c1eT!q@+;(nPcRH@Z-=FiuSb1v|Tic;(=`%#}cCiot%6;-*7hmM%T$xI?;xCy? zKUGt8OJdrv43WuzyEnMhrQje&e&8@%_~7bdc{?YRaCvcuOl7$s1fy{14kZ?fq0EJL zNL$Q*m8kS7cj~)|o26I%>*`<3FQY#S~l20 zRBp3^Z&$oWvf0uwzo|aN{qlD_DHsdNNRuF@xB<7(g3tf6(Lc2kxLUZX^6?Jc_H!oE z>oimgme{VT;#~_tdVBc@(g^VCUjwBwJve^h;-D!F^q^!HdL5LbgpB(Tvll>MCh{!u zc6`|x8J;e@mj)dF#$~fdiHp-V$#0fxdwJg5F(X&@QW*V}b{?rx#N}3@lLVF*PX22{ z^}{`^tK(RPXYE{A_w-zh-aCTp4CNBt@-2pV@3Cj%8_gmT`w~pySU1q!aqDF{HPs~Q z$*5TxUYY2EHv4K799j(`ymkMn7~Ao>&3W_l11ipiBh-nX^tcH)z=d-6!lj`ycMA}~ zmvFqzruj|uL>G^mVo|HfEkow2%k`69k!)-R_p<%lGw^MmsR2fok~qe_C@j})<{b4W zVF;hIh5r`I&w-k<-PP!nh#2}_62nq9*&MT%_-g_WF$1TtZ%A71)qvqtgPU?O$_4kZTM2F-PgX%JLk9W#U8#>cv zseVKbyh*{1#`>j2UXREdm9&SiNmdCbH`eGJx26qtOBX96h?-ofN^A06*fJEhjye!v z-ZfO>+2vO0-SfKJMHfF22IZm_&jWGI%*dQhea>4*3+2(_`+?^g! z#t4vFH{&(xM;_q2mQQ-Hve`~~b#Ks2gLB?iIX(?-yvsBc9O+QMbT_{)VX^ACU~HSi zX%|QT%T#Ua+~cc$16pK?qrV{Ay5P0^-BKo;L)uU4$k`_cCBpVN(S;f0y(E zS(42O52eFAPL8;(&5>p6U9_#Y1N19FTR&2JdFleU67}Eh5Ppv5!mw`o#T6xFU_l zwGypO{lmyhU5x3ScXGu(wEOt4Mjc-7PjV6Juw5_88|Kp+4r-Wdu+IrUTf38`_GPQ6 z)om!#$0TT~=!KQZ^BQqP9~pH!DR7JSvN$!Qx3jRSGBxzGOS?{_d-$Ypa{A-J zO;^)8>~5t+Y>9)ajWQ2e`e0}3m*^epSgj79iO&7q_vUz)Tysw{Hn~JqDTY*Ritmg) z*^cqx2DFX5TKmjJS+lcSJC!HnyLwkZ&;kZ2XJJcA4)-V3`4ueY%rf)g(Pr#}{a0&5Y;67e>cnn3u%wh=Jct*9!c{xK2nA!N`4gJ$TKDhoUCzC$+po*%9#VPf)nNSj$ zEE+mh)S>*(?anLGM5%5AjxO&JPJQXC3lT~!r-@c?|HeE}0zhks@0ZlAg=}-x7Kg?+ z?Fn8<1BRDQIw+`D`jafsUGDis$@t{7?SEV=?*@&nc{w@?(b33_K9|Ya+y=r2n~3QOy><-ZsNl^iU=upC^|YXb+3p93XUr+XYGXDz zS31`5^k_ztH`ns`ZcY-IALLy5J2qFV?;jrr>emr}AFMv!5vw0M=}`RAr{axE-&GPeI&4)jR(MW_I8t zaXSLPjx0)iLRBkcs(q9ZZI?MZcA9EffaFKiPfod~lDiCs%w~(2B6YN6 zW9BSFzx#)pje$icRNeufq4@K!^eq_7ck~J$i z(l3trcOZ$>n!qM+t%OKWBKUCH_Q>O#5p{2S`ZXS6N%k4RV;>__2Cj%$vHPUap0N8; zCC$a>F-gqcgv0)x14XO4WF|+_d8LlbF8nu{<7S9SrQEIZdAAs+{dxmheF}0GJQ?pm zD&_;fyCQUZ9~=Wd9w}TxiwWDa+#)??(s6i;ufP`+yq60bnKRf@Ka3S|h*>*##KgwS zh-t2qEV!1_pVamvOORW$%)u={cm56Mp@x3lR-ayt)A;+5Jzgg}SJ@hZ8v9N-HNRPX zMeAdkk4bimw;1iWwMcgR{_!}Q0VzTJwN~0>M5Z?%n&%ApmC6H`{1Pj>*U}Cx!VYF3 zjS#897o0hdgWllm|HT0GrFjhvq!OOsMLW>T=*dj(ZF^_rVwNY=*YiGCsIS?9WW*AO zc3Qc2%hE@Xy+hKZ;~$^su(2DDe%?tT>G2#!>!;}i5h`Lv{fHKt)XZHm~r%<8`O*% zVvK1Jd0kS#?5(n`;7ioIZml$pdhhX_OVIt$W(2-lX=y=5Ni@Xd~8!Lrme4xY>?;E zg54aK*^1-8Oc}07e1N2HoHL-WDOJpB-(2-G;R$r<*Dd4X&Pc(z+T<|u>zCnA?z(Cw zfc)lH<%c~25%1gF4eG3lyDJyizF8${4J}fF#&Qet4z^{{rT+Rfy!0X#i1DI3gd`Rg5-NWn6M%^0J z1@nz(X~90qXNstGPl^rq&Od%*a)*3HM%7s4ENP{JQnqbhkXK-*RG~d+Y;Lt) zk&HK1^={7~ZfM<)o9(i{w5VA(P=!8t{nDH-F$Ig@w+oIQ?={tFrhX})JtFuH_m?Er zTIAfqC(;v0(P%*Pm_1jwM)PpdwZup8dvD(p;nZ=l0ppl=9^qFTaI`mU2a}xktLp#E*PwpdQ*m2$ZO_3|N z&$(Z+k3?%!iP`6=;Z8ow6|&$hO||Nth>eq!W@twrb`$Y*OOBI2)kVootD80c?4IJF zFLQO+IDG^pz-tFqjhka%ZNgsMqBP!7LXRj+JY~^F%=W%|c35O=e=GTo+jmavfRoxk z+nwXDHvXxMD65LJHi_%Vo9eb}Fn;ju>X_l}G)t&|uNe2(v_ALtV=1aTxsxGY^Z9H2 z{V!tO{pRAe=UHU%K{-{7P7Y7>MD*A@q)ZrDP0USpe=^2DI@*)_A=Sy0c`~stF<7lP zB)VY&MGl|AkOi{rlw}A$`VICkf{7*!y&>!5AIdFCrzghBDH|vvx+>qzA0TfI{1A(G zOe!&rv{Re>XvO5GyDmB4TB6~P-DxE{j&$qhlOu}w=nce#(WrKR41(V}!a?TP>{Rv9 zZyu|H-5>(%-w)5ar;pQqEE9P~F|`@7s6DZzSar?f1-*#qE|1A`4zigsgnny}j z3i#W38)#=V%06i>*SK5GcfCs?^=?$3Mv{`YuP;ECWjSu`Z_wg&9{ue?pH?uYb>~d% zYHUb$u-feG_91zpV2r}t)(er%?qUaa;>$BKmxhh?Kd@o%KQiUrP;EI=oaU{BDp0R< z`N}%R{=r%i%00{jq8Eo@2E!z2`@tUSd@&Ow*y$UvLo4R|kZI#~uUt>Kzu^P=Z^_fW zKs`BpdeSf3`}EZ5`)Fl$4<(QXxQo{oC*%svL&f}*#9g)Eh1?6IY{Ba$(^az;;h_uT z_wua89@%9U4Yp0sCWce;C$}Wc!pAE)yYG@K^Of04P-QKP7G}HyxfZ7v%^wx)e6nN; z{4xzEkvfV_?A%~a7n}5m{~Om<755OODfRKzUqq2dWarW5>^IIUwYBMwdnO#1^^i3*}6M{ZI0(Q$) zP~Bo>!bcNI))>>o*w2=2eX{og1;k8AUp zKYZ~56mbJ|wS&)p>y*&P&G5@+LxrxO)r)&CLe|vc_2sOc*kK* zNWTlDML3}w@S*}aoST@E+TnN|GW3Vb)uRNn|6E#OZi$Ce(NN*GG%A{sFTPR6`4l#;~3!%k3PkYnYVA3$7apGZuxrt+{9Uf~&ia~G**KUny718)hy zAgX-R6iq=$Jp`w*LF0h7502p7#AG?3&pt>e`ua};LK0(h)JgeXmWac=Z!X@(?s~87 zpWLM;?dsE(BX{2yO4sbfAAX%iwXQRju2~EYC6BKiPS)$f%OX6Iy=^P~sL$HIg@J|w zLsnHuc37k1jHQ&6%p9v`KDy!^%E^2xU?$%Z%#?=qxq}h;cyKUg)yATD0Hxg$oaAX~ zYm0^;w;7lX3vcQuk@beayin0A*(VdBT=xwPQ3hu22=SAV$!W=%V6ch{LSEzmJrOXm zbEUG+V3(VDe0P<{BQCYc)KRofFP~VeXAyr5dPvs&)j3WPlj$JXX@WREpwHy1jp-fb zSM<7bv*yORBg%f8OGUF|r5|oOgTBiRFkyRLlKlpln1k$|h>W#rf6zzI!*`o^yv)6j5Ez&ZQ)7<0Jn7ZA{% z^fJC1GW!s!x^}1I8k~eGWJ{d?OYf+sX3!X8bLAn|F1L~&06IbpDoh1!Zo+Wr5|z(Tz5}mVSrf0!qn64Qst_p5+rLqb1zTAx^dAev6-l@Y}HbA)4q@k%w`{`I#aD`rrWq z?8h$Mg}GVu8d|vkyER)3W^hNxFBH@i56B|{Yv3Lt5VI8C?0Wz+A&SHj7zR&mLxuSn z;a&g|x%QRpr|)3W-f#g+bv(_dDag?XpAs6{5S?_SPij9D111#NVTcUqy{-Py#nIa8 z0b8{z9j>dG^u>KhF#rm(x06?5L>>yNuF508w~gV6crXjq zDKe(#2w*M5G)~n3JoQnnh;0+FV|M@9&UZ#Ba9b1~%MWB;JJ}aN$dl{G?!b3Q-@Ah_ zJgsQX;UXdxYJWT2G^euYZo$PasgwiWkRgCfdLs-odX*jMsWP|@S_}XQ_%7$X07p0P z&Ax(W`gmtAa4c>u0lc?PVqa7C>+Z7DC2mOn-`@lcbi8~0c0Bk^4)c=fB9zOxw=anw z!V~fz>2nFm!UWT}`+JFnwCpoahq_hi$Noe7b}e|P3$3B2D(D;hZ7!? zr?|b;zyxt>{-z&pD?OZ>GZPTETUS|q_%2j<2i~wij`!i*aW_h;U_9;xrr%T#1w-U} zl$5B{JKC@UQ{mnnSVf6mKr|NC{GmEKED$VjfA@tUi?hx=jg9Rd?lUkq11J9xd~>U~ z1MluBk+@`$hlae&Apir#088m~TBqgxx@Prm?@%P_ETS>Gpn>K2gUQKDUcYD$5J@~2 zSF^zPr1#$R+?U)lw9o3w@4~tD@6!L&<22}Nf_VZcOq0r;bvz4nIa~Pm{gM(9jWc0B z<1b}6kq$QcT*Mn)Ku+L$S-_-#SfsPUK~J2WCRliOW~vAPz*z`wfsXv)qy9sHP;UU2 z^f7v&U(i+Ukye5e{Ba2Uk#aKU;yzek!m@>%1+r!Rt_qd|DCWG3Kid%k&~~*g@bAFx zg9>he6^9FlAamG*Vl6E#vT_ST7VJ!V^?w+n1|u@Ic2*n5m@EG1v~Fl4;zjt#DSG72 zAMJ1f>^iyN1ItU~PP^7ypxmqE7<(l5kd;uWKC{`ypjXfWROQ~?HWUId7UD9YU~+MU z<=NSQ{VN+k08$4q%;tZV7H5w2NG(WvCh~9^W99BL+#_&BdWaXdwI`?3+34pvuM9aP zdvY%V*oQSaIrWf?{<3Jbdg!xU7TV$IMIsCZTyoCassK_`qetNrU4QgM{2( zsJQ=I&U1FQoHP64G~B;tA9%I_xR8WO<=Dr3ir%i#mY8~(bjC-t2 zdM)x0nIV<)V%5_I))Pyh0|Aiz7maNI5<`v_`0hV@%8&FMd^a!4@g5kb3kGx+gv}ys z6aqiNkPQgh#dD{y^BBBO6wm;9mjCAgiCchwXUvo99QO1(a`KPJQ~?4O z-UrLyQ$vCF#R|V5Wgx*ouvR5IeJ=PE@!B6S#oGR8!N)H}9;VwR25bQHpUVpC%fcJk zXZBSIKY`IJ+VaXrhpE1qRoqC+Kb%mNZlvW2)ciX7IpQmwSxntCaI&thxac83>2|kyrtTxp4mBq{7s!Gy4n^01rsffkQ4J7I=yY(tjfI zFfbG>R7kdq@GYbT5(SRV9#xUOU{Qv~fY5PI_a1Q?{hTiDKElj&M+w-tLW$or2)y-L zWDJ0eEEtvnZzOOsx~`o0xl{E#X4C>K@BJR1w@-J|=DOQ}ljn1~2>udcCk4hn0AeNy z*Z7df^$?BOeA;)SVZAPDPFsm*f!Z&ecBvl&Xd!ES1h4UzwK%U`+745o2LN2T3#WDj zRezh0E6KAMMh^Oifql_+O#rrFZYOeE3A;XQhbm!6SB1A%^eDh>u+stivh#xdBmqoE zG6j16VS;`U_@X(u7*M{tVeBpm_!{i!G4R)z_rP?7-2nXD@yB7xJqNZy(17LvT)@5- z!3rSqk5k|u2$wwhWUajV#+5o>-d&vwR@oYV#S1_dO!k{N&3~WcDX)^YX%HnALh*wT_4GCJG~LypeH=EhVx_p26l^`c>7UtKF>9fC zGiY+#a~#&)5a|h=^|jKYTUJ+IiS8drLNpp_1GdrU>c*I0;wk$3mL# zY6Tjg-hY@EJd67Pck~5z`7Sqc`PS*4SIpu9bL8p1iM20sM8-l|?xGq5WVq-AQ())2 zD@U+cQ_9dp9Ker2Re7V;vaYKVPh1e!b9E#O`D~OvU4tR33=h4@$~y4kEW!eyO{52M z2P#t*n}Fd~vJR>SUFaLMYC4HlSf}=QlQjzUo@Lz2tx`P*xmSiI>}uj08QKJjK{taQfSvACnfP2!( z{%RC;e&y`#_iqS1%3h{VVv8qVm$W|;auBVSUJ#9UqCK8To|(C~P$_iYbYc1*t0MXf zkERZY>U3`_l1tKdE~$u@ih4G8bpNEcyNx{_%X0!W&bv;=GED&pgLfco6l_1M&M$86 zw{ihwoJF6OE%5uV=~JF@jf(CrjqM8WuI?|0_q^{{QnT8u(qS}3q)}AFb~ktBYjL)+ zJ0Hx2Ws<2*RqWglVO5TSH*bYFc*?2X+#t3n-;Uv+!%=3w{(Elp%{m^V{1 zD}W!`du|JvKsBx$r}(nmF7_PVOsidz{p3PFp~u1Rxx9N1DFu)Wx~n|CyJu|M-@9d~ zRe50Xe942tI|=Tm9{|)5reT1}1HzfWR_E0c^E66j2{`;aUyc;++D@r6>d#1WOg#8E zWXNYub3I=^YMg%S7cb?aHhBFj{C?sL4nXZB8gF^5ru2Qi{@CipR0kP4+=|cAr=iiN zJHPj)AM3X6UmxNQUHCCEPB8Ey2fAQW}-B=zHsPZ>S?(DesxE)b&o(c0Egm{irD@#BFj2C%gNNWi5vRB-)7 zQrga8CTQO;00Uka>=fClQ9+%>tE8-&2K{K4+0A*fSm{PxTvv*>9ph4;b6~ylY<|?N z4P*rZFTx4ULMzLaN`+LOTC^a59^o~ zQNGqP&x$Yy7s!!T-V7`xmDrmECZ~Jfk#aHC`=CZy8?H)A_0Lyyomjq)Q$*(yYjkNw zL-OwHMMTiqx^VSt00OXW-OI9zkiOB@F#vZXSZ*T1oT{4#bbobs?bhtvg!f@3J*YqE zzb4BJlSX}aT=7LrY()?oV3%5wjG-i@L;wnxvd-Y-K1-`v)FPr zEmn0;KDZ9~S=xL~A`2vI4=Q%lKI30NNU=m19V#&M$jW6=p%S(V&p6f;y;kylD(?2( zy`-;6t8+JCEtE@RsHPIdiy(}%h>J2z?vxWov0{dfD;&144}k3So%oEsZ|V5GwWY4W z)NOsTg|8gUD3&FEc8xMyOMCjXF?wl3+<#J~Th!BoM2fc`88YeiJMHq}%vRA0nuQ#t zVgvM>Tnn0LDox+~tga#F$%6k6a=8>V|SS!1+H zR3@v=#jW=jcERZSnH5=%RtJ*o-$q&xEa9OyH>;fpbMiMp z{-NZFBu)fQ??@bF+a4YG%b$Hhu+V`epw8>Z1ynb%q@!!NU47}U?c|aTcGQ!{m@j_j z#@d&IXVfE=?{`sRbY4gKQv<{SuNU}|Y%z%y|9riU21$B!X@Jc-i?-BKe4S}?Tt6r1 zGyFe*3qB@o5#k*>0dPU#Z8-CCJ0hf_69z7@d5U3voy0x}8nexJYYWKzayupNQ%c^+ z%xj7Gto=2e&vG?~1xu(iG{TDIf$w z5eP+^sGtN0f`k%^fFwZZ5JKmD{&DRy=biV%nRCwGGv}Q-d<-+0lqdIn-R1f{gEU;M zVq7mGhF1g>Kf9YWKrSfJ4K8X7?=wstwE4)R|Ink^#$O*_lAo;7b$2|_%PC%Oz*lt2 z{2#q@ODWmo0e~N5Fu3^SiO!mTi*BECUeL81yx!*{sXin>tn-NdNz0I3IR~ptv}z*` zlOyeDv=~SgOP-VQ4Jmi8W5p()3-1!y8^e>*(+@oFmVYfNah`w{KnsEFu8|m8#ed}n zP#LX^U5UiM##B2@VaBnUNgV-Mvl?VrRO{MN~!d&fsSaz*!!p{m-l#_3$q$)cs~DCXy5DDk*Y z74K^WQKs*aLG-tE3BUOebh(*#ioHIi;pbgAiOI2US(+Ylw~^*|WY0-=7&OiJE;8QK zt@7sI3C=&)$Z35*CujbxhMGX1ahJy0=rdZ#{8v$?hQdsU1z~&l6g+qD=f?fKzjyn^ zk==k|t&3eMzj@JQ^{m;DT(tb5@IP`q_vfHfq*PeT-v_9{h&DcXl~dh%G(q5 zULsTuQm&2H{HGXiWOjYr47Y6Nt8b6&`qVZ08BJ*nzG|iF8Yb?1PIRr!>m9Z+ycJnB zIw-}!JzoZVmR|BBn^#(IK0b4p<4H&UK_=*rUMkrS9mt1T8%KL7H*_1eyU*ocDH7=; z8k`kZpUsZ4>upY}!_jj0v;u(uf8osw41ul}|GZsM@lyNH7=U!3>LrZi6AzZ`AHECH zCrAC}5~{~hqQTWEZ=w-yr38f|9^`@}ma?WpzlH;N9g=A8s z$tZyZiGncomMkH@YZHLfw zEt!*~^xb8HE!yy9X2{Iy@K0YoG+r7hn2hg8ggE-c(f%(fat% zYJ-M^6sWo&(W>tYvet~tdhBr1$Oyx9y-Qabi~CXjR~1G?wcd{6-eD%W9hs4bI6Kli zi4KHvB0O?;ylrzAQQWGC_-ICxprJ+-q8)NGF2Zz^C~g93je6SH;buK}tPDSL!0adO z$A=x~v05Lj;c;s?)NQ}w5V+X5J5<14RQVD}zNiV8DC?>V5@)C58+}wm+>4v4kk_${ z!HVsXLskZlraeSDIjj#w+wx+^`moNf+n-RCE$VU z?X<@B$`8A)TpJ3w^Ri@GzEHYM?7A>l3B-q3i=*XHM~H6Ei+_J$rQ5AX@Afiwgl)c)wH3^rG< zW%5q+REC(tOp$NXR=6hiy{`w)1S%hS_w?-pjY~QdfUe{E4)stI+Mv7+4kQzZ{Q0Mt zT|F}spw!o?3S2 zFIob^7PF>+zI~$v6;?F8G%IiHhE(f>@W#XB6!>!$(Xhpjhe({wP*uQpTcmIkNRR4; zrhY)|eSYsBkg|m}vZJA)o<%LYt}--$2Y>$x)%`CJ?*IQO{=Mt<|1B;4|JKvlHa_}H zSW>EXq3H|Jz`%ex%prXN${#S;)w2sjU}Vx?jiG7Z^GJZkBJ*nC{>MA4-2~XvwY3%v zWQXE-Bod~IGPOjB%=g!Lxv(Yefs{Qg^i*zcZY${8QV%=_4j;ye(3?K0T1L6*q*sATZ4Q!0z%HGEH<3hundT2VlrfWnX5w5lX14o|BI2V97UO zN?{mq1UjHigu#Wc7|?`)DovQRUiD7ce7wr48VY7=4hL!r4G2e2eN^?ISOK8b0=+a| z$kl$#238X2DI+->1X?QK?KGyGJj-dyk6ITonL{#wZB$w^rQRz(n4gMk6SxSIL( zBdfguYUY|cgU#czjXM6oad9kzvQqz3@x0SPW! z>6(&aHn*lk6a>b3Kj6>cdJN_k{}VMJnxgztfXxf?JG432*1+DdzJo@dDmsR;`3TqA z9J1M{9d^HoSorpzgn+faaj@Ug;I2ZO>A$#uZIM$Ur7tUH?HoDgI)^wav+k)b{C@Sd z5@`YxM>-@8$U5Mqir>LS?0fR*MeGvTgO(7-TY#GwbM>(nR%++@TwP>NS?=>?E%w5W zI+<##w$jCQ#Y$3V)j9WmhEHQo@J!zr2>az&5&!$?%=x(ELNABeZyYF7_MCzCKm%x} z<3CN&xf|a@;usdXH??^7+QiaqDp!G2{3nPx%Q~snzj>pFzB32Gy7UZ#gui82YaaBM z0r=dKXG8K_w?F6GlJp>diM8?&mp)~Kc`(j;3t0Wz%ur(9Wkp-!_G-&f7e53dJ`Bwm z2r2|F4chK@+_y&@@fk33?U$8v=-;5DJqK}21m!rRs;9-w4_Il20}6r~?h~7B>FtVL z`)vm?mt1F*Xm3M)d@5}jwlrAg`_vtxVg*i9Yv=NicUuAJ3Po-`jfO&YG8F%!E_SS8 z)SLUsnWWpzTVJx?>}7RkygHcj)%Fy&qYPeX`6F>H4|OSG<4L91-Dm+ zRmKets}w#`4x>SyZN~=*hZ(?vpHa!9^1+FW$apB(QhI_@p$A1QEqe3=fIC*x3O49o zN^Z%a3(yuIulNa*r%_TZcnMcm0SAH{pm%UE)rc|vlAK@r+}D2k$&72bOnQQ)&R}~L z5_ejs)Hwe`ZkD}&a|QChmBMO^W|K_nUIUAU_X=Kt811~6??5ND&<=>uhJSA5q?+4v`?;Wr+PEk zOYad?y=Sj4&s}Bjf?S8zNvZqCB3)kDz9@C*zc+5~+~Yx9dj<~N;zW`W@6RZkdfTK~ zkwMg-&NIv4QM=0tnl19hQJ&nM^*8D|aF+<+8b`*&h8d338mOT2E(ZQ`A3r0>ez2OY zq!Y8xg*ASF9X%f(t_RsDhUbk^SEV#ULMwmgaXUA8%U2F3mgy3w0~YCP(F0;JGDVQ1 zg&ZqA>TlsiceR`Iv&#|xS#^NRlJ2>4!^|haLfcSzuCLNnIp>(0`~lo?(mMZl+!?&_ zA4;6?J2bA0f=l9LKY_P@Y_y%zcC(P&l$lIAi2KyxZ#<1Tc#C!87R)rWJkb#+7J&-F z5PY1}9;krjFO%%;Ygz>Rs`%A^K!FX3>Q6clLQ3VGvvc*UUuUcnIwk4r?HcNR7VfJT ziLx;_(%-MGw$KV2l!`zwH3{A=bqJ8)Kb>V&)&vYe9g1X;Q!R5?wSWMU#)iQ5DI~-E zr(oon&0$LvAsKDbayH1P48;%|s})r^Ky|qnYc2n-Xk}clVbCbMvO?->sUS_+NrlEA z5baJ##JobE%fn|Z&lqf{v<1tNe7t6ji1$_U_ZOLTJ8UcYI5}O?1@q2vok^^1taXK1 zeA*?6)v`Mp!_^*fv-WN6%g6kR%yVC222^uXxcR3Z;iQ2=w>Xgr~tZXk7cs z>g~b3?tO(?LmpI8qDT3Yc1+98M%!XQP9@&Rt5>f0Yw>(Ykxv_MFxBVL(gqzNJW9Qk z_o)9K&c$Jl_EX!1D<-h{=%(3MrF@jF5~rC%SnO@7~0e6AT_r2S$8FW<>6pAd@ z8pUDi0(JjDMEi0UrH7zI=&+aM-VUgiG_oLuM!u1_qG#RDEuYHfOY!9m?fLjl$Krj| z%Gsi`;kWFZwI;${7|!GCX9F$t(LEjTT*~T&QEQn-EWS_Z#48a;*McfNX^N~qib^PT z2t?bGDmqnZ45oc#ep0(%D~pd%m(5E=#DDVg8iF7nM(B5Ate>urU} znvB_->dZ1^!#6f)9SMr|S1e9axDWB3#6R4OxZR9M3@tKi-zY^Xd)d#FeXz1vs|LOd zJYDDTpsU3t-KNg+6WUV!I5{ms<;%Do#K7uugz(OG;{*lwnWj>lZ;viZ5}R~>zdf#U z`N4YB?rPZO=smglt9Y@<%AGNz_3wUMTv`RVh1}049?L5odrYQyr%!`sS&J;`KuT4~ z`WDa+rH$FY5?j&$?m+h;h4>USBL8B4@N4wz63tP|>__GK^@rM9$(nH;QD;z7202U4 zdzv1XBNh&-xOKa)Fg4QMPghS&n^6cu0EG01StK^q@@=B~Zl`t}34a~kyIRtl67)88 zU?70|-Qbah5wAd~Z7OoTlZ*cyt%h8!vEFOTT8t z*^S=IUFgmdX?)!_7w;v$-@Zm!<#Z~*>Uyo5e4a14J>WocKM--Zdb91DiGM?P`YFu~ z#hz56U0*$+;-u-<&Y(H(&5Q9ZQy!Ly=clX$Gp>D{YpH-J}uL~M7Kx#6+2p&&OjV!<$ zcr?}I6dAgvN=tnB(H^5tVFo@LswKa2{?_WVyD^b$_+U<*##fGa?;%P*^VBGQtPnOzijLLU}A#C$T|JEK}~EU8B{;FjTP4i_n< z%u=h&d3T^`gpk1ZxQTw-4)sh=OF?^8$ritFgMZ{ehHA0NZHq+P{=&TW_D4@#eJOwg$Zg!sb@71C}V)3=u#g}>K- zY(9-rd~%a~R4Z2leI#Wf$+CX?YLHqGo$WrwWc_ZC<&&>wEzvR|^~I z3ZWLU*HW9!K14;HY|({+TZ&0R2G%DYM_X7tVd&`exMM8&1H>{gvlDC7=4+T_DkHVA zb~yShkv5ROu}ob|lvI81BPgO-6oc)y#3hQGI2L zx1n(Q&mZxhN4woJ(baoAiQj@xXWjEJk#9g|WCoa4xIHQ*6t{`0>i3ad6{KB*;3w&$ zP8Lt>kBn8+`Me@?!kH-)l5)X*uG-OZW^<2M}p@Rt)k>ly=h(7jkRbH;au<02z)(XjqP4hatn@L@--En z4eu%lQu~{;09)2MPbPcYe7}6FcU;HAevv>_{~PHyUAq4K&l9&}-82Zj7Tdpt$_D#abE?h|tXd(EC& zy?IbjQamk$nU4uL6*%;{QUjU3HcH%UYR1YFlSvY(`WHZmap}MSMxWg&$96zB>tPO_ ztw_%su>U?Aw4n`3k8kXqET9@p2m_Yy83>Q3nQK-o|NriK48#1)C*WRB%@ga5eD4?C z?P6xg&CU8GVUF4vgDYp%-N!G~1y}oVLrd6DQ_ftsVUd8WzT&_VU72St^HTt$u4UlD z_q0k698{wR?COUM12L)bK+UYBC3uQPB%`p(tCj=D&l*7(EW+)ZU~M!*b*s#qPk0(H zvpc-G^34`6UDWUAJzq@?I9=x})&H(kH_bP2-oszQfxA7pseR(|aG|Z?$Y?qG%N}np zCk=4ljOOM`+&He{Dj_k$SqLz$dIHeje<*EQq!l~2^;A6*qB##vT^)M0Xk=R?{88Ra zzUO;81{6YK0PFA|Z}t=a_Ov39sU?-yhGTGtbYc0Y;vn_qr;&sr0=iG0C~Lm88M zQ{ZCrwPTNl>no$q*|rJF4+BRgxgahnGQVpKEG4qJ@_ zi|J4+MmlECpCLOI8GOfiZm)4$tRT2T!+J} zW(l!dZlwGr+4jo@ zk9GraT~dP#%+<2d*ZtQ-l3Y1+5q7T4B|vS~q3>7Qe|FWH5OUZFbl-gbjZ7X*Bq>)^ z4BfEkal=%XdqdbrLHZ3WCRVy~XBgPA>zy-RBnbFPo^pLOSOe0JoAvw@L5zkSXVJ~F zgOx=2Q5{Y}6hEx|?_TIrzR^MCM9d49v$k$@a9Ik=0|$F}jlE$DySlS8{aq7L07s=F zir|i^S)Px+)oMrxi^17=ojVl5EZDz=M?J>bQ)C0U95$Vvbry1WnDP>gc{m*++cQ^z z8^tt13lrdn-F7#o%U5QMZ=o`jy0}nuP2cT4L{LOGdv+1B$0BW9_jvCs8YS98Yr@`| zqFV3ge&llR$1&%odgnR@d}~^LqsN`p4|s)i%T_Xf;}{EE2IDSwq371q?(h$YjG{zh z3j}huKiQ4=+7!krro>IuZBQ-nab-~(mh1yLo${BEM_w!RaP)*vB>$8=h0$8xDvv5_ zw`m#nj^<1o5C7b5{5yjw`p~Vl+L^K|SfGiP2#sg$Uc&esTatxm0wnKQTb$%;|7*DX zA)g4FJ20SLw^&H>k~2=MTiG3aJp8{ED`OEdDVtf0a`)Q6%iUmo7v>lK_b9JIS|bF3>k)F)^5+Q219 z$l&AlNj|d=O~cf8Vg|kI%J;6tdtCe)Ho?MufeWnfe5PPS-OjZgqmp)aM)6X{?{EU9 zzwols2?X`p$J_-X6IMG4N;SyLYQZe3zt_O#hziZ)Q4gWSyA*wT zK=$e&wi5eMbDCDAlun@Ab$)u5FMEO@WNpWa+?lom1Da>cT`==;&o;&M=K(W}zT_u~ z{_O;4)h$6SmEgIJ!PTa%lrVZ`0ynB=VdkCTW1J6wUxJx%`3Fa?oNfRgh>qFONL&>e z5g-(We`x=@_-?L{!&K#0bsFur*y6sTmMAU7X6^sugz{2r>4}Q6ZRd#DYhGq1T2v~*f%2m5Knm|=6hMu#8G{=y z!PnHO39{;gAK?$%kM4yhgn%UAIIE$etu^NQ_*=H8?<>P?kVP(lL?t)xspJ6BTEdDm zDm~mXro!t$SofvhfB3j%$<;-8qL2sQW^o^`NtW$iS^SmMJyA|~ z4J*Wasgo;h0jP`M4Op4hmj9V2QchGdbDfyg>hJ&yRY9L@NFG5HXuq~th~>K!W2qEl zu&)ZCn(}|m4E}{1e}y$bJ>B_F&3B*S=DT2u+cIL6!m7RtYlIwvOTfw^pbC{22mN a$GH3LX2qquPT=IhZeG{FR-|S9=sy4+Kon>I diff --git a/docs/faq.md b/docs/faq.md index 0b0397a45..a934df5c2 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -245,7 +245,7 @@ See the docs on all [Merlin Commands](./user_guide/command_line.md) that are ava ### How do I set up a workspace without executing step scripts? -Use [Merlin's Dry Run](./user_guide/command_line.md#dry-run) capability: +Use [Merlin's Dry Run](./user_guide/running_studies.md#dry-runs) capability: === "Locally" @@ -438,7 +438,7 @@ run: procs: 3 ``` -See [The `LAUNCHER` and `VLAUNCHER` Variables](./user_guide/variables.md#the-launcher-and-vlauncher-variables) and the [Scheduler Specific Properties](./user_guide/specification.md#scheduler-specific-properties) sections for more information. +See [The `LAUNCHER` and `VLAUNCHER` Variables](./user_guide/variables.md#the-launcher-and-vlauncher-variables) and [The Run Property](./user_guide/specification.md#the-run-property) sections for more information. ### What is `level_max_dirs`? diff --git a/docs/gen_ref_pages.py b/docs/gen_ref_pages.py index e46c0eb93..dfa24b5c2 100644 --- a/docs/gen_ref_pages.py +++ b/docs/gen_ref_pages.py @@ -6,10 +6,27 @@ nav = mkdocs_gen_files.Nav() -# print(sorted(Path("merlin").rglob("*.py"))) +IGNORE_PATTERNS = [ + Path("merlin/examples/workflows"), + Path("merlin/examples/dev_workflows"), + Path("merlin/data"), + "*/ascii_art.py", +] + + +def should_ignore(path): + """Check if the given path matches any ignore patterns.""" + for pattern in IGNORE_PATTERNS: + pattern = str(pattern) + if path.is_relative_to(Path(pattern)): + return True + if path.match(pattern): + return True + return False + for path in sorted(Path("merlin").rglob("*.py")): - if "merlin/examples" in str(path): + if should_ignore(path): continue module_path = path.relative_to("merlin").with_suffix("") doc_path = path.relative_to("merlin").with_suffix(".md") diff --git a/docs/index.md b/docs/index.md index 46466494f..1abc4936a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -71,7 +71,7 @@ First, let's create a folder to store our server files and our examples. We'll a mkdir merlin_examples ; cd merlin_examples/ ``` -Now let's set up a [containerized server](./user_guide/configuration/merlin_server.md) that Merlin can connect to. +Now let's set up a [containerized server](./user_guide/configuration/containerized_server.md) that Merlin can connect to. 1. Initialize the server files: diff --git a/docs/tutorial/2_installation.md b/docs/tutorial/2_installation.md index 5883a8946..ef818d1d2 100644 --- a/docs/tutorial/2_installation.md +++ b/docs/tutorial/2_installation.md @@ -148,37 +148,19 @@ Configure and Install Singularity: cd singularity-ce-3.9.9 ./mconfig && make -C ./builddir && sudo make -C ./builddir install ``` -## Configuring Merlin +## Configuring and Connecting to a Redis Server -Merlin requires a configuration script for the Celery interface in order to know which server(s) to connect to. Run this configuration method to create the `app.yaml` configuration file. +!!! warning "LC Users" -```bash -merlin config --broker redis -``` - -The `merlin config` command above will create a file called `app.yaml` in the `~/.merlin` directory. If you are running a Redis server locally then you are all set, look in the `~/.merlin/app.yaml` file to see the configuration, it should look like the configuration below. + If you're a Livermore Computing (LC) user, it may be advantageous for you to set your server up through [LaunchIT](https://launchit.llnl.gov/). Instructions for this can be found [here](https://lc.llnl.gov/confluence/display/MERLIN/LaunchIT+Configuration). -???+ abstract "app.yaml" - - ```yaml - broker: - name: redis - server: localhost - port: 6379 - db_num: 0 - - results_backend: - name: redis - server: localhost - port: 6379 - db_num: 0 - ``` +!!! warning "Non-LC Users" -More detailed information on configuring Merlin can be found in the [Configuration](../user_guide/configuration/index.md) page. + If you're not an LC user but you don't want to use a singularity container, you can host your own servers and point Merlin to them. Instructions for this can be found [here](../user_guide/configuration/external_server.md). -## Checking/Verifying Installation +Merlin requires a configuration file for the Celery interface in order to know which server(s) to connect to. For the purposes of this tutorial we'll spin up a Redis server in a Singularity container using the [`merlin server`](../user_guide/command_line.md#server-merlin-server) command and then connect to it. More detailed information on configuring Merlin with other server options can be found in the [Configuration](../user_guide/configuration/index.md) page. -First launch the Merlin server containers by using the `merlin server` commands. +### Initializing the Server Initialize the server files: @@ -190,6 +172,7 @@ This will create a `merlin_server/` folder in the current run directory. The str ```bash merlin_server/ +|-- app.yaml |-- redis.conf |-- redis.pass |-- redis.users @@ -198,10 +181,11 @@ merlin_server/ The files in this folder are: -1. `redis.conf`: The Redis configuration file that contains all of the settings to be used for our Redis server -2. `redis.pass`: A password for the Redis server that we'll start up next -3. `redis.users`: A file defining the users that are allowed to access the Redis server and their permissions -4. `redis_latest.sif`: A singularity file that contains the latest Redis docker image that was pulled behind the scenes by Merlin +1. `app.yaml`: The configuration file that Merlin will eventually read from +2. `redis.conf`: The Redis configuration file that contains all of the settings to be used for our Redis server +3. `redis.pass`: A password for the Redis server that we'll start up next +4. `redis.users`: A file defining the users that are allowed to access the Redis server and their permissions +5. `redis_latest.sif`: A singularity file that contains the latest Redis docker image that was pulled behind the scenes by Merlin If you'd like to modify the configuration of your server, you can either modify the files directly or use: @@ -209,42 +193,59 @@ If you'd like to modify the configuration of your server, you can either modify merlin server config ``` +!!! note + + Since Merlin servers are created locally on your run directory you are allowed to create multiple instances of Merlin server with their unique configurations for different studies. Simply create different directories for each study and run the following command in each directory to create an instance for each: + + ```bash + merlin server init + ``` + +### Starting the Server + Now that we have the necessary server files initialized, start the server: ```bash merlin server start ``` -With this command, the containerized server should now be started. Notice that two new files were added to the `merlin_server` folder: +With this command, the containerized server should now be started. Notice that a new file was added to the `merlin_server` folder: `merlin_server.pf`. This is a process file containing information regarding the Redis process. Additionally, the `merlin_server/app.yaml` file is updated to add `broker` and `results_backend` sections that point to the server that was just started. -1. `merlin_server.pf`: A process file containing information regarding the Redis process -2. `app.yaml`: A new `app.yaml` file configured specifically for the containerized Redis server that we just started +### Pointing Merlin to the Server -To have Merlin read this configuration, copy it to your current run directory: +To have Merlin read the `app.yaml` configuration that was generated by starting the server, we have three options: -```bash -cp merlin_server/app.yaml . -``` +1. [RECOMMENDED] Utilize the `merlin config use` command to point Merlin to this file. -You can also make this server container your main server configuration by replacing the one located in your home directory. Make sure you make back-ups of your current `app.yaml` file in case you want to use your previous configurations. + ```bash + merlin config use merlin_server/app.yaml + ``` -```bash -mv ~/.merlin/app.yaml ~/.merlin/app.yaml.bak -``` +2. Copy the `app.yaml` file to the current working directory. -```bash -cp ./merlin_server/app.yaml ~/.merlin/ -``` + !!! warning -!!! note + If you change your current working directory you will no longer be able to connect to this server. - Since Merlin servers are created locally on your run directory you are allowed to create multiple instances of Merlin server with their unique configurations for different studies. Simply create different directories for each study and run the following command in each directory to create an instance for each: + ```bash + cp merlin_server/app.yaml . + ``` + +3. Copy the `app.yaml` file to the `~/.merlin/` directory. + + !!! warning + + If you've configured Merlin in the past, make sure you have back-ups of your current `app.yaml` file in case you want to use your previous configurations. + + ```bash + mv ~/.merlin/app.yaml ~/.merlin/app.yaml.bak + ``` ```bash - merlin server init + cp ./merlin_server/app.yaml ~/.merlin/ ``` -The `merlin info` command will check that the configuration file is installed correctly, display the server configuration strings, and check server access. +Once we've pointed Merlin to the `app.yaml` file for our server, we can now check that Merlin is able to connect to the server. The `merlin info` command will check that the configuration file is installed correctly, display the server configuration strings, and check server access. ```bash merlin info @@ -303,6 +304,15 @@ If everything is set up correctly, you should see: "echo $PYTHONPATH" ``` +Specifically, we are looking for `OK` messages in the `Checking server connections` section: + +```bash +Checking server connections: +---------------------------- +broker server connection: OK +results server connection: OK +``` + ## Docker Advanced Installation (Optional) This optional section details the setup of a RabbitMQ server and a Redis TLS (Transport Layer Security) server for Merlin. For this section, we'll start with the following `docker-compose.yml` file: @@ -404,13 +414,13 @@ The RabbitMQ docker service can be added to the previous `docker-compose.yml` fi ``` -When running the RabbitMQ broker server, the config can be created with the default `merlin config` command. If you have already run the previous command then remove the `~/.merlin/app.yaml` or `~/merlinu/.merlin/app.yaml` file , and run the `merlin config` command again. +When running the RabbitMQ broker server, the config can be created with the default `merlin config create` command. If you have already run the previous command then you can create a new configuration with a different filename using the `-o` flag. You can make sure you're using the new configuration with: ```bash -merlin config +merlin config use /path/to/new_config.yaml ``` -The `app.yaml` file will need to be edited to add the RabbitMQ settings in the broker section of the `app.yaml` file. The `server:` should be changed to `my-rabbit`. The RabbitMQ server will be accessed on the default TLS port, 5671. +The configuration file will need to be edited to add the RabbitMQ settings in the broker section. The `server:` should be changed to `my-rabbit`. The RabbitMQ server will be accessed on the default TLS port, 5671. ???+ abstract "RabbitMQ app.yaml" @@ -513,4 +523,4 @@ The `rabbitmq.conf` file contains the configuration, including ssl, for the Rabb ssl.options.fail_if_no_peer_cert = false ``` -Once this docker-compose file is run, the Merlin `app.yaml` file is changed to use the Redis TLS server `rediss` instead of `redis`. +Once this docker-compose file is run, the Merlin configuration file is changed to use the Redis TLS server `rediss` instead of `redis`. diff --git a/docs/tutorial/3_hello_world.md b/docs/tutorial/3_hello_world.md index e5ac6802a..a77642b5f 100644 --- a/docs/tutorial/3_hello_world.md +++ b/docs/tutorial/3_hello_world.md @@ -222,7 +222,7 @@ A lot of stuff, right? Here's what it means: !!! warning "Important Note" - Before trying this, make sure you've properly set up your Merlin config file `app.yaml`. If you can run `merlin info` and see no errors you should be good to go. Otherwise, see either the [Configuring Merlin](./2_installation.md#configuring-merlin) section of the installation step in the Tutorial or the [Configuration](../user_guide/configuration/index.md) page for more information. + Before trying this, make sure you've properly set up your Merlin config file. If you can run `merlin info` and see no errors you should be good to go. Otherwise, see either the [Configuring Merlin](./2_installation.md#configuring-and-connecting-to-a-redis-server) section of the installation step in the Tutorial or the [Configuration](../user_guide/configuration/index.md) page for more information. Now we will run the same workflow, but in parallel on our task server: diff --git a/docs/tutorial/4_run_simulation.md b/docs/tutorial/4_run_simulation.md index 7d1287c08..79c981f1d 100644 --- a/docs/tutorial/4_run_simulation.md +++ b/docs/tutorial/4_run_simulation.md @@ -47,7 +47,7 @@ Check that the virtual environment with Merlin installed is activated and that y merlin info ``` -This is covered more in depth in the [Verifying Installation](./2_installation.md#checkingverifying-installation) section of the Installation module and at the [Configuration](../user_guide/configuration/index.md) page. +This is covered more in depth in the [Pointing Merlin to the Server](./2_installation.md#pointing-merlin-to-the-server) section of the Installation module and at the [Configuration](../user_guide/configuration/index.md) page. There are a few ways to do this example, including with singularity and with docker. To go through the version with singularity, get the necessary files for this module by running: diff --git a/docs/user_guide/celery.md b/docs/user_guide/celery.md index cc2d67953..9c97aab32 100644 --- a/docs/user_guide/celery.md +++ b/docs/user_guide/celery.md @@ -6,7 +6,7 @@ Merlin queues tasks to the broker which receives and routes tasks. Merlin by def Celery has many functions, it defines the interface to the task broker, the backend results database and the workers that will run the tasks. -As discussed in the [Configuration](./configuration/index.md) page, the broker and backend are configured through [the app.yaml file](./configuration/index.md#the-appyaml-file). A configuration for the rabbit AMQP server is shown below. +As discussed in the [Configuration](./configuration/index.md) page, the broker and backend are configured through [the configuration file](./configuration/index.md#the-configuration-file). A configuration for the rabbit AMQP server is shown below. ???+ abstract "Config File for RabbitMQ Broker and Redis Backend" diff --git a/docs/user_guide/command_line.md b/docs/user_guide/command_line.md index 30faee9ce..8072372cf 100644 --- a/docs/user_guide/command_line.md +++ b/docs/user_guide/command_line.md @@ -28,29 +28,54 @@ See the [Configuration Commands](#configuration-commands), [Workflow Management Since running Merlin in a distributed manner requires the [configuration](./configuration/index.md) of a centralized server, Merlin comes equipped with three commands to help users get this set up: -- *[config](#config-merlin-config)*: Create the skeleton `app.yaml` file needed for configuration +- *[config](#config-merlin-config)*: Create, update, or select a configuration file - *[info](#info-merlin-info)*: Ensure stable connections to the server(s) - *[server](#server-merlin-server)*: Spin up containerized servers ### Config (`merlin config`) -Create a default [config (app.yaml) file](./configuration/index.md#the-appyaml-file) in the `${HOME}/.merlin` directory using the `config` command. This file can then be edited for your system configuration. +Create, update, or select a [configuration file](./configuration/index.md#the-configuration-file) that Merlin will use to connect to your server(s). -See more information on how to set this file up at the [Configuration](./configuration/index.md) page. +Merlin config has a list of commands for interacting with configuration files. These commands allow the user to create and update configuration files, and select which one should be the active configuration. + +See more information on how to set up the configuration file at the [Configuration](./configuration/index.md) page. **Usage:** ```bash -merlin config [OPTIONS] +merlin config [OPTIONS] COMMAND [ARGS] ... ``` **Options:** +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +**Commands:** + +| Name | Description | +| ------------ | ----------- | +| [create](#config-create-merlin-config-create) | Create a template configuration file | +| [update-backend](#config-update-backend-merlin-config-update-backend) | Update broker settings in the configuration file | +| [update-broker](#config-update-broker-merlin-config-update-broker) | Update backend settings in the configuration file | +| [use](#config-use-merlin-config-use) | Use a different configuration setup | + +#### Config Create (`merlin config create`) + +The `merlin config create` command creates a template [configuration file](./configuration/index.md#the-configuration-file) that you can customize to connect to your central server. Detailed instructions for completing this template are available in the [Configuring the Broker and Results Backend](./configuration/index.md#configuring-the-broker-and-results-backend) guide. + +By default, the generated configuration file is saved at `$(HOME)/.merlin/app.yaml`. If you prefer to rename the file or save it to a different location, you can use the `-o` option to specify the desired path. Note that the file must have the `.yaml` extension. + +The default configuration sets the broker to use a RabbitMQ server and the results backend to Redis. While the Redis results backend is mandatory, the broker can be configured to use either RabbitMQ or Redis. To switch to a Redis broker, use the `--broker` option. + +**Options:** + | Name | Type | Description | Default | | ------------ | ------- | ----------- | ------- | | `-h`, `--help` | boolean | Show this help message and exit | `False` | | `--task_server` | string | Select the appropriate configuration for the given task server. Currently only "celery" is implemented. | "celery" | -| `-o`, `--output_dir` | path | Output the configuration in the given directory. This file can then be edited and copied into `${HOME}/.merlin`. | None | +| `-o`, `--output-file` | path | Optional yaml file name for your configuration. Default: $(HOME)/.merlin/app.yaml. | None | | `--broker` | string | Write the initial `app.yaml` config file for either a `rabbitmq` or `redis` broker. The default is `rabbitmq`. The backend will be `redis` in both cases. The redis backend in the `rabbitmq` config shows the use on encryption for the backend. | "rabbitmq" | **Examples:** @@ -58,19 +83,179 @@ merlin config [OPTIONS] !!! example "Create an `app.yaml` File at `~/.merlin`" ```bash - merlin config + merlin config create + ``` + +!!! example "Create a Configuration File at a Custom Path" + + ```bash + merlin config create -o /Documents/configuration/merlin_config.yaml + ``` + +!!! example "Create a Configuration File With a Redis Broker" + + ```bash + merlin config create --broker redis + ``` + +#### Config Update-Backend (`merlin config update-backend`) + +The `merlin config update-backend` command allows you to modify the [`results_backend`](./configuration/index.md#what-is-a-results-backend) section of your configuration file directly from the command line. See the options table below to see exactly what settings can be set. + +**Usage:** + +```bash +merlin config update-backend -t {redis} [OPTIONS] ... +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | +| `-t`, `--type` | choice(`redis`) | Type of results backend to configure. | None | +| `--cf`, `--config-file` | string | Path to the config file that will be updated. | `$(HOME)/.merlin/app.yaml` | +| `-u`, `--username` | string | The backend username. | None | +| `--pf`, `--password-file` | string | Path to a password file that contains the password to the backend. | None | +| `-s`, `--server` | string | The URL of the backend server. | None | +| `-p`, `--port` | int | The port number that this backend server is using. | None | +| `-d`, `--db-num` | int | The backend database number. | None | +| `-c`, `--cert-reqs` | string | Backend cert requirements. | None | +| `-e`, `--encryption-key` | string | Path to the encryption key file. | None | + +**Examples:** + +!!! example "Update Every Setting Required for Redis" + + ```bash + merlin config update-backend -t redis --pf ~/.merlin/redis.pass -s my-redis-server.llnl.gov -p 6379 -d 0 -c none + ``` + + This will create the following `results_backend` section in your `app.yaml` file: + + ```yaml + results_backend: + cert_reqs: none + db_num: 0 + encryption_key: ~/.merlin/encrypt_data_key + name: rediss + password: ~/.merlin/redis.pass + port: 6379 + server: my-redis-server.llnl.gov + username: '' + ``` + +!!! example "Update Just the Port" + + ```bash + merlin config update-backend -t redis -p 6379 + ``` + +!!! example "Update a Custom Configuration File Path" + + ```bash + merlin config update-backend -t redis --cf /path/to/custom_config.yaml -s new-server.gov + ``` + +#### Config Update-Broker (`merlin config update-broker`) + +The `merlin config update-broker` command allows you to modify the [`broker`](./configuration/index.md#what-is-a-broker) section of your configuration file directly from the command line. See the options table below to see exactly what settings can be set. + +**Usage:** + +```bash +merlin config update-broker -t {rabbitmq,redis} [OPTIONS] ... +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | +| `-t`, `--type` | choice(`rabbitmq` \| `redis`) | Type of broker to configure. | None | +| `--cf`, `--config-file` | string | Path to the config file that will be updated. | `$(HOME)/.merlin/app.yaml` | +| `-u`, `--username` | string | The broker username (only for `rabbitmq` broker). | None | +| `--pf`, `--password-file` | string | Path to a password file that contains the password to the broker. | None | +| `-s`, `--server` | string | The URL of the broker server. | None | +| `-p`, `--port` | int | The port number that this broker server is using. | None | +| `-v`, `--vhost` | string | The vhost for the broker (only for `rabbitmq` broker). | None | +| `-d`, `--db-num` | int | The backend database number (only for `redis` broker). | None | +| `-c`, `--cert-reqs` | string | Backend cert requirements. | None | + +**Examples:** + +!!! example "Update Every Setting Required for a Redis Broker" + + ```bash + merlin config update-broker -t redis --pf ~/.merlin/redis.pass -s my-redis-server.llnl.gov -p 6379 -d 0 -c none + ``` + + This will create the following `broker` section in your `app.yaml` file: + + ```yaml + broker: + cert_reqs: none + db_num: 0 + name: rediss + password: ~/.merlin/redis.pass + port: 6379 + server: my-redis-server.llnl.gov + username: '' ``` -!!! example "Create an `app.yaml` File at a Custom Path" +!!! example "Update Every Setting Required for a RabbitMQ Broker" ```bash - merlin config -o /Documents/configuration/ + merlin config update-broker -t rabbitmq -u my_rabbit_username --pf ~/.merlin/rabbit.pass -s my-rabbitmq-server.llnl.gov -p 5672 -v host4rabbit -c none ``` -!!! example "Create an `app.yaml` File With a Redis Broker" + This will create the following `broker` section in your `app.yaml` file: + + ```yaml + broker: + cert_reqs: none + name: rabbitmq + password: ~/.merlin/rabbit.pass + port: 5672 + server: my-rabbitmq-server.llnl.gov + username: my_rabbit_username + vhost: host4rabbit + ``` + +!!! example "Update Just the Username" ```bash - merlin config --broker redis + merlin config update-broker -t rabbitmq -u my_new_username + ``` + +!!! example "Update a Custom Configuration File Path" + + ```bash + merlin config update-broker -t redis --cf /path/to/custom_config.yaml -s new-server.gov + ``` + +#### Config Use (`merlin config use`) + +The `merlin config use` command allows you to switch which configuration file to use as your active configuration. + +**Usage:** + +```bash +merlin config use [OPTIONS] CONFIG_FILE +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +**Examples:** + +!!! example "Use a Custom Configuration" + + ```bash + merlin config use /path/to/custom_config.yaml ``` ### Info (`merlin info`) @@ -95,7 +280,7 @@ Create a local containerized server for Merlin to connect to. Merlin server crea Merlin server has a list of commands for interacting with the broker and results server. These commands allow the user to manage and monitor the exisiting server and create instances of servers if needed. -More information on configuring with Merlin server can be found at the [Merlin Server Configuration](./configuration/merlin_server.md) page. +More information on configuring with Merlin server can be found at the [Containerized Server Configuration](./configuration/containerized_server.md) page. **Usage:** @@ -263,6 +448,7 @@ merlin server config [OPTIONS] The Merlin library provides several commands for setting up and managing your Merlin workflow: +- *[database](#database-merlin-database)*: Interact with Merlin's backend database - *[example](#example-merlin-example)*: Download pre-made workflow specifications that can be modified for your own workflow needs - *[purge](#purge-merlin-purge)*: Clear any tasks that are currently living in the central server - *[restart](#restart-merlin-restart)*: Restart a workflow @@ -270,6 +456,411 @@ The Merlin library provides several commands for setting up and managing your Me - *[run workers](#run-workers-merlin-run-workers)*: Start up workers that will execute the tasks that exist on the central server - *[stop workers](#stop-workers-merlin-stop-workers)*: Stop existing workers +### Database (`merlin database`) + +This command allows you to interact with Merlin's backend database by viewing database info, retrieving and printing entries, and deleting entries. If you ran your study locally, use the `--local` option here as well when running database commands. + +More information on this command can be found below or at [The Database Command](./database/database_cmd.md) page. See [Merlin's Database](./database/index.md) for more general information on the database itself. + + + + +**Usage:** + +``` +merlin database [OPTIONS] COMMAND ... +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | +| `-l`, `--local` | boolean | Use the local SQLite database for this command. | `False` | + +**Commands:** + +| Name | Description | +| ------------ | ----------- | +| [info](#database-info-merlin-database-info) | Print general information about the database | +| [get](#database-get-merlin-database-get) | Retrieve and print entries from the database | +| [delete](#database-delete-merlin-database-delete) | Delete entries from the database | + +#### Database Info (`merlin database info`) + +The `info` subcommand prints general information about the database, including the database type, version, and brief details about the existing entries. + +**Usage:** + +```bash +merlin database info [OPTIONS] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +#### Database Get (`merlin database get`) + +The `get` subcommand allows users to retrieve entries from the database and print them to the console. + +**Usage:** + +```bash +merlin database get [OPTIONS] SUBCOMMAND ... +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +**Subcommands:** + +| Name | Description | +| ------------ | ----------- | +| [study](#get-study-merlin-database-get-study) | Retrieve and print specific study(ies) from the database | +| [run](#get-run-merlin-database-get-run) | Retrieve and print specific run(s) from the database | +| [logical-worker](#get-logical-worker-merlin-database-get-logical-worker) | Retrieve and print specific logical worker(s) from the database | +| [physical-worker](#get-physical-worker-merlin-database-get-physical-worker) | Retrieve and print specific physical worker(s) from the database | +| [all-studies](#get-all-studies-merlin-database-get-all-studies) | Retrieve and print all studies from the database | +| [all-runs](#get-all-runs-merlin-database-get-all-runs) | Retrieve and print all runs from the database | +| [all-logical-workers](#get-all-logical-workers-merlin-database-get-all-logical-workers) | Retrieve and print all logical workers from the database | +| [all-physical-workers](#get-all-physical-workers-merlin-database-get-all-physical-workers) | Retrieve and print all physical workers from the database | +| [everything](#get-everything-merlin-database-get-everything) | Retrieve and print every entry from the database | + +##### Get Study (`merlin database get study`) + +The `get study` subcommand allows users to retrieve specific study entries from the database by study ID or name and print them to the console. + +**Usage:** + +```bash +merlin database get study [OPTIONS] STUDY_ID_OR_NAME [STUDY_ID_OR_NAME ...] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +##### Get Run (`merlin database get run`) + +The `get run` subcommand allows users to retrieve specific run entries from the database by run ID or workspace and print them to the console. + +**Usage:** + +```bash +merlin database get run [OPTIONS] RUN_ID_OR_WORKSPACE [RUN_ID_OR_WORKSPACE ...] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +##### Get Logical-Worker (`merlin database get logical-worker`) + +The `get logical-worker` subcommand allows users to retrieve specific logical-worker entries from the database by ID and print them to the console. + +**Usage:** + +```bash +merlin database get logical-worker [OPTIONS] LOGICAL_WORKER_ID [LOGICAL_WORKER_ID ...] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +##### Get Physical-Worker (`merlin database get physical-worker`) + +The `get physical-worker` subcommand allows users to retrieve specific physical-worker entries from the database by ID or name and print them to the console. + +**Usage:** + +```bash +merlin database get physical-worker [OPTIONS] PHYSICAL_WORKER_ID_OR_NAME [PHYSICAL_WORKER_ID_OR_NAME ...] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +##### Get All-Studies (`merlin database get all-studies`) + +The `get all-studies` subcommand allows users to retrieve all study entries from the database and print them to the console. + +**Usage:** + +```bash +merlin database get all-studies [OPTIONS] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +##### Get All-Runs (`merlin database get all-runs`) + +The `get all-runs` subcommand allows users to retrieve all run entries from the database and print them to the console. + +**Usage:** + +```bash +merlin database get all-runs [OPTIONS] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +##### Get All-Logical-Workers (`merlin database get all-logical-workers`) + +The `get all-logical-workers` subcommand allows users to retrieve all logical-worker entries from the database and print them to the console. + +**Usage:** + +```bash +merlin database get all-logical-workers [OPTIONS] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +##### Get All-Physical-Workers (`merlin database get all-physical-workers`) + +The `get all-physical-workers` subcommand allows users to retrieve all physical-worker entries from the database and print them to the console. + +**Usage:** + +```bash +merlin database get all-physical-workers [OPTIONS] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +##### Get Everything (`merlin database get everything`) + +The `get everything` subcommand allows users to retrieve every entry from the database and print them all to the console. + +**Usage:** + +```bash +merlin database get everything [OPTIONS] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +#### Database Delete (`merlin database delete`) + +The `delete` subcommand allows users to delete entries from the database. + +**Usage:** + +```bash +merlin database delete [OPTIONS] SUBCOMMAND ... +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +**Subcommands:** + +| Name | Description | +| ------------ | ----------- | +| [study](#delete-study-merlin-database-delete-study) | Delete specific study(ies) from the database | +| [run](#delete-run-merlin-database-delete-run) | Delete specific run(s) from the database | +| [logical-worker](#delete-logical-worker-merlin-database-delete-logical-worker) | Delete specific logical worker(s) from the database | +| [physical-worker](#delete-physical-worker-merlin-database-delete-physical-worker) | Delete specific physical worker(s) from the database | +| [all-studies](#delete-all-studies-merlin-database-delete-all-studies) | Delete all studies from the database | +| [all-runs](#delete-all-runs-merlin-database-delete-all-runs) | Delete all runs from the database | +| [all-logical-workers](#delete-all-logical-workers-merlin-database-delete-all-logical-workers) | Delete all logical workers from the database | +| [all-physical-workers](#delete-all-physical-workers-merlin-database-delete-all-physical-workers) | Delete all physical workers from the database | +| [everything](#delete-everything-merlin-database-delete-everything) | Delete everything from the database | + +##### Delete Study (`merlin database delete study`) + +!!! warning + + By default, this command will also delete all of the runs associated with a study. To disable this, use the `-k` option mentioned in the table below. + +The `delete study` subcommand allows users to delete specific study entries from the database by ID or name. + +**Usage:** + +```bash +merlin database delete study [OPTIONS] STUDY_ID_OR_NAME [STUDY_ID_OR_NAME ...] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | +| `-k`, `--keep-associated-runs` | boolean | Keep runs associated with the study that's being deleted | `False` | + +##### Delete Run (`merlin database delete run`) + +The `delete run` subcommand allows users to delete specific run entries from the database by ID or workspace. + +**Usage:** + +```bash +merlin database delete run [OPTIONS] RUN_ID_OR_WORKSPACE [RUN_ID_OR_WORKSPACE ...] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +##### Delete Logical-Worker (`merlin database delete logical-worker`) + +The `delete logical-worker` subcommand allows users to delete specific logical worker entries from the database by ID. + +**Usage:** + +```bash +merlin database delete logical-worker [OPTIONS] LOGICAL_WORKER_ID [LOGICAL_WORKER_ID ...] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +##### Delete Physical-Worker (`merlin database delete physical-worker`) + +The `delete physical-worker` subcommand allows users to delete specific physical worker entries from the database by ID or name. + +**Usage:** + +```bash +merlin database delete physical-worker [OPTIONS] PHYSICAL_WORKER_ID_OR_NAME [PHYSICAL_WORKER_ID_OR_NAME ...] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +##### Delete All-Studies (`merlin database delete all-studies`) + +!!! warning + + This command will also delete all of the runs in the database. + +The `delete all-studies` subcommand allows users to delete all study entries from the database. + +**Usage:** + +```bash +merlin database delete all-studies [OPTIONS] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | +| `-k`, `--keep-associated-runs` | boolean | Keep runs associated with the studies | `False` | + +##### Delete All-Runs (`merlin database delete all-runs`) + +The `delete all-runs` subcommand allows users to delete all run entries from the database. + +**Usage:** + +```bash +merlin database delete all-runs [OPTIONS] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +##### Delete All-Logical-Workers (`merlin database delete all-logical-workers`) + +The `delete all-logical-workers` subcommand allows users to delete all logical worker entries from the database. + +**Usage:** + +```bash +merlin database delete all-logical-workers [OPTIONS] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +##### Delete All-Physical-Workers (`merlin database delete all-physical-workers`) + +The `delete all-physical-workers` subcommand allows users to delete all physical worker entries from the database. + +**Usage:** + +```bash +merlin database delete all-physical-workers [OPTIONS] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | + +##### Delete Everything (`merlin database delete everything`) + +The `delete everything` subcommand allows users to delete every entry from the database. + +**Usage:** + +```bash +merlin database delete everything [OPTIONS] +``` + +**Options:** + +| Name | Type | Description | Default | +| ------------ | ------- | ----------- | ------- | +| `-h`, `--help` | boolean | Show this help message and exit | `False` | +| `-f`, `--force` | boolean | Delete everything in the database without confirmation | `False` | + ### Example (`merlin example`) If you want to obtain an example workflow, use Merlin's `merlin example` command. First, view all of the example workflows that are available with: diff --git a/docs/user_guide/configuration/containerized_server.md b/docs/user_guide/configuration/containerized_server.md index 218e97f23..de731886e 100644 --- a/docs/user_guide/configuration/containerized_server.md +++ b/docs/user_guide/configuration/containerized_server.md @@ -39,6 +39,7 @@ The main configuration in `~/.merlin/server/` deals with defaults and technical In addition to the main server configuration, a local server configuration will be created in your current working directory in a folder called `merlin_server/`. This directory will contain: +- `app.yaml`: The configuration file that Merlin will eventually read from - `redis.conf`: The Redis configuration file that contains all of the settings to be used for our Redis server - `redis.pass`: A password for the Redis server that we'll start up next - `redis.users`: A file defining the users that are allowed to access the Redis server and their permissions @@ -76,31 +77,24 @@ You can check that the server was started properly with: merlin server status ``` -The `merlin server start` command will add new files to the local configuration `merlin_server/` folder: - -- `merlin_server.pf`: A process file containing information regarding the Redis process -- `app.yaml`: A new app.yaml file configured specifically for the containerized Redis server that we just started +The `merlin server start` command will add a new file to the local configuration `merlin_server/` folder: `merlin_server.pf`. This is a process file containing information regarding the Redis process. Additionally, the `merlin server start` command will update the `merlin_server/app.yaml` file that was created when we ran `merlin server init`, so that it has `broker` and `results_backend` sections that point to our started server. To have Merlin read this server configuration: -=== "Copy Configuration to CWD" - - ```bash - cp merlin_server/app.yaml . - ``` - === "Make This Server Configuration Your Main Configuration" - If you're going to use the server configuration as your main configuration, it's a good idea to make a backup of your current server configuration (if you have one): + !!! Tip + + The `merlin config use` command allows you to have multiple server configuration files that you can easily swap between. ```bash - mv ~/.merlin/app.yaml ~/.merlin/app.yaml.bak + merlin config use merlin_server/app.yaml ``` - From here, simply copy the server configuration to your `~/.merlin/` folder: +=== "Copy Configuration to CWD" ```bash - cp merlin_server/app.yaml ~/.merlin/app.yaml + cp merlin_server/app.yaml . ``` You can check that Merlin recognizes the containerized server connection with: @@ -124,15 +118,17 @@ If your servers are running and set up properly, this should output something si ~~~~~~~~~~ |_| |_|\___|_| |_|_|_| |_| *~~~~~~~~~~~ ~~~*~~~* Machine Learning for HPC Workflows + + v. 1.13.0 - + [2025-05-28 11:57:14: INFO] Reading app config from file merlin_server/app.yaml Merlin Configuration ------------------------- - config_file | /path/to/app.yaml + config_file | merlin_server/app.yaml is_debug | False - merlin_home | /path/to/.merlin + merlin_home | ~/.merlin merlin_home_exists | True broker server | redis://default:******@127.0.0.1:6379/0 broker ssl | False @@ -147,19 +143,20 @@ If your servers are running and set up properly, this should output something si Python Configuration ------------------------- - $ which python3 - /path/to/python3 - - $ python3 --version - Python x.y.z - - $ which pip3 - /path/to/pip3 + Python Packages - $ pip3 --version - pip x.y.x from /path/to/pip (python x.y) + Package Version Location + --------- --------- ---------------------------------------------------------------------- + python 3.13.2 /path/to/python3 + pip 25.0.1 /path/to/python3.13/site-packages + merlin 1.12.2 /path/to/merlin + maestrowf 1.1.10 /path/to/python3.13/site-packages + celery 5.5.2 /path/to/python3.13/site-packages + kombu 5.5.3 /path/to/python3.13/site-packages + amqp 5.3.1 /path/to/python3.13/site-packages + redis 5.2.1 /path/to/python3.13/site-packages - "echo $PYTHONPATH" + $PYTHONPATH: ``` ## Stopping the Server @@ -302,12 +299,15 @@ if __name__ == "__main__": Finally, we'll add a function `update_app_yaml` to do the actual updating of the `app.yaml` file. This function will load in the current contents of the `app.yaml` file, update the necessary `server` settings, and dump the updated settings back to the `app.yaml` file. -```python title="update_app_hostname.py" hl_lines="2 6-29 39" +```python title="update_app_hostname.py" hl_lines="2-3 7-32 42" import argparse +import logging import yaml from merlin.utils import verify_filepath +LOG = logging.getLogger(__name__) + def update_app_yaml(hostname, app_yaml): """ Read in the app.yaml contents, update them, then write the updated @@ -368,7 +368,14 @@ To accomplish this, we'll use the `hostname` command to obtain the name of the h python update_app_hostname.py `hostname` ${APP_YAML_PATH} ``` -When Merlin reads in the `app.yaml` file, it will search for this file in two locations: your current working directory and `~/.merlin`. In order for Merlin to read in this `app.yaml` file that we just updated, we need to copy it to the directory where you'll launch your study from (AKA the current working directory): +Now let's make sure Merlin is pointing to this `app.yaml` file: + +```bash title="server.sbatch" linenums="55" +# Tell Merlin to use this app.yaml file +merlin config use ${APP_YAML_PATH} +``` + + Finally, let's add in a statement to see if our server is connected properly (this will help with debugging) and a call to sleep forever so that this server stays up and running until our allocation terminates: @@ -456,9 +463,8 @@ Below are the full scripts: # Update the app.yaml file generated by merlin server start to point to the hostname of this node python update_app_hostname.py `hostname` ${APP_YAML_PATH} - # Move the app.yaml to the project directory - PROJECT_DIR=`pwd` - cp ${MERLIN_SERVER_DIR}/app.yaml ${PROJECT_DIR} + # Tell Merlin to use this app.yaml file + merlin config use ${APP_YAML_PATH} # Check the server connection merlin info @@ -471,10 +477,13 @@ Below are the full scripts: ```python title="update_app_hostname.py" import argparse + import logging import yaml from merlin.utils import verify_filepath + LOG = logging.getLogger(__name__) + def update_app_yaml(hostname, app_yaml): """ Read in the app.yaml contents, update them, then write the updated @@ -519,12 +528,12 @@ Using the scripts is as easy as: 2. Updating the `VENV` variable in `servers.sbatch` to point to your venv with Merlin installed 3. Starting the server by submitting the script with `sbatch server.sbatch` -Once your allocation is granted, the server should spin up. You can check that it's been started by executing `merlin info` from the directory where these scripts exist. This should output a message like is shown at the end of [Starting the Server and Linking it to Merlin](#starting-the-server-and-linking-it-to-merlin). +Once your allocation is granted, the server should spin up. You can check that it's been started by executing `merlin info`. This should output a message like is shown at the end of [Starting the Server and Linking it to Merlin](#starting-the-server-and-linking-it-to-merlin). Specifically, you should see that it's pointing to the server we just spun up. From here, you should be able to start your workers by submitting a `workers.sbatch` script like is shown in [Distributed Runs](../running_studies.md#distributed-runs). To ensure that this script doesn't start prior to your server spinning up, you should submit this script with: ```bash -sbatch -d after:+1 workers.sbatch +sbatch -d after: workers.sbatch ``` This will make it so that workers.sbatch cannot start until the server job has been running for 1 minute. @@ -601,7 +610,7 @@ From here, all we need to do is: 2. Start the workers with: ```bash - sbatch -d after:+1 workers.sbatch + sbatch -d after: workers.sbatch ``` 3. Wait for the server to start (step 1), then queue the tasks with: @@ -616,6 +625,6 @@ You can check that everything ran properly with: merlin status hello_samples.yaml ``` -Or, if you're using a version of Merlin prior to v1.12.0, you can ensure that the `hello_samples_/` output workspace was created. More info on the expected output can be found in [the Hello World Examples page](../../examples/hello.md#expected-output-1). +Or, if you're using a version of Merlin prior to v1.12.0, you can ensure that the `hello_samples_/` output workspace was created. More info on the expected output can be found in [the Hello World Examples page](../../examples/hello.md#expected-output_1). Congratulations, you just ran a cross-node workflow with a containerized server! diff --git a/docs/user_guide/configuration/external_server.md b/docs/user_guide/configuration/external_server.md index 9b86e3512..40a411c8e 100644 --- a/docs/user_guide/configuration/external_server.md +++ b/docs/user_guide/configuration/external_server.md @@ -284,7 +284,7 @@ Once your server is set up, we'll need six keys in the `results_backend` section 3. `server`: A URL to the server that you're connecting 4. `port`: The port that your server is running on. Default is 6379. 5. `db_num`: The database index (this will likely be 0). -6. `encryption_key`: The path to the encryption key (this is automatically generated by `merlin config`) +6. `encryption_key`: The path to the encryption key (this is automatically generated by [`merlin config create`](../command_line.md#config-create-merlin-config-create)) Using these settings, Merlin will construct a connection string of the form: diff --git a/docs/user_guide/configuration/index.md b/docs/user_guide/configuration/index.md index c6dbcbb27..0e5003352 100644 --- a/docs/user_guide/configuration/index.md +++ b/docs/user_guide/configuration/index.md @@ -30,16 +30,22 @@ The results backend enables the asynchronous nature of Celery. Instead of blocki See the [Configuring the Broker and Results Backend](#configuring-the-broker-and-results-backend) section below for more information on configuring your results backend. -## The app.yaml File +## The Configuration File -In order to read in configuration options for your Celery settings, broker, and results backend, Merlin utilizes an app.yaml file. +!!! note + + Prior to Merlin v1.13.0, this configuration file had to be named `app.yaml`. If you hear any references to `app.yaml`, this is why. + +In order to read in configuration options for your Celery settings, broker, and results backend, Merlin utilizes a configuration file. -There's a built-in command with Merlin to set up a skeleton app.yaml for you: +There's a built-in command with Merlin to set up a skeleton configuration file for you: ```bash -merlin config +merlin config create ``` +_*Note:* If you want this file to be at a different path or if you want it to have a different name, use the `-o` option._ + This command will create an app.yaml file in the `~/.merlin/` directory that looks like so: @@ -264,3 +270,15 @@ For all other users, we recommend configuring with either: - [Dedicated External Servers](./external_server.md) - [Containerized Servers](./containerized_server.md) + +With any server setup that you use, it is possible to configure everything in your server from the command line. See [`merlin config update-broker`](../command_line.md#config-update-broker-merlin-config-update-broker) and [`merlin config update-backend`](../command_line.md#config-update-backend-merlin-config-update-backend) for more details on how this can be done. + +## Switching Between Configurations + +It's not uncommon to have two different configuration files with settings to connect to different servers. To switch between servers you can utilize the [`merlin config use`](../command_line.md#config-use-merlin-config-use) command. + +For example, you may have one configuration file that uses a RabbitMQ broker located at `/path/to/rabbitmq_config.yaml` and another that uses Redis as a broker located at `/path/to/redis_config.yaml`. If you want to switch to your Redis configuration, use: + +```bash +merlin config use /path/to/redis_config.yaml +``` diff --git a/docs/user_guide/configuration/merlin_server.md b/docs/user_guide/configuration/merlin_server.md deleted file mode 100644 index 39cf1327d..000000000 --- a/docs/user_guide/configuration/merlin_server.md +++ /dev/null @@ -1,177 +0,0 @@ -# Merlin Server Configuration - -!!! warning - - It's recommended that you read through the [Configuration Overview](./index.md) page before proceeding with this module. - -The merlin server command allows users easy access to containerized broker and results servers for Merlin workflows. This allows users to run Merlin without a dedicated external server. - -The main configuration will be stored in the subdirectory called `server/` by default in the main Merlin configuration directory `~/.merlin`. However, different server images can be created for different use cases or studies by simplying creating a new directory to store local configuration files for Merlin server instances. - -This module will walk through how to initalize the server, start it, and ensure it's linked to Merlin. - -## Initializing the Server - -First create and navigate into a directory to store your local Merlin configuration for a specific use case or study: - -```bash -mkdir study1/ ; cd study1/ -``` - -Afterwards you can instantiate Merlin server in this directory by running: - -```bash -merlin server init -``` - -A main server configuration will be created in the `~/.merlin/server/` directory. This will have the following files: - -- docker.yaml -- merlin_server.yaml -- podman.yaml -- singularity.yaml - -The main configuration in `~/.merlin/server/` deals with defaults and technical commands that might be used for setting up the Merlin server local configuration and its containers. Each container has their own configuration file to allow users to be able to switch between different containerized services freely. - -In addition to the main server configuration, a local server configuration will be created in your current working directory in a folder called `merlin_server/`. This directory will contain: - -- `redis.conf`: The Redis configuration file that contains all of the settings to be used for our Redis server -- `redis.pass`: A password for the Redis server that we'll start up next -- `redis.users`: A file defining the users that are allowed to access the Redis server and their permissions -- `redis_latest.sif`: A singularity file that contains the latest Redis Docker image that was pulled behind the scenes by Merlin - -The local configuration `merlin_server/` folder contains configuration files specific to a certain use case or run. In the case above you can see that we have a Redis singularity container called `redis_latest.sif` with the Redis configuration file called `redis.conf`. This Redis configuration will allow the user to configure Redis to their specified needs without have to manage or edit the Redis container. When the server is run this configuration will be dynamically read, so settings can be changed between runs if needed. - -Once the Merlin server has been initialized in the local directory the user will be allowed to run other Merlin server commands such as `start`, `status`, and `stop` to interact with the Merlin server. A detailed list of commands can be found in the [Merlin Server](../command_line.md#server-merlin-server) section of the [Command Line](../command_line.md) page. - -!!! note - - Running `merlin server init` again will *not* override any exisiting configuration that the users might have set or edited. By running this command again any missing files will be created for the users with exisiting defaults. *However,* it is highly advised that users back up their configuration in case an error occurs where configuration files are overriden. - -## Starting the Server and Linking it to Merlin - -!!! bug - - For LC users, servers cannot be started outside your home (`~/`) directory. - -!!! warning - - Newer versions of Redis have started requiring a global variable `LC_ALL` to be set in order for this to work. To set this properly, run: - - ```bash - export LC_ALL="C" - ``` - - If this is not set, the `merlin server start` command may seem to run forever until you manually terminate it. - -After initializing the server, starting the server is as simple as running: - -```bash -merlin server start -``` - -You can check that the server was started properly with: - -```bash -merlin server status -``` - -The `merlin server start` command will add new files to the local configuration `merlin_server/` folder: - -- `merlin_server.pf`: A process file containing information regarding the Redis process -- `app.yaml`: A new app.yaml file configured specifically for the containerized Redis server that we just started - -To have Merlin read this server configuration: - -=== "Copy Configuration to CWD" - - ```bash - cp merlin_server/app.yaml . - ``` - -=== "Make This Server Configuration Your Main Configuration" - - If you're going to use the server configuration as your main configuration, it's a good idea to make a backup of your current server configuration (if you have one): - - ```bash - mv ~/.merlin/app.yaml ~/.merlin/app.yaml.bak - ``` - - From here, simply copy the server configuration to your `~/.merlin/` folder: - - ```bash - cp merlin_server/app.yaml ~/.merlin/app.yaml - ``` - -You can check that Merlin recognizes the containerized server connection with: - -```bash -merlin info -``` - -If your servers are running and set up properly, this should output something similar to this: - -???+ success - - ```bash - * - *~~~~~ - *~~*~~~* __ __ _ _ - / ~~~~~ | \/ | | (_) - ~~~~~ | \ / | ___ _ __| |_ _ __ - ~~~~~* | |\/| |/ _ \ '__| | | '_ \ - *~~~~~~~ | | | | __/ | | | | | | | - ~~~~~~~~~~ |_| |_|\___|_| |_|_|_| |_| - *~~~~~~~~~~~ - ~~~*~~~* Machine Learning for HPC Workflows - - - - Merlin Configuration - ------------------------- - - config_file | /path/to/app.yaml - is_debug | False - merlin_home | /path/to/.merlin - merlin_home_exists | True - broker server | redis://default:******@127.0.0.1:6379/0 - broker ssl | False - results server | redis://default:******@127.0.0.1:6379/0 - results ssl | False - - Checking server connections: - ---------------------------- - broker server connection: OK - results server connection: OK - - Python Configuration - ------------------------- - - $ which python3 - /path/to/python3 - - $ python3 --version - Python x.y.z - - $ which pip3 - /path/to/pip3 - - $ pip3 --version - pip x.y.x from /path/to/pip (python x.y) - - "echo $PYTHONPATH" - ``` - -## Stopping the Server - -Once you're done using your containerized server, it can be stopped with: - -```bash -merlin server stop -``` - -You can check that it's no longer running with: - -```bash -merlin server status -``` diff --git a/docs/user_guide/database/database_cmd.md b/docs/user_guide/database/database_cmd.md new file mode 100644 index 000000000..cddfc586b --- /dev/null +++ b/docs/user_guide/database/database_cmd.md @@ -0,0 +1,1103 @@ +# The Database Command + +The [`merlin database`](../command_line.md#database-merlin-database) command provides a straightforward way to interact with the data stored in Merlin's database. For more information on what type of data is stored in Merlin's database, see [Understanding Merlin's Database Entities](./entities.md). + +**Usage:** + +```bash +merlin database SUBCOMMAND +``` + +Currently, the `merlin database` command supports three subcommands: + +- [`info`](#getting-general-database-information): Displays general information about the database. +- [`get`](#retrieving-and-displaying-entries): Retrieves specific entries from the database and prints them to the console. +- [`delete`](#deleting-entries): Removes entries from the database. + +For demonstration purposes, we'll start by running the `hello.yaml` and `hello_samples.yaml` files from the [Hello World Example](../../examples/hello.md) one time each. + +However, let's make a slight modification to these files to help distinguish the workers between them. Add the following [`merlin` block](../specification.md#the-merlin-block) to the `hello.yaml` file: + +```yaml +merlin: + resources: + workers: + hello_worker: + args: -l INFO --concurrency 1 --prefetch-multiplier 1 + steps: [all] +``` + +Similarly, we're going to update the `merlin` block of the `hello_samples.yaml` file as well: + +```yaml +merlin: + samples: + generate: + cmd: python3 $(SPECROOT)/make_samples.py --filepath=$(MERLIN_INFO)/samples.csv --number=$(N_SAMPLES) + file: $(MERLIN_INFO)/samples.csv + column_labels: [WORLD] + resources: + workers: + hello_samples_worker: + args: -l INFO --concurrency 1 --prefetch-multiplier 1 + steps: [all] +``` + +We'll also add task queues to the steps in `hello_samples.yaml`: + +```yaml +study: + - name: step_1 + description: say hello + run: + cmd: echo "$(GREET), $(WORLD)!" + task_queue: step_1_queue + + - name: step_2 + description: print a success message + run: + cmd: print("Hurrah, we did it!") + depends: [step_1_*] + shell: /usr/bin/env python3 + task_queue: step_2_queue +``` + +Now let's run these studies: + +```bash +merlin run hello.yaml ; merlin run hello_samples.yaml +``` + +And execute them by submitting worker launch scripts (examples of these scripts can be found in the [Distributed Runs](../running_studies.md#distributed-runs) section). + +## Getting General Database Information + +The `merlin database info` subcommand displays general information about the database including the type of database that's connected, the version of the database, and some basic information about the entries currently in the database. + +Let's execute this command: + +```bash +merlin database info +``` + +This provides us with the following output: + +```bash +Merlin Database Information +--------------------------- +General Information: +- Database Type: redis +- Database Version: 7.0.12 +- Connection String: rediss://:******@server.gov:12345/0 + +Studies: +- Total: 2 + +Runs: +- Total: 2 + +Logical Workers: +- Total: 2 + +Physical Workers: +- Total: 2 +``` + +## Retrieving and Displaying Entries + +The `merlin database get` subcommand allows users to query the database for specific information. This subcommand includes several options for retrieving data: + +- [`all-studies`](#retrieving-all-studies): Retrieves information about all studies in the database. +- [`study`](#retrieving-specific-studies): Retrieves information about specific studies in the database. +- [`all-runs`](#retrieving-all-runs): Retrieves information about all runs in the database. +- [`run`](#retrieving-specific-runs): Retrieves information about specific runs in the database. +- [`all-logical-workers`](#retrieving-all-logical-workers): Retrieves information about all logical workers in the database. +- [`logical-worker`](#retrieving-specific-logical-workers): Retrieves information about specific logical workers in the database. +- [`all-physical-workers`](#retrieving-all-physical-workers): Retrieves information about all physical workers in the database. +- [`physical-worker`](#retrieving-specific-physical-workers): Retrieves information about specific physical workers in the database. +- [`everything`](#retrieving-everything): Retrieves information about every entry in the database. + +A **study** represents a collection of related data, while a **run** refers to an individual execution of a study. A study is unique by study name where a run is created each time `merlin run` is executed. + +A **logical worker** represents the worker that's defined in a [specification file](../specification.md), while a **physical worker** represents the actual instantiation of the logical worker. Each logical worker entry is unique based upon the worker's name and associated task queues. A physical worker entry will not be created until the `merlin run-workers` command is executed and a worker is started. + +The following sections demonstrate how to use each option listed above. + +### Retrieving All Studies + +Using the `all-studies` option will retrieve and display each study in our database. When displayed, each study will display every field stored in a study entry. + +Let's try this out. Executing the following command: + +```bash +merlin database get all-studies +``` + +Will provide output similar to the following: + +```bash +Study with ID 75f49c2d-7135-41a8-a858-efad4ff19961 +------------------------------------------------ +Name: hello +Runs: + - ID: 0bdbae0b-c321-4178-a5a2-ab1ea6067be7 + Workspace: /path/to/hello_20250508-161150 +Additional Data: {} + + +Study with ID 837fafbe-4f40-4e47-8dd7-abb17142caed +------------------------------------------------ +Name: hello_samples +Runs: + - ID: c735ade0-9b28-4b9e-bb46-d9429d7cf61a + Workspace: /path/to/hello_samples_20250508-161159 +Additional Data: {} +``` + +The output includes the following fields: + +| Field | Description | +|------------------|-------------------------------------------------------------------| +| **ID** | A unique identifier for the study. | +| **Name** | The name assigned to the study. | +| **Runs** | A list of associated runs, with details such as ID and workspace. | +| **Additional Data** | Any extra metadata stored with the study. | + +### Retrieving Specific Studies + +To obtain information about specific studies, users can pass the name or ID of one or more studies to the `merlin database get study` command. + +For example, let's query just the "hello_samples" study: + +```bash +merlin database get study hello_samples +``` + +Which should display just the "hello_samples" study entry: + +```bash +Study with ID 837fafbe-4f40-4e47-8dd7-abb17142caed +------------------------------------------------ +Name: hello_samples +Runs: + - ID: c735ade0-9b28-4b9e-bb46-d9429d7cf61a + Workspace: /path/to/hello_samples_20250508-161159 +Additional Data: {} +``` + +### Retrieving All Runs + +Using the `all-runs` option will retrieve and display each run in our database. When displayed, each run will display every field stored in a run entry. + +Let's try this out. Executing the following command: + +```bash +merlin database get all-runs +``` + +Will provide output similar to the following: + +```bash +Run with ID c735ade0-9b28-4b9e-bb46-d9429d7cf61a +------------------------------------------------ +Workspace: /path/to/hello_samples_20250508-161159 +Study: + - ID: 837fafbe-4f40-4e47-8dd7-abb17142caed + Name: hello_samples +Queues: ['[merlin]_step_1_queue', '[merlin]_step_2_queue'] +Workers: ['4b0cd8f6-35a3-b484-4603-fa55eb0e7134'] +Parent: None +Child: None +Run Complete: False +Additional Data: {} + + +Run with ID 0bdbae0b-c321-4178-a5a2-ab1ea6067be7 +------------------------------------------------ +Workspace: /path/to/hello_20250508-161150 +Study: + - ID: 75f49c2d-7135-41a8-a858-efad4ff19961 + Name: hello +Queues: ['[merlin]_merlin'] +Workers: ['2f740737-a727-ea7d-6de4-17dc643183bb'] +Parent: None +Child: None +Run Complete: False +Additional Data: {} +``` + +The output includes the following fields: + +| Field | Description | +|--------------------|------------------------------------------------| +| **ID** | A unique identifier for the run. | +| **Workspace** | The workspace directory for the run. | +| **Study ID** | The unique identifier of the associated study. | +| **Queues** | A list of queues used by the run. | +| **Workers** | A list of workers assigned to the run. | +| **Parent** | The parent run ID, if any. | +| **Child** | The child run ID, if any. | +| **Run Complete** | Indicates whether the run is complete. | +| **Additional Data**| Any extra metadata stored with the run. | + +### Retrieving Specific Runs + +To obtain information about specific runs, users can pass the ID or workspace of one or more runs to the `merlin database get run` command. + +For example, let's query the run associated with the "hello" study: + +```bash +merlin database get run f93eecdf-d573-43d1-a3f9-c728c15802ea +``` + +Which should display just the one run entry: + +```bash +Run with ID 0bdbae0b-c321-4178-a5a2-ab1ea6067be7 +------------------------------------------------ +Workspace: /path/to/hello_20250508-161150 +Study: + - ID: 75f49c2d-7135-41a8-a858-efad4ff19961 + Name: hello +Queues: ['[merlin]_merlin'] +Workers: ['2f740737-a727-ea7d-6de4-17dc643183bb'] +Parent: None +Child: None +Run Complete: False +Additional Data: {} +``` + +### Retrieving All Logical Workers + +Using the `all-logical-workers` option will retrieve and display each logical worker in our database. When displayed, each logical worker will display every field stored in a logical worker entry. + +Let's try this out. Executing the following command: + +```bash +merlin database get all-logical-workers +``` + +Will provide output similar to the following: + +```bash +Logical Worker with ID 4b0cd8f6-35a3-b484-4603-fa55eb0e7134 +------------------------------------------------ +Name: hello_samples_worker +Runs: + - ID: c735ade0-9b28-4b9e-bb46-d9429d7cf61a + Workspace: /path/to/hello_samples_20250508-161159 +Queues: {'[merlin]_step_1_queue', '[merlin]_step_2_queue'} +Physical Workers: + - ID: 9a6b8bec-2ede-4a8c-bb07-0778c5c5f356 + Name: celery@hello_samples_worker.%ruby10 +Additional Data: {} + + +Logical Worker with ID 2f740737-a727-ea7d-6de4-17dc643183bb +------------------------------------------------ +Name: hello_worker +Runs: + - ID: 0bdbae0b-c321-4178-a5a2-ab1ea6067be7 + Workspace: /path/to/hello_20250508-161150 +Queues: {'[merlin]_merlin'} +Physical Workers: + - ID: 8549ed5f-83df-4922-aaac-16f676112322 + Name: celery@hello_worker.%ruby9 +Additional Data: {} +``` + +The output includes the following fields: + +| Field | Description | +|----------------------|------------------------------------------------------------------| +| **ID** | A unique identifier for the logical worker. | +| **Name** | The name of the logical worker from the spec. | +| **Runs** | The runs utilizing this logical worker. | +| **Queues** | A list of queues that the logical worker is watching. | +| **Physical Workers** | A list of physical worker instantiations of this logical worker. | +| **Additional Data** | Any extra metadata stored with the logical worker. | + +### Retrieving Specific Logical Workers + +To obtain information about specific logical workers, users can pass the ID of one or more logical workers to the `merlin database get logical-worker` command. + +For example, let's query the logical worker with the name "hello_worker": + +```bash +merlin database get logical-worker 2f740737-a727-ea7d-6de4-17dc643183bb +``` + +Which should display just the one run entry: + +```bash +Logical Worker with ID 2f740737-a727-ea7d-6de4-17dc643183bb +------------------------------------------------ +Name: hello_worker +Runs: + - ID: 0bdbae0b-c321-4178-a5a2-ab1ea6067be7 + Workspace: /path/to/hello_20250508-161150 +Queues: {'[merlin]_merlin'} +Physical Workers: + - ID: 8549ed5f-83df-4922-aaac-16f676112322 + Name: celery@hello_worker.%ruby9 +Additional Data: {} +``` + +### Retrieving All Physical Workers + +Using the `all-physical-workers` option will retrieve and display each physical worker in our database. When displayed, each physical worker will display every field stored in a physical worker entry. + +Let's try this out. Executing the following command: + +```bash +merlin database get all-physical-workers +``` + +Will provide output similar to the following: + +```bash +Physical Worker with ID 8549ed5f-83df-4922-aaac-16f676112322 +------------------------------------------------ +Name: celery@hello_worker.%ruby9 +Logical Worker ID: 2f740737-a727-ea7d-6de4-17dc643183bb +Launch Command: None +Args: {} +Process ID: 228105 +Status: WorkerStatus.RUNNING +Last Heartbeat: 2025-05-08 16:13:22.793487 +Last Spinup: 2025-05-08 16:13:22.793490 +Host: ruby9 +Restart Count: 0.0 +Additional Data: {} + + +Physical Worker with ID 9a6b8bec-2ede-4a8c-bb07-0778c5c5f356 +------------------------------------------------ +Name: celery@hello_samples_worker.%ruby10 +Logical Worker ID: 4b0cd8f6-35a3-b484-4603-fa55eb0e7134 +Launch Command: None +Args: {} +Process ID: 650803 +Status: WorkerStatus.RUNNING +Last Heartbeat: 2025-05-08 16:13:25.766678 +Last Spinup: 2025-05-08 16:13:25.766681 +Host: ruby10 +Restart Count: 0.0 +Additional Data: {} +``` + +The output includes the following fields: + +| Field | Description | +|-----------------------|--------------------------------------------------------| +| **ID** | A unique identifier for the physical worker. | +| **Name** | The name of the physical worker from Celery. | +| **Logical Worker ID** | The logical worker that this is associated with. | +| **Launch Command** | The command used to launch this worker. | +| **Args** | The arguments passed to this worker. | +| **Process ID** | The ID of the process that's running this worker | +| **Status** | The current status of this worker. | +| **Last Heartbeat** | The last time a heartbeat was received by this worker. | +| **Last Spinup** | The last time this worker was spun up. | +| **Host** | The host that this worker is running on. | +| **Restart Count** | The number of times this worker has been restarted. | +| **Additional Data** | Any extra metadata stored with the physical worker. | + +### Retrieving Specific Physical Workers + +To obtain information about specific physical workers, users can pass the IDl or name of one or more physical workers to the `merlin database get physical-worker` command. + +For example, let's query the physical worker for the "hello_samples" workflow: + +```bash +merlin database get physical-worker celery@hello_samples_worker.%ruby10 +``` + +Which should display just the one run entry: + +```bash +Physical Worker with ID 9a6b8bec-2ede-4a8c-bb07-0778c5c5f356 +------------------------------------------------ +Name: celery@hello_samples_worker.%ruby10 +Logical Worker ID: 4b0cd8f6-35a3-b484-4603-fa55eb0e7134 +Launch Command: None +Args: {} +Process ID: 650803 +Status: WorkerStatus.RUNNING +Last Heartbeat: 2025-05-08 16:13:25.766678 +Last Spinup: 2025-05-08 16:13:25.766681 +Host: ruby10 +Restart Count: 0.0 +Additional Data: {} +``` + +### Retrieving Everything + +Using the `everything` option will retrieve and display every entry in our database. In our case this would be 8 entries: 2 studies, 2 runs, 2 logical workers, and 2 physical workers. + +If you want to give this a shot use: + +```bash +merlin database get everything +``` + +## Deleting Entries + +The `merlin database delete` subcommand allows users to delete entries from the database. This subcommand includes several options for removing data: + +- [`all-studies`](#removing-all-studies): Remove all studies in the database. +- [`study`](#removing-specific-studies): Remove specific studies in the database. +- [`all-runs`](#removing-all-runs): Remove all runs in the database. +- [`run`](#removing-specific-runs): Remove specific runs in the database. +- [`all-logical-workers`](#removing-all-logical-workers): Remove all logical workers in the database. +- [`logical-worker`](#removing-specific-logical-workers): Remove specific logical workers in the database. +- [`all-physical-workers`](#removing-all-physical-workers): Remove all physical workers in the database. +- [`physical-worker`](#removing-specific-physical-workers): Remove specific physical workers in the database. +- [`everything`](#removing-everything): Removes every entry in the database, wiping the whole thing clean. + +The following sections explain how to use each option. + +### Removing All Studies + +!!! warning + + Using the `delete all-studies` option without the `-k` argument will remove *all* of the associated runs for *every* study. + +Using the `all-studies` option will delete every study in our database, along with their associated runs. For the purposes of this example, we'll refrain from removing all of the runs as well by utilizing the `-k` option. + +Assuming we still have our runs for the "hello" and "hello_samples" studies available to us, we can remove both studies *without* removing their associated runs using the following command: + +```bash +merlin database delete all-studies -k +``` + +This will provide output similar to the following: + +```bash +[2025-05-12 16:48:59: INFO] Reading app config from file /path/to/.merlin/app.yaml +[2025-05-12 16:48:59: INFO] Fetching all studys from Redis... +[2025-05-12 16:48:59: INFO] Successfully retrieved 2 studys from Redis. +[2025-05-12 16:48:59: INFO] Deleting study with id or name '8d3935ce-5eff-4f80-b55d-66e9dacd88b2' from the database... +[2025-05-12 16:48:59: INFO] Successfully deleted study with id '8d3935ce-5eff-4f80-b55d-66e9dacd88b2'. +[2025-05-12 16:48:59: INFO] Study '8d3935ce-5eff-4f80-b55d-66e9dacd88b2' has been successfully deleted. +[2025-05-12 16:48:59: INFO] Deleting study with id or name '77eddf31-bce5-4422-8782-e96cf372af43' from the database... +[2025-05-12 16:48:59: INFO] Successfully deleted study with id '77eddf31-bce5-4422-8782-e96cf372af43'. +[2025-05-12 16:48:59: INFO] Study '77eddf31-bce5-4422-8782-e96cf372af43' has been successfully deleted. +``` + +We can verify that this removed the studies from the database using the [`merlin database get all-studies`](#retrieving-all-studies) command: + +```bash +merlin database get all-studies +``` + +This should provide us with output denoting no studies were found: + +```bash +[2025-02-20 18:27:28: INFO] Fetching all studies from Redis... +[2025-02-20 18:27:29: INFO] Successfully retrieved 0 studies from Redis. +``` + +Similarly, we can verify that this *did not* remove all of the runs by using the [`merlin database get all-runs`](#retrieving-all-runs) command: + +```bash +merlin database get all-runs +``` + +This should still display the two runs we had before: + +```bash +Run with ID f93eecdf-d573-43d1-a3f9-c728c15802ea +------------------------------------------------ +Workspace: /path/hello_20250220-172743 +Study: + - ID: 849515a9-767c-4104-9ae9-6820ff000b65 + Name: hello +Queues: ['[merlin]_merlin'] +Workers: ['0bdbae0b-c321-4178-a5a2-ab1ea6067be7'] +Parent: None +Child: None +Run Complete: False +Additional Data: {} + + +Run with ID c55993e3-c210-4872-902d-e99e250e6f00 +------------------------------------------------ +Workspace: /path/to/hello_samples_20250220-182655 +Study: + - ID: 794836d9-1797-4b41-b962-d3688b93db52 + Name: hello_samples +Queues: ['[merlin]_merlin'] +Workers: ['4b0cd8f6-35a3-b484-4603-fa55eb0e7134'] +Parent: None +Child: None +Run Complete: False +Additional Data: {} +``` + +### Removing Specific Studies + +!!! warning + + Using the `delete study` option without the `-k` argument will remove *all* of the associated runs for this study. + +To remove specific studies from the database, users can pass the ID or name of a study to the `merlin database delete study` command. + +Assuming we still have our runs for the "hello" and "hello_samples" studies available to us, we can remove the "hello_samples" study using the following command: + +```bash +merlin database delete study hello_samples +``` + +This will remove the study and all runs associated with this study from the database. The output will look similar to this: + +```bash +[2025-05-12 16:50:31: INFO] Reading app config from file /path/to/.merlin/app.yaml +[2025-05-12 16:50:31: INFO] Attempting to delete run with id '42955780-087b-4b82-b0e3-99f82c1448be' from Redis... +[2025-05-12 16:50:31: INFO] Successfully deleted run '42955780-087b-4b82-b0e3-99f82c1448be' from Redis. +[2025-05-12 16:50:31: INFO] Run with id or workspace '42955780-087b-4b82-b0e3-99f82c1448be' has been successfully deleted. +[2025-05-12 16:50:31: INFO] Deleting study with id or name 'hello_samples' from the database... +[2025-05-12 16:50:31: INFO] Successfully deleted study with name 'hello_samples'. +[2025-05-12 16:50:31: INFO] Study 'hello_samples' has been successfully deleted. +``` + +We can verify that this removed the study from the database using the [`merlin database get all-studies`](#retrieving-all-studies) command: + +```bash +merlin database get all-studies +``` + +Which provides output similar to the following: + +```bash +Study with ID 849515a9-767c-4104-9ae9-6820ff000b65 +------------------------------------------------ +Name: hello +Runs: + - ID: f93eecdf-d573-43d1-a3f9-c728c15802ea + Workspace: /path/to/hello_20250220-172743 +Additional Data: {} +``` + +Here we see that the hello_samples study is no longer showing up. What about the run that was associated with the study though? Using the [`merlin database get all-runs`](#retrieving-all-runs) command, we can check: + +```bash +merlin database get all-runs +``` + +Which gives us: + +```bash +Run with ID f93eecdf-d573-43d1-a3f9-c728c15802ea +------------------------------------------------ +Workspace: /path/to/hello_20250220-172743 +Study ID: 849515a9-767c-4104-9ae9-6820ff000b65 +Queues: ['[merlin]_merlin'] +Workers: ['4b0cd8f6-35a3-b484-4603-fa55eb0e7134'] +Parent: None +Child: None +Run Complete: False +Additional Data: {} +``` + +There is no longer a run associated with the "hello_samples" study in our database. + +To prevent this, run the `merlin database delete study` command with the `-k` option. An example of this is shown in the [above section](#removing-all-studies). + +### Removing All Runs + +Using the `all-runs` option will delete every run in our database. + +Assuming we still have our runs for the "hello" and "hello_samples" studies available to us, we can remove both runs using the following command: + +```bash +merlin database delete all-runs +``` + +This will provide output similar to the following: + +```bash +[2025-05-12 16:52:16: INFO] Reading app config from file /path/to/.merlin/app.yaml +[2025-05-12 16:52:16: INFO] Fetching all runs from Redis... +[2025-05-12 16:52:16: INFO] Successfully retrieved 2 runs from Redis. +[2025-05-12 16:52:16: INFO] Attempting to delete run with id '5fb17970-8af6-40b4-a600-305170bca580' from Redis... +[2025-05-12 16:52:16: INFO] Successfully deleted run '5fb17970-8af6-40b4-a600-305170bca580' from Redis. +[2025-05-12 16:52:16: INFO] Run with id or workspace '5fb17970-8af6-40b4-a600-305170bca580' has been successfully deleted. +[2025-05-12 16:52:16: INFO] Attempting to delete run with id '60b60d60-cc3c-440e-ac51-cdf9c8cbd2e6' from Redis... +[2025-05-12 16:52:16: INFO] Successfully deleted run '60b60d60-cc3c-440e-ac51-cdf9c8cbd2e6' from Redis. +[2025-05-12 16:52:16: INFO] Run with id or workspace '60b60d60-cc3c-440e-ac51-cdf9c8cbd2e6' has been successfully deleted. +``` + +We can verify that this removed the runs from the database using the [`merlin database get all-runs`](#retrieving-all-runs) command: + +```bash +merlin database get all-runs +``` + +This should provide us with output denoting no runs were found: + +```bash +[2025-02-20 18:33:21: INFO] Fetching all runs from Redis... +[2025-02-20 18:33:21: INFO] Successfully retrieved 0 runs from Redis. +``` + +Similarly, utilizing the [`merlin database get all-studies`](#retrieving-all-studies) command: + +```bash +merlin database get all-studies +``` + +We should see that both studies do not have any runs: + +```bash +Study with ID c2a8082e-b651-42cc-9d21-c3bafeb7d8ed +------------------------------------------------ +Name: hello_samples +Runs: + No runs found. +Additional Data: {} + + +Study with ID e94efa15-af1f-41d1-b3fd-947a79f18387 +------------------------------------------------ +Name: hello +Runs: + No runs found. +Additional Data: {} +``` + +### Removing Specific Runs + +To remove specific runs from the database, users can pass the ID or workspace of one or more runs to the `merlin database delete run` command. + +Assuming we still have our runs for the "hello" and "hello_samples" studies available to us, we can remove the run for the "hello_samples" study using the following command: + +```bash +merlin database delete run 62fe3a0f-fc27-4e84-a8ea-cbd19aebc53f +``` + +This will remove the run from the database. The output will look similar to this: + +```bash +[2025-05-12 16:56:53: INFO] Reading app config from file /path/to/.merlin/app.yaml +[2025-05-12 16:56:53: INFO] Attempting to delete run with id '62fe3a0f-fc27-4e84-a8ea-cbd19aebc53f' from Redis... +[2025-05-12 16:56:53: INFO] Successfully deleted run '62fe3a0f-fc27-4e84-a8ea-cbd19aebc53f' from Redis. +[2025-05-12 16:56:53: INFO] Run with id or workspace '62fe3a0f-fc27-4e84-a8ea-cbd19aebc53f' has been successfully deleted. +``` + +We can verify that this removed the run from the database using the [`merlin database get all-runs`](#retrieving-all-runs) command: + +```bash +merlin database get all-runs +``` + +Which provides output showing only one run: + +```bash +Run with ID 97b9b899-f509-4460-9ecd-2ead5629bed3 +------------------------------------------------ +Workspace: /usr/WS1/gunny/debug/database_testing/hello/hello_20250220-183534 +Study ID: e94efa15-af1f-41d1-b3fd-947a79f18387 +Queues: ['[merlin]_merlin'] +Workers: ['default_worker'] +Parent: None +Child: None +Run Complete: False +Additional Data: {} +``` + +Similarly, utilizing the [`merlin database get all-studies`](#retrieving-all-studies) command: + +```bash +merlin database get all-studies +``` + +We should see that only the "hello" study has a run: + +```bash +Study with ID c2a8082e-b651-42cc-9d21-c3bafeb7d8ed +------------------------------------------------ +Name: hello_samples +Runs: + No runs found. +Additional Data: {} + + +Study with ID e94efa15-af1f-41d1-b3fd-947a79f18387 +------------------------------------------------ +Name: hello +Runs: + - ID: 97b9b899-f509-4460-9ecd-2ead5629bed3 + Workspace: /path/to/hello_20250220-183534 +Additional Data: {} +``` + +### Removing All Logical Workers + +Using the `all-logical-workers` option will delete every logical-worker in our database. + +Assuming we still have our logical workers for the "hello" and "hello_samples" studies available to us, we can remove both logical workers using the following command: + +```bash +merlin database delete all-logical-workers +``` + +This will provide output similar to the following: + +```bash +[2025-05-12 16:57:38: INFO] Reading app config from file /path/to/.merlin/app.yaml +[2025-05-12 16:57:38: INFO] Fetching all logical_workers from Redis... +[2025-05-12 16:57:38: INFO] Successfully retrieved 2 logical_workers from Redis. +[2025-05-12 16:57:38: INFO] Attempting to delete logical_worker with id '2f740737-a727-ea7d-6de4-17dc643183bb' from Redis... +[2025-05-12 16:57:38: INFO] Successfully deleted logical_worker '2f740737-a727-ea7d-6de4-17dc643183bb' from Redis. +[2025-05-12 16:57:38: INFO] Worker with id '2f740737-a727-ea7d-6de4-17dc643183bb' has been successfully deleted. +[2025-05-12 16:57:38: INFO] Attempting to delete logical_worker with id '4b0cd8f6-35a3-b484-4603-fa55eb0e7134' from Redis... +[2025-05-12 16:57:38: INFO] Successfully deleted logical_worker '4b0cd8f6-35a3-b484-4603-fa55eb0e7134' from Redis. +[2025-05-12 16:57:38: INFO] Worker with id '4b0cd8f6-35a3-b484-4603-fa55eb0e7134' has been successfully deleted. +``` + +We can verify that this removed the logical workers from the database using the [`merlin database get all-logical-workers`](#retrieving-all-logical-workers) command: + +```bash +merlin database get all-logical-workers +``` + +This should provide us with output denoting no logical workers were found: + +```bash +[2025-05-08 17:11:37: INFO] Fetching all logical workers from Redis... +[2025-05-08 17:11:37: INFO] Successfully retrieved 0 logical workers from Redis. +[2025-05-08 17:11:37: INFO] No logical workers found in the database. +``` + +Similarly, utilizing the [`merlin database get all-runs`](#retrieving-all-runs) command: + +```bash +merlin database get all-runs +``` + +We should see that both runs do not have any workers: + +```bash +Run with ID 0bdbae0b-c321-4178-a5a2-ab1ea6067be7 +------------------------------------------------ +Workspace: /path/to/hello_20250508-161150 +Study: + - ID: 75f49c2d-7135-41a8-a858-efad4ff19961 + Name: hello +Queues: ['[merlin]_merlin'] +Workers: [] +Parent: None +Child: None +Run Complete: True +Additional Data: {} + + +Run with ID c735ade0-9b28-4b9e-bb46-d9429d7cf61a +------------------------------------------------ +Workspace: /path/to/hello_samples_20250508-161159 +Study: + - ID: 837fafbe-4f40-4e47-8dd7-abb17142caed + Name: hello_samples +Queues: ['[merlin]_step_1_queue', '[merlin]_step_2_queue'] +Workers: [] +Parent: None +Child: None +Run Complete: True +Additional Data: {} +``` + +### Removing Specific Logical Workers + +To remove specific logical workers from the database, users can pass the ID of a logical worker to the `merlin database delete logical-worker` command. + +Assuming we still have our logical workers for the "hello" and "hello_samples" studies available to us, we can remove the logical worker for the "hello_samples" study using the following command: + +```bash +merlin database delete logical-worker 4b0cd8f6-35a3-b484-4603-fa55eb0e7134 +``` + +This will remove the logical worker from the database. The output will look similar to this: + +```bash +[2025-05-12 16:59:16: INFO] Reading app config from file /path/to/.merlin/app.yaml +[2025-05-12 16:59:16: INFO] Attempting to delete logical_worker with id '4b0cd8f6-35a3-b484-4603-fa55eb0e7134' from Redis... +[2025-05-12 16:59:16: INFO] Successfully deleted logical_worker '4b0cd8f6-35a3-b484-4603-fa55eb0e7134' from Redis. +[2025-05-12 16:59:16: INFO] Worker with id '4b0cd8f6-35a3-b484-4603-fa55eb0e7134' has been successfully deleted. +``` + +We can verify that this removed the logical worker from the database using the [`merlin database get all-logical-workers`](#retrieving-all-logical-workers) command: + +```bash +merlin database get all-logical-workers +``` + +Which provides output showing only one logical worker: + +```bash +Logical Worker with ID 2f740737-a727-ea7d-6de4-17dc643183bb +------------------------------------------------ +Name: hello_worker +Runs: + - ID: 0bdbae0b-c321-4178-a5a2-ab1ea6067be7 + Workspace: /path/to/hello_20250508-161150 +Queues: {'[merlin]_merlin'} +Physical Workers: + - ID: 8549ed5f-83df-4922-aaac-16f676112322 + Name: celery@hello_worker.%ruby9 +Additional Data: {} +``` + +Similarly, utilizing the [`merlin database get all-runs`](#retrieving-all-runs) command: + +```bash +merlin database get all-runs +``` + +We should see that only the "hello" run has a worker: + +```bash +Run with ID 0bdbae0b-c321-4178-a5a2-ab1ea6067be7 +------------------------------------------------ +Workspace: /path/to/hello_20250508-161150 +Study: + - ID: 75f49c2d-7135-41a8-a858-efad4ff19961 + Name: hello +Queues: ['[merlin]_merlin'] +Workers: ['2f740737-a727-ea7d-6de4-17dc643183bb'] +Parent: None +Child: None +Run Complete: True +Additional Data: {} + + +Run with ID c735ade0-9b28-4b9e-bb46-d9429d7cf61a +------------------------------------------------ +Workspace: /path/to/hello_samples_20250508-161159 +Study: + - ID: 837fafbe-4f40-4e47-8dd7-abb17142caed + Name: hello_samples +Queues: ['[merlin]_step_1_queue', '[merlin]_step_2_queue'] +Workers: [] +Parent: None +Child: None +Run Complete: True +Additional Data: {} +``` + +### Removing All Physical Workers + +Using the `all-physical-workers` option will delete every physical worker in our database. + +Assuming we still have our physical workers for the "hello" and "hello_samples" studies available to us, we can remove both physical workers using the following command: + +```bash +merlin database delete all-physical-workers +``` + +This will provide output similar to the following: + +```bash +[2025-05-12 17:04:20: INFO] Reading app config from file /path/to/.merlin/app.yaml +[2025-05-12 17:04:20: INFO] Fetching all physical_workers from Redis... +[2025-05-12 17:04:20: INFO] Successfully retrieved 2 physical_workers from Redis. +[2025-05-12 17:04:20: INFO] Successfully deleted physical_worker with id '04536fdd-d096-4b58-bc26-9b44c489b4c6'. +[2025-05-12 17:04:20: INFO] Worker '04536fdd-d096-4b58-bc26-9b44c489b4c6' has been successfully deleted. +[2025-05-12 17:04:20: INFO] Successfully deleted physical_worker with id '6387b7b9-4bbd-4067-ae0a-e6003b8c1186'. +[2025-05-12 17:04:20: INFO] Worker '6387b7b9-4bbd-4067-ae0a-e6003b8c1186' has been successfully deleted. +``` + +We can verify that this removed the physical workers from the database using the [`merlin database get all-physical-workers`](#retrieving-all-physical-workers) command: + +```bash +merlin database get all-physical-workers +``` + +This should provide us with output denoting no physical workers were found: + +```bash +[2025-05-12 17:06:22: INFO] Fetching all physical_workers from Redis... +[2025-05-12 17:06:23: INFO] Successfully retrieved 0 physical_workers from Redis. +[2025-05-12 17:06:23: INFO] No physical workers found in the database. +``` + +Similarly, utilizing the [`merlin database get all-logical-workers`](#retrieving-all-logical-workers) command: + +```bash +merlin database get all-logical-workers +``` + +We should see that both logical workers do not have any physical workers: + +```bash +Logical Worker with ID 2f740737-a727-ea7d-6de4-17dc643183bb +------------------------------------------------ +Name: hello_worker +Runs: + No runs found. +Queues: {'[merlin]_merlin'} +Physical Workers: + No physical workers found. +Additional Data: {} + + +Logical Worker with ID 4b0cd8f6-35a3-b484-4603-fa55eb0e7134 +------------------------------------------------ +Name: hello_samples_worker +Runs: + No runs found. +Queues: {'[merlin]_step_1_queue', '[merlin]_step_2_queue'} +Physical Workers: + No physical workers found. +Additional Data: {} +``` + +### Removing Specific Physical Workers + +To remove specific physical workers from the database, users can pass the ID of a physical worker to the `merlin database delete physical-worker` command. + +Assuming we still have our physical workers for the "hello" and "hello_samples" studies available to us, we can remove the physical worker for the "hello_samples" study using the following command (yours will likely be slightly different depending on the worker name): + +```bash +merlin database delete physical-worker celery@hello_samples_worker.%rzadams1017 +``` + +This will remove the physical worker from the database. The output will look similar to this: + +```bash +[2025-05-12 17:10:12: INFO] Reading app config from file /path/to/.merlin/app.yaml +[2025-05-12 17:10:12: INFO] Successfully deleted physical_worker with name 'celery@hello_samples_worker.%rzadams1017'. +[2025-05-12 17:10:12: INFO] Worker 'celery@hello_samples_worker.%rzadams1017' has been successfully deleted. +``` + +We can verify that this removed the physical worker from the database using the [`merlin database get all-physical-workers`](#retrieving-all-physical-workers) command: + +```bash +merlin database get all-physical-workers +``` + +Which provides output showing only one physical worker: + +```bash +Physical Worker with ID cc6934ab-f6c2-41e8-ab61-03cc7d3e3f5e +------------------------------------------------ +Name: celery@hello_worker.%rzadams1017 +Logical Worker ID: 2f740737-a727-ea7d-6de4-17dc643183bb +Launch Command: None +Args: {} +Process ID: None +Status: WorkerStatus.STOPPED +Last Heartbeat: 2025-05-12 17:09:14.699762 +Last Spinup: 2025-05-12 17:09:14.699767 +Host: rzadams1017 +Restart Count: 0.0 +Additional Data: {} +``` + +Similarly, utilizing the [`merlin database get all-logical-workers`](#retrieving-all-logical-workers) command: + +```bash +merlin database get all-logical-workers +``` + +We should see that only the "hello" logical worker has a physical worker implementation: + +```bash +Logical Worker with ID 2f740737-a727-ea7d-6de4-17dc643183bb +------------------------------------------------ +Name: hello_worker +Runs: + No runs found. +Queues: {'[merlin]_merlin'} +Physical Workers: + - ID: cc6934ab-f6c2-41e8-ab61-03cc7d3e3f5e + Name: celery@hello_worker.%rzadams1017 +Additional Data: {} + + +Logical Worker with ID 4b0cd8f6-35a3-b484-4603-fa55eb0e7134 +------------------------------------------------ +Name: hello_samples_worker +Runs: + No runs found. +Queues: {'[merlin]_step_2_queue', '[merlin]_step_1_queue'} +Physical Workers: + No physical workers found. +Additional Data: {} +``` + +### Removing Everything + +Using the `everything` option will delete every entry in our database. + +Assuming we still have all of our entries for the "hello" and "hello_samples" studies available to us, we can remove everything with the following command: + +```bash +merlin database delete everything +``` + +This will bring up a prompt asking if you really want to flush the database: + +```bash +[2025-05-12 17:16:44: INFO] Reading app config from file /path/to/.merlin/app.yaml +Are you sure you want to flush the entire database? (y/n): +``` + +If you answer `n` to this prompt, the command will stop executing and nothing will be deleted. However, if you answer `y` to this prompt, you will see output similar to the following: + +```bash +[2025-05-12 17:16:44: INFO] Reading app config from file /path/to/.merlin/app.yaml +Are you sure you want to flush the entire database? (y/n): y +[2025-05-12 17:17:48: INFO] Flushing the database... +[2025-05-12 17:17:48: INFO] Database successfully flushed. +``` + +We can verify that this removed the every entry from the database using the [`merlin database get everything`](#retrieving-everything) command: + +```bash +merlin database get everything +``` + +This should provide us with output denoting nothing was found: + +```bash +[2025-05-12 17:21:24: INFO] Reading app config from file /path/to/.merlin/app.yaml +[2025-05-12 17:21:24: INFO] Fetching all logical_workers from Redis... +[2025-05-12 17:21:24: INFO] Successfully retrieved 0 logical_workers from Redis. +[2025-05-12 17:21:24: INFO] Fetching all physical_workers from Redis... +[2025-05-12 17:21:24: INFO] Successfully retrieved 0 physical_workers from Redis. +[2025-05-12 17:21:24: INFO] Fetching all runs from Redis... +[2025-05-12 17:21:24: INFO] Successfully retrieved 0 runs from Redis. +[2025-05-12 17:21:24: INFO] Fetching all studies from Redis... +[2025-05-12 17:21:24: INFO] Successfully retrieved 0 studies from Redis. +[2025-05-12 17:21:24: INFO] Nothing found in the database. +``` + +Similarly, utilizing the [`merlin database info`](#getting-general-database-information) command: + +```bash +merlin database info +``` + +We should see every entry should have a total of 0: + +```bash +Merlin Database Information +--------------------------- +General Information: +- Database Type: redis +- Database Version: 7.0.12 +- Connection String: rediss://:******@server.gov:12345/0 + +Studies: +- Total: 0 + +Runs: +- Total: 0 + +Logical Workers: +- Total: 0 + +Physical Workers: +- Total: 0 +``` diff --git a/docs/user_guide/database/entities.md b/docs/user_guide/database/entities.md new file mode 100644 index 000000000..82b45f242 --- /dev/null +++ b/docs/user_guide/database/entities.md @@ -0,0 +1,90 @@ +# Understanding Merlin's Database Entities + +Merlin's database has the following entities: + +- [`StudyEntity`](#study-entity) +- [`RunEntity`](#run-entity) +- Two worker entities: + - [`LogicalWorkerEntity`](#logical-worker-entity) + - [`PhysicalWorkerEntity`](#physical-worker-entity) + +## Study Entity + +The `StudyEntity` represents a single “study” or experiment grouping in Merlin. Each entry is unique to study name (defined in the [`description` block](../specification.md#the-description-block) of the specification file). Each study acts as a namespace under which runs are organized and tracked. + +| Key | Type | Description | +| ------ | ------------- | ---------------------------------------------------------------- | +| `id` | `uuid4` | Primary key. Unique identifier for the study. | +| `name` | `str` | Human-readable name for the study, unique to each `StudyEntity`. | +| `runs` | `List[uuid4]` | List of Run IDs associated with this study. | + +**Relationships:** + +- One-to-many with [`RunEntity`](#run-entity): A single study can have multiple runs. + +## Run Entity + +The `RunEntity` represents a single execution of a study. It captures the configuration, intermediate data, and relationships to other entities (like workers and studies). + +| Column | Type | Description | +| -------------- | ---------------- | --------------------------------------------------------------------------------- | +| `id` | `uuid4` | Primary key. Unique identifier for the run. | +| `study_id` | `uuid4` | Foreign key → StudyEntity(`id`). Which study this run belongs to. | +| `workspace` | `str` | Filesystem path where outputs of the study are stored. | +| `steps` | `List[uuid4]` | Ordered list of Step IDs executed in this run. | +| `queues` | `List[str]` | List of task queue names used by this run. | +| `workers` | `List[uuid4]` | List of [`LogicalWorker`](#logical-worker-entity) IDs serving tasks for this run. | +| `parent` | `uuid4 \| NULL` | ID of parent run (if this run was started by another run). | +| `child` | `uuid4 \| NULL` | ID of child run (if this run spawned a new run). | +| `run_complete` | `bool` | Indicates whether the run has finished. | +| `parameters` | `Dict` | Arbitrary key/value parameters provided to the run. | +| `samples` | `Dict` | Arbitrary samples provided to the run. | + +**Relationships:** + +- Many-to-one with [`StudyEntity`](#study-entity): Multiple runs can be assigned to the same study. +- Many-to-many with [`LogicalWorkerEntity`](#logical-worker-entity): Multiple runs can be linked to multiple logical workers. +- Optional one-to-one with parent/child `RunEntity`: A single run can link to another run. + +## Worker Entities + +Merlin supports two distinct worker models: logical and physical. Logical workers define high-level behavior and configuration. Physical workers represent actual runtime processes launched from logical definitions. The below sections will go into further detail on both entities. + +### Logical Worker Entity + +The `LogicalWorkerEntity` defines an abstract worker configuration — including queues and a name — which serves as a template for actual (physical) worker instances. Each logical worker is unique to its name and queues. For instance, `LogicalWorker(name=worker1, queues=[queue1, queue2])` is different from `LogicalWorker(name=worker1, queues=[queue1])` which is also different from `LogicalWorker(name=worker2, queues=[queue1, queue2])`. + +| Column | Type | Description | +| ------------------ | ---------------- | ------------------------------------------------------------------------------------------------ | +| `id` | `uuid4` | Primary key. Deterministically generated from `name` + `queues`. | +| `name` | `str` | Logical name of the worker (e.g., `"data-processor"`). | +| `queues` | `List[str]` | The set of queue names this logical worker listens on. | +| `runs` | `List[uuid4]` | List of [`RunEntity`](#run-entity) IDs currently using this logical worker. | +| `physical_workers` | `List[uuid4]` | List of [`PhysicalWorker`](#physical-worker-entity) IDs instantiated from this logical template. | + +**Relationships:** + +- One-to-many with [`PhysicalWorkerEntity`](#physical-worker-entity): A single logical worker can have multiple physical worker instances. +- Many-to-many with [`RunEntity`](#run-entity): Multiple logical workers can be linked to multiple runs. + +### Physical Worker Entity + +The `PhysicalWorkerEntity` represents an actual running instance of a worker process, created from a logical worker definition. It contains runtime-specific metadata for monitoring and control. + +| Column | Type | Description | +| --------------------- | -------------- | --------------------------------------------------------------- | +| `id` | `uuid4` | Primary key. Unique identifier for this running process. | +| `logical_worker_id` | `uuid4` | Foreign key → LogicalWorkerEntity(`id`). | +| `name` | `str` | Full Celery worker name (e.g., `celery@hostname`). | +| `launch_cmd` | `str` | Exact CLI used to start the worker. | +| `args` | `Dict` | Additional runtime args or config passed to the worker process. | +| `pid` | `str` | OS process ID in string format. | +| `status` | `WorkerStatus` | Current status (e.g., `RUNNING`, `STOPPED`). | +| `heartbeat_timestamp` | `datetime` | Last time the worker checked in. | +| `latest_start_time` | `datetime` | When this process was most recently (re)launched. | +| `host` | `str` | Hostname or IP where this process is running. | +| `restart_count` | `int` | How many times the process has been restarted. | + +**Relationships:** + +- Many-to-one with [`LogicalWorkerEntity`](#logical-worker-entity): Multiple physical workers can be linked to multiple logical workers. diff --git a/docs/user_guide/database/index.md b/docs/user_guide/database/index.md new file mode 100644 index 000000000..21f21fb7e --- /dev/null +++ b/docs/user_guide/database/index.md @@ -0,0 +1,28 @@ +# Merlin's Database + +!!! Note + + If running in local mode, results will be stored in a sqlite file in your Merlin home directory. This file will be named `~/.merlin/merlin.db`. To interact with this file while using the [`merlin database`](../command_line.md#database-merlin-database) command, use the `-l` (or `--local`) option. For example, + + ```bash + merlin database -l info + ``` + +When a Merlin study is run with the [`merlin run`](../command_line.md#run-merlin-run) and [`merlin run-workers`](../command_line.md#run-workers-merlin-run-workers) commands, information about the study is stored as data in the [results backend database](../configuration/index.md#what-is-a-results-backend). + +See the below sections for [how the database works](#how-it-works) and [why it's needed](#why-is-this-needed). Additional info about Merlin's database can be found at the below pages: + +- [The Database Command](./database_cmd.md) +- [Understanding Merlin's Database Entities](./entities.md) + +## How it Works + +Unless [running locally](../running_studies.md#local-runs), Merlin requires a connection to a [results backend](../configuration/index.md#what-is-a-results-backend) database by default. Therefore, Merlin's database will automatically link to this results backend connection, using it as a store for study-related information. + +When a study is converted to Celery tasks and sent to the [broker](../configuration/index.md#what-is-a-broker) with the `merlin run` command, Merlin will create the following entities in your database: a `StudyEntity` (if one does not yet exist), a `RunEntity`, and one or more `LogicalWorkerEntity` instances (if they don't already exist). + +Similarly, when workers are started on your allocation with the `merlin run-workers` command, Merlin will create one or more `LogicalWorkerEntity` instances (if they don't already exist) and one or more `PhysicalWorkerEntity` instances. + +## Why is this Needed? + +The [entities](./entities.md) that are created in the database are used by the [`merlin monitor`](../command_line.md#monitor-merlin-monitor) command to help keep your workflow alive during the lifetime of your allocation. You can read more about how this process works on the [Monitoring Studies for Persistent Allocations](../monitoring/monitor_for_allocation.md) page. diff --git a/docs/user_guide/installation.md b/docs/user_guide/installation.md index d73d51db5..29f4df04b 100644 --- a/docs/user_guide/installation.md +++ b/docs/user_guide/installation.md @@ -1,6 +1,6 @@ # Installation -The Merlin library can be installed by using [virtual environments and pip](#installing-with-pip) or [spack](#installing-with-spack). +The Merlin library can be installed by using [virtual environments and pip](#installing-with-virtual-environments-pip) or [spack](#installing-with-spack). Contributors to Merlin should follow the [Developer Setup](#developer-setup) below. diff --git a/docs/user_guide/interpreting_output.md b/docs/user_guide/interpreting_output.md index a60e5e490..96c81862b 100644 --- a/docs/user_guide/interpreting_output.md +++ b/docs/user_guide/interpreting_output.md @@ -677,7 +677,7 @@ A visual representation of the `merlin_info/` subdirectory with sample generatio f.write(result) ``` - Since this script uses some third party libraries ([`names`](https://pypi.org/project/names/) and [`numpy`](https://numpy.org/)), you'll need to install them to your current environment in order to run this example. If you're using a [virtual environment](./installation.md#installing-with-virtual-environments--pip), these can be installed with: + Since this script uses some third party libraries ([`names`](https://pypi.org/project/names/) and [`numpy`](https://numpy.org/)), you'll need to install them to your current environment in order to run this example. If you're using a [virtual environment](./installation.md#installing-with-virtual-environments-pip), these can be installed with: ```bash pip install names numpy diff --git a/docs/user_guide/monitoring/monitor_for_allocation.md b/docs/user_guide/monitoring/monitor_for_allocation.md index c2e4a17e5..0c2fcc435 100644 --- a/docs/user_guide/monitoring/monitor_for_allocation.md +++ b/docs/user_guide/monitoring/monitor_for_allocation.md @@ -12,12 +12,17 @@ merlin monitor ## How Does the Monitor Work? -The `merlin monitor` command takes a spec file as input, using it to identify the task queues and workers it needs to observe. This monitoring process involves two key actions: +The `merlin monitor` command uses a [spec file](../specification.md) to query Merlin's [backend database](../configuration/index.md#what-is-a-results-backend) for information about a study. It retrieves the study's entry from the database, including all individual runs associated with the study. The monitor then iterates through each run, ensuring they all complete successfully. Once all runs complete, the monitor will terminate, allowing your allocation to end. + +For each run of a study, the monitor ensures completion by performing the following steps: 1. Verifying the presence of tasks in the designated queues. 2. Confirming the ongoing processing of tasks by the assigned workers when the queues are empty. +3. Restarting the study if there are no tasks in the queues and no workers processing tasks, but the workflow has not yet finished. + +The monitor includes a [`--sleep` option](#sleep), which introduces a deliberate delay. Before starting, the monitor waits for the specified `--sleep` duration, giving users time to populate the task queues for their run using the [`merlin run`](../command_line.md#run-merlin-run) command. Additionally, the monitor pauses for the `--sleep` duration between each check of the run. Finally, it will wait up to 10 times the specified `--sleep` duration for workers to spin up for the run. -The monitor comes with a [`--sleep` option](#sleep), which introduces a deliberate delay. Before the monitoring initiates, the monitor waits up to 10 times the specified sleep duration, providing users with a window to populate the task queues with the [`merlin run`](../command_line.md#run-merlin-run) command. Subsequently, it waits for the specified sleep duration between each check to determine if the queues have tasks (step 1 above). If no tasks are found, and no workers are processing tasks, the monitor concludes that the workflow has finished, allowing the allocation to end. This way, the monitor command acts as a blocking process, ensuring the continuous and effective management of tasks within the specified workflow. +A run is considered complete when the monitor reads the run's `run_complete` entry from the database and it returns `True`. This entry is always set as the final task of a run. The resulting flowchart of this process can be seen below. @@ -126,16 +131,16 @@ Adding the `merlin monitor` command to your workflow process is as simple as put There are three useful options that come with the `merlin monitor` command: - [`--sleep`](#sleep): The delay between checks on the task queues -- [`--steps`](#steps): Only monitor specific steps in your workflow +- [`--steps`](#steps) (*Deprecated*): Only monitor specific steps in your workflow - [`--vars`](#vars): Modify environment variables in a spec from the command line ### Sleep -The `--sleep` option in the `monitor` command allows users to specify a custom delay duration between consecutive inspections of the task queues. The default value for this option is 60 seconds. +The `--sleep` option in the `monitor` command allows users to specify a custom delay duration between consecutive inspections of runs. The default value for this option is 60 seconds. -As detailed in the ["How Does the Monitor Work?"](#how-does-the-monitor-work) section, the monitor periodically examines task queues to determine task presence. If the queues are currently occupied, the monitor will enter a sleep state for a designated duration before conducting the next inspection. Similarly, if the monitor discovers no tasks in the queues but identifies active workers processing tasks, it will initiate a sleep interval before re-evaluating both the queues and the workers. The `--sleep` option allows you to modify this sleep interval. +As detailed in the ["How Does the Monitor Work?"](#how-does-the-monitor-work) section, the monitor periodically examines runs to determine the status of their queues, workers, and whether the run is complete or not. If the queues are occupied, workers are processing tasks, or the run is not yet finished, the monitor will enter a sleep state for a designated duration before conducting the next inspection. The `--sleep` option allows you to modify this sleep interval. -The value that you provide for the `--sleep` option will be an integer representing the number of seconds to sleep before the next inspection of the task queues and workers is conducted. +The value that you provide for the `--sleep` option will be an integer representing the number of seconds to sleep before the next inspection of the run is conducted. **Usage:** @@ -213,7 +218,7 @@ merlin monitor --sleep From the time stamps in our worker logs we can see that the custom 30 second sleep duration was applied: - ```bash hl_lines="20-25 33-34" + ```bash hl_lines="20 24-30 43-44" [2024-02-05 09:13:52,891: INFO] Connected to amqps://rabbitmerlin:**@cz-gunny-rabbitmerlin.apps.czapps.llnl.gov:31118/host4gunny [2024-02-05 09:13:52,911: INFO] mingle: searching for neighbors [2024-02-05 09:13:53,956: INFO] mingle: all alone @@ -232,26 +237,40 @@ merlin monitor --sleep [2024-02-05 09:13:54,276: INFO] Status for step_1 successfully written. [2024-02-05 09:13:54,276: INFO] Submitting script for step_1 [2024-02-05 09:13:54,548: INFO] Task merlin.common.tasks.expand_tasks_with_samples[78530a48-95f0-4b0e-90ca-7011e81a7808] succeeded in 0.40144235407933593s: None - [2024-02-05 09:14:16: INFO] Reading app config from file /g/g20/gunny/.merlin/app.yaml - [2024-02-05 09:14:17: INFO] Monitor: found 0 jobs in queues and 1 workers alive - [2024-02-05 09:14:18: INFO] Monitor: found tasks in queues and/or tasks being processed - [2024-02-05 09:14:50: INFO] Monitor: found 0 jobs in queues and 1 workers alive - [2024-02-05 09:14:51: INFO] Monitor: found tasks in queues and/or tasks being processed - [2024-02-05 09:15:22: INFO] Monitor: found 0 jobs in queues and 1 workers alive - [2024-02-05 09:15:23: INFO] Monitor: found tasks in queues and/or tasks being processed - [2024-02-05 09:15:24,298: INFO] Execution returned status OK. - [2024-02-05 09:15:24,304: INFO] Writing status for step_1 to '/usr/WS1/gunny/hello/sleep_demo_20240205-091232/step_1/MERLIN_STATUS.json... - [2024-02-05 09:15:24,307: INFO] Status for step_1 successfully written. - [2024-02-05 09:15:24,307: INFO] Step 'step_1' in '/usr/WS1/gunny/hello/sleep_demo_20240205-091232/step_1' finished successfully. - [2024-02-05 09:15:24,498: INFO] Task merlin:chordfinisher[f442f13e-0436-4162-86ab-eaa28943f526] received - [2024-02-05 09:15:24,501: INFO] Task merlin.common.tasks.merlin_step[117b28c9-eacd-4e77-9771-01b4ebc29e01] succeeded in 90.27513551106676s: 0 - [2024-02-05 09:15:24,507: INFO] Task merlin:chordfinisher[f442f13e-0436-4162-86ab-eaa28943f526] succeeded in 0.007889348082244396s: 'SYNC' - [2024-02-05 09:15:54: INFO] Monitor: found 0 jobs in queues and 1 workers alive - [2024-02-05 09:15:55: INFO] Monitor: ... stop condition met + [2025-02-20 14:41:27: INFO] Reading app config from file /g/g20/gunny/.merlin/app.yaml + [2025-02-20 14:41:27: INFO] Monitor: Monitoring run with workspace '/usr/WS1/gunny/debug/temp/sleep_demo_20250220-143820'... + [2025-02-20 14:41:27: INFO] Checking for the following workers: ['default_worker'] + [2025-02-20 14:41:27: INFO] Overriding default celery config with 'celery.override' in 'app.yaml': + visibility_timeout: 86400 + [2025-02-20 14:41:28: INFO] Monitor: checking for workers, running workers = ['celery@default_worker.%ruby1'] ... + [2025-02-20 14:41:31: INFO] Monitor: Found workers processing tasks, keeping allocation alive. + [2025-02-20 14:42:01: INFO] Checking for the following workers: ['default_worker'] + [2025-02-20 14:42:02: INFO] Monitor: checking for workers, running workers = ['celery@default_worker.%ruby1'] ... + [2025-02-20 14:42:04: INFO] Monitor: Found workers processing tasks, keeping allocation alive. + [2025-02-20 14:42:34: INFO] Checking for the following workers: ['default_worker'] + [2025-02-20 14:42:35: INFO] Monitor: checking for workers, running workers = ['celery@default_worker.%ruby1'] ... + [2025-02-20 14:42:35,928: INFO] Execution returned status OK. + [2025-02-20 14:42:35,940: INFO] Writing status for step_1 to '/usr/WS1/gunny/debug/temp/sleep_demo_20250220-143820/step_1/MERLIN_STATUS.json... + [2025-02-20 14:42:35,946: INFO] Status for step_1 successfully written. + [2025-02-20 14:42:35,946: INFO] Step 'step_1' in '/usr/WS1/gunny/debug/temp/sleep_demo_20250220-143820/step_1' finished successfully. + [2025-02-20 14:42:36,028: INFO] Task merlin:chordfinisher[75d2720a-de21-473f-bce0-6f8ac0186b14] received + [2025-02-20 14:42:36,030: INFO] Task merlin.common.tasks.merlin_step[/usr/WS1/gunny/debug/temp/sleep_demo_20250220-143820/step_1] succeeded in 91.18941793194972s: 0 + [2025-02-20 14:42:36,061: INFO] Task merlin:mark_run_as_complete[0695381d-f9dd-4efb-b397-f47c752d5eb4] received + [2025-02-20 14:42:36,064: INFO] Task merlin:chordfinisher[75d2720a-de21-473f-bce0-6f8ac0186b14] succeeded in 0.03424098785035312s: 'SYNC' + [2025-02-20 14:42:36,093: INFO] Attempting to update run with id '8cd39508-ff77-409f-af7e-1642b95b753b'... + [2025-02-20 14:42:36,094: INFO] Successfully updated run with id '8cd39508-ff77-409f-af7e-1642b95b753b'. + [2025-02-20 14:42:36,107: INFO] Data successfully dumped to /usr/WS1/gunny/debug/temp/sleep_demo_20250220-143820/merlin_info/run_metadata.json. + [2025-02-20 14:42:36,110: INFO] Task merlin:mark_run_as_complete[0695381d-f9dd-4efb-b397-f47c752d5eb4] succeeded in 0.04832656914368272s: 'Run Completed' + [2025-02-20 14:42:37: INFO] Monitor: Run with workspace '/usr/WS1/gunny/debug/temp/sleep_demo_20250220-143820' has completed. Moving on to the next run. + [2025-02-20 14:42:37: INFO] Monitor: ... stop condition met ``` ### Steps +!!! danger "Deprecated" + + The `--steps` option is deprecated and set for removal in Merlin v1.14+. If you use this option in Merlin v1.13 you will see the same monitor functionality that existed in Merlin v1.12 and older. In other words, you will not have the auto-restart capability with the monitor. + !!! warning It's essential to note that using this option might lead to the termination of the allocation while your study is still processing. This outcome occurs if any subsequent steps in your study were not included in the steps provided to the `--steps` option. diff --git a/docs/user_guide/running_studies.md b/docs/user_guide/running_studies.md index 91215c19e..5ff9dbfbe 100644 --- a/docs/user_guide/running_studies.md +++ b/docs/user_guide/running_studies.md @@ -152,7 +152,7 @@ merlin stop-workers --spec my_specification.yaml !!! note - If you wish to execute a workflow after dry-running it, simply use [`merlin restart`](#restart-merlin-restart) (to understand why this works, see the section below on [Restarting Workflows](#restarting-workflows)). + If you wish to execute a workflow after dry-running it, simply use [`merlin restart`](./command_line.md#restart-merlin-restart) (to understand why this works, see the section below on [Restarting Workflows](#restarting-workflows)). 'Dry run' means telling workers to create a study's workspace and all of its necessary subdirectories and scripts (with variables expanded) without actually executing the scripts. diff --git a/docs/user_guide/specification.md b/docs/user_guide/specification.md index 53c682875..cabf16b1c 100644 --- a/docs/user_guide/specification.md +++ b/docs/user_guide/specification.md @@ -82,7 +82,7 @@ The `batch` block is an optional block that enables specification of HPC schedul | Property Name | Required? | Type | Description | | ------------- | --------- | ---- | ----------- | | `bank` | Yes | str | Account to charge computing time to | -| `dry_run` | No | bool | Execute a [dry run](./command_line.md#dry-run) of the study | +| `dry_run` | No | bool | Execute a [dry run](./running_studies.md#dry-runs) of the study | | `launch_args` | No | str | Extra arguments for the parallel launch command | | `launch_pre` | No | str | Any configuration needed before the scheduler launch command (`srun`, `jsrun`, etc.) | | `nodes` | No | int | The number of nodes to use for all workers. This can be overridden in [the `resources` property of the `merlin` block](#resources). If this is unset the number of nodes will be queried from the environment, failing that, the number of nodes will be set to 1. | diff --git a/lgtm.yml b/lgtm.yml deleted file mode 100644 index e3f53c87d..000000000 --- a/lgtm.yml +++ /dev/null @@ -1,25 +0,0 @@ -########################################################################################## -# Customize file classifications. # -# Results from files under any classifier will be excluded from LGTM # -# statistics. # -########################################################################################## - -########################################################################################## -# Use the `path_classifiers` block to define changes to the default classification of # -# files. # -########################################################################################## - -path_classifiers: - test: - # Classify all files in the top-level directories tests/ as test code. - - exclude: - - tests - - merlin/examples - -######################################################################################### -# Use the `queries` block to change the default display of query results. # -######################################################################################### - -queries: - # Specifically hide the results of clear-text-logging-sensitive-data - - exclude: py/clear-text-logging-sensitive-data diff --git a/merlin/__init__.py b/merlin/__init__.py index b6c065fd7..c4c11fe8b 100644 --- a/merlin/__init__.py +++ b/merlin/__init__.py @@ -1,32 +1,8 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ Merlin: Machine Learning for HPC Workflows. @@ -38,7 +14,7 @@ import sys -__version__ = "1.12.2" +__version__ = "1.13.0b1" VERSION = __version__ PATH_TO_PROJ = os.path.join(os.path.dirname(__file__), "") diff --git a/merlin/ascii_art.py b/merlin/ascii_art.py index bcada70af..1647bcc08 100644 --- a/merlin/ascii_art.py +++ b/merlin/ascii_art.py @@ -1,32 +1,8 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## # pylint: skip-file diff --git a/merlin/backends/__init__.py b/merlin/backends/__init__.py new file mode 100644 index 000000000..07e2b9645 --- /dev/null +++ b/merlin/backends/__init__.py @@ -0,0 +1,28 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Backend infrastructure for the Merlin application. + +The `backends` package provides a unified interface and implementations for persisting +and retrieving Merlin's core data entities (such as studies, runs, and workers) across +various storage technologies. It defines an abstract backend interface (`ResultsBackend`) +along with concrete implementations using Redis and SQLite, as well as utility functions, +store abstractions, and a backend factory. + +Subpackages: + redis: Redis-based backend implementation, including store classes and a Redis-backed `ResultsBackend`. + sqlite: SQLite-based backend implementation, with support for persistent local storage of Merlin data models. + +Modules: + backend_factory: Contains `MerlinBackendFactory`, used to dynamically select and instantiate a backend. + results_backend: Defines the abstract `ResultsBackend` base class for backend implementations. + store_base: Provides the abstract `StoreBase` class, the foundation for all store implementations. + utils: Utility functions for serialization, deserialization, and error handling across backends. + +This package serves as the backend layer in Merlin's architecture, enabling flexible, +pluggable data persistence with a consistent API regardless of the underlying storage engine. +""" diff --git a/merlin/backends/backend_factory.py b/merlin/backends/backend_factory.py new file mode 100644 index 000000000..b32841470 --- /dev/null +++ b/merlin/backends/backend_factory.py @@ -0,0 +1,90 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Backend factory for selecting and instantiating results backends in Merlin. + +This module defines the `MerlinBackendFactory` class, which serves as an abstraction +layer for managing available backend implementations. It supports dynamic selection +and instantiation of backend handlers such as Redis or SQLite, based on user input +or system configuration. + +The factory maintains mappings of backend names and aliases, and raises a clear error +if an unsupported backend is requested. +""" + +from typing import Dict, List + +from merlin.backends.redis.redis_backend import RedisBackend +from merlin.backends.results_backend import ResultsBackend +from merlin.backends.sqlite.sqlite_backend import SQLiteBackend +from merlin.exceptions import BackendNotSupportedError + + +# TODO add register_backend call to this when we create task server interface? +# TODO could this factory replace the functions in config/results_backend.py? +# - Perhaps it should be a class outside of this? +class MerlinBackendFactory: + """ + Factory class for managing and instantiating supported Merlin backends. + + This class maintains a registry of available results backends (e.g., Redis, SQLite) + and provides a unified interface for retrieving a backend implementation by name. + + Attributes: + _backends (Dict[str, backends.results_backend.ResultsBackend]): Mapping of backend names to their classes. + _backend_aliases (Dict[str, str]): Optional aliases for resolving canonical backend names. + + Methods: + get_supported_backends: Returns a list of supported backend names. + get_backend: Instantiates and returns the backend associated with the given name. + """ + + def __init__(self): + """Initialize the Merlin backend factory.""" + # Map canonical backend names to their classes + self._backends: Dict[str, ResultsBackend] = { + "redis": RedisBackend, + "sqlite": SQLiteBackend, + } + # Map aliases to canonical backend names + self._backend_aliases: Dict[str, str] = {"rediss": "redis"} + + def get_supported_backends(self) -> List[str]: + """ + Get a list of the supported backends in Merlin. + + Returns: + A list of names representing the supported backends in Merlin. + """ + return list(self._backends.keys()) + + def get_backend(self, backend: str) -> ResultsBackend: + """ + Get backend handler for whichever backend the user is using. + + Args: + backend: The name of the backend to load up. + + Returns: + An instantiation of a [`ResultsBackend`][backends.results_backend.ResultsBackend] object. + + Raises: + (exceptions.BackendNotSupportedError): If the requested backend is not supported. + """ + # Resolve the alias to the canonical backend name + backend = self._backend_aliases.get(backend, backend) + + # Get the correct backend class + backend_object = self._backends.get(backend) + + if backend_object is None: + raise BackendNotSupportedError(f"Backend unsupported by Merlin: {backend}.") + + return backend_object(backend) + + +backend_factory = MerlinBackendFactory() diff --git a/merlin/backends/redis/__init__.py b/merlin/backends/redis/__init__.py new file mode 100644 index 000000000..63c221933 --- /dev/null +++ b/merlin/backends/redis/__init__.py @@ -0,0 +1,19 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Redis-based backend infrastructure for the Merlin application. + +This package provides Redis-backed components for persisting and managing the core data +entities in Merlin (such as studies, runs, logical and physical workers). It includes +base store classes, concrete store implementations, and a full backend interface built +on top of Redis. + +Modules: + redis_backend: Implements the `ResultsBackend` interface using Redis. + redis_store_base: Provides shared base logic for Redis-backed stores. + redis_stores: Contains entity-specific Redis store classes and mixins. +""" diff --git a/merlin/backends/redis/redis_backend.py b/merlin/backends/redis/redis_backend.py new file mode 100644 index 000000000..7498446a5 --- /dev/null +++ b/merlin/backends/redis/redis_backend.py @@ -0,0 +1,106 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Redis backend implementation for the Merlin application. + +This module provides a concrete implementation of the `ResultsBackend` interface using Redis +as the underlying database. It defines the `RedisBackend` class, which manages interactions +with Redis-specific store classes for different data models, including CRUD operations and +database flushing. +""" + +import logging + +from redis import Redis + +from merlin.backends.redis.redis_stores import ( + RedisLogicalWorkerStore, + RedisPhysicalWorkerStore, + RedisRunStore, + RedisStudyStore, +) +from merlin.backends.results_backend import ResultsBackend + + +LOG = logging.getLogger("merlin") + + +# TODO might be able to make ResultsBackend classes replace the config/results_backend.py file +# - would help get a more OOP approach going within Merlin's codebase +# - instead of calling get_connection_string that logic could be handled in the base class? +class RedisBackend(ResultsBackend): + """ + A Redis-based implementation of the `ResultsBackend` interface for interacting with a Redis database. + + Attributes: + backend_name (str): The name of the backend (e.g., "redis"). + client (Redis): The Redis client used for database operations. + + Methods: + get_version: + Query Redis for the current version. + + get_connection_string: + Retrieve the connection string used to connect to Redis. + + flush_database: + Remove every entry in the Redis database. + + save: + Save a `BaseDataModel` entity to the Redis database. + + retrieve: + Retrieve an entity from the appropriate store based on the given query identifier and store type. + + retrieve_all: + Retrieve all entities from the specified store. + + delete: + Delete an entity from the specified store. + """ + + def __init__(self, backend_name: str): + """ + Initialize the `RedisBackend` instance, setting up the Redis client connection and store mappings. + + Args: + backend_name (str): The name of the backend (e.g., "redis"). + """ + from merlin.config.configfile import CONFIG # pylint: disable=import-outside-toplevel + from merlin.config.results_backend import get_connection_string # pylint: disable=import-outside-toplevel + + # Get the Redis client connection + redis_config = {"url": get_connection_string(), "decode_responses": True} + if CONFIG.results_backend.name == "rediss": + redis_config.update({"ssl_cert_reqs": getattr(CONFIG.results_backend, "cert_reqs", "required")}) + self.client: Redis = Redis.from_url(**redis_config) + + # Create instances of each store in our database + stores = { + "study": RedisStudyStore(self.client), + "run": RedisRunStore(self.client), + "logical_worker": RedisLogicalWorkerStore(self.client), + "physical_worker": RedisPhysicalWorkerStore(self.client), + } + + super().__init__(backend_name, stores) + + def get_version(self) -> str: + """ + Query the Redis backend for the current version. + + Returns: + A string representing the current version of Redis. + """ + client_info = self.client.info() + return client_info.get("redis_version", "N/A") + + def flush_database(self): + """ + Remove everything stored in Redis. + """ + self.client.flushdb() diff --git a/merlin/backends/redis/redis_store_base.py b/merlin/backends/redis/redis_store_base.py new file mode 100644 index 000000000..5b52515d3 --- /dev/null +++ b/merlin/backends/redis/redis_store_base.py @@ -0,0 +1,252 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Base Classes for Redis-Backed Data Stores in Merlin + +This module provides foundational classes for implementing Redis-based persistence +in the Merlin system. It defines reusable components that standardize how data models +are stored, retrieved, and managed in Redis. + +See also: + - merlin.backends.store_base: Base class + - merlin.backends.redis.redis_stores: Concrete store implementations + - merlin.db_scripts.data_models: Data model definitions +""" + +import logging +from typing import Generic, List, Optional, Type + +from redis import Redis + +from merlin.backends.store_base import StoreBase, T +from merlin.backends.utils import deserialize_entity, get_not_found_error_class, serialize_entity + + +LOG = logging.getLogger(__name__) + + +class RedisStoreBase(StoreBase[T], Generic[T]): + """ + Base class for Redis-based stores. + + This class provides common functionality for saving, retrieving, and deleting + entities in a Redis database. + + Attributes: + client (Redis): The Redis client used for database operations. + key (str): The prefix key used for Redis entries. + model_class (Type[T]): The model class used for deserialization. + + Methods: + save: Save or update an entity in the database. + retrieve: Retrieve an entity from the database by ID. + retrieve_all: Query the database for all entities of this type. + delete: Delete an entity from the database by ID. + """ + + def __init__(self, client: Redis, key: str, model_class: Type[T]): + """ + Initialize the Redis store with a Redis client. + + Args: + client: A Redis client instance used to interact with the Redis database. + key: The prefix key used for Redis entries. + model_class: The model class used for deserialization. + """ + self.client: Redis = client + self.key: str = key + self.model_class: Type[T] = model_class + + def _get_full_key(self, entity_id: str) -> str: + """ + Get the full Redis key for an entity. + + Args: + entity_id: The entity ID. + + Returns: + The full Redis key. + """ + return entity_id if entity_id.startswith(f"{self.key}:") else f"{self.key}:{entity_id}" + + def save(self, entity: T): + """ + Save or update an entity in the Redis database. + + Args: + entity: The entity to save. + """ + entity_key = f"{self.key}:{entity.id}" + + if self.client.exists(entity_key): + LOG.debug(f"Attempting to update {self.key} with id '{entity.id}'...") + # Get the existing data from Redis and convert it to an instance of BaseDataModel + existing_data = self.client.hgetall(entity_key) + existing_data_class = deserialize_entity(existing_data, self.model_class) + + # Update the fields and save it to Redis + existing_data_class.update_fields(entity.to_dict()) + updated_data = serialize_entity(existing_data_class) + self.client.hset(entity_key, mapping=updated_data) + LOG.debug(f"Successfully updated {self.key} with id '{entity.id}'.") + else: + LOG.debug(f"Creating a {self.key} entry in Redis...") + serialized_data = serialize_entity(entity) + self.client.hset(entity_key, mapping=serialized_data) + LOG.debug(f"Successfully created a {self.key} with id '{entity.id}' in Redis.") + + def retrieve(self, identifier: str) -> Optional[T]: + """ + Retrieve an entity from the Redis database by ID. + + Args: + identifier: The ID of the entity to retrieve. + + Returns: + The entity if found, None otherwise. + """ + LOG.debug(f"Retrieving identifier {identifier} in RedisStoreBase.") + entity_key = self._get_full_key(identifier) + if not self.client.exists(entity_key): + return None + + data_from_redis = self.client.hgetall(entity_key) + return deserialize_entity(data_from_redis, self.model_class) + + def retrieve_all(self) -> List[T]: + """ + Query the Redis database for all entities of this type. + + Returns: + A list of entities. + """ + entity_type = f"{self.key}s" if self.key != "study" else "studies" + LOG.info(f"Fetching all {entity_type} from Redis...") + + pattern = f"{self.key}:*" + all_entities = [] + + # Exclude name mapping key if it exists + keys_to_exclude = {f"{self.key}:name"} + + # Loop through all entities using scan_iter for better efficiency with large datasets + for key in self.client.scan_iter(match=pattern): + if key in keys_to_exclude: + continue + + entity_id = key.split(":")[1] # Extract the ID for logging + try: + entity_info = self.retrieve(key) + if entity_info: + all_entities.append(entity_info) + else: + LOG.warning(f"{self.key.capitalize()} with id '{entity_id}' could not be retrieved or does not exist.") + except Exception as exc: # pylint: disable=broad-except + LOG.error(f"Error retrieving {self.key} with id '{entity_id}': {exc}") + + LOG.info(f"Successfully retrieved {len(all_entities)} {entity_type} from Redis.") + return all_entities + + def delete(self, identifier: str): + """ + Delete an entity from the Redis database by ID. + + Args: + identifier: The ID of the entity to delete. + """ + LOG.info(f"Attempting to delete {self.key} with id '{identifier}' from Redis...") + + entity = self.retrieve(identifier) + if entity is None: + error_class = get_not_found_error_class(self.model_class) + raise error_class(f"{self.key.capitalize()} with id '{identifier}' does not exist in the database.") + + # Delete the entity's hash entry + entity_key = f"{self.key}:{entity.id}" + LOG.debug(f"Deleting {self.key} hash with key '{entity_key}'...") + self.client.delete(entity_key) + LOG.debug(f"Successfully removed {self.key} hash from Redis.") + + LOG.info(f"Successfully deleted {self.key} '{identifier}' from Redis.") + + +class NameMappingMixin: + """ + Mixin class that adds name-to-ID mapping functionality to Redis stores. + + This mixin extends Redis stores to support retrieval and deletion by name. + + Methods: + save: Save or update an entity in the database. + retrieve: Retrieve an entity from the database by ID or name. + delete: Delete an entity from the database by ID or name. + """ + + def save(self, entity: Type[T]): + """ + Save an entity and update the name-to-ID mapping. + + Args: + entity: The entity to save. + """ + name_or_ws = entity.name if hasattr(entity, "name") else entity.workspace + LOG.debug(f"Saving entity {name_or_ws} with id {entity.id} in NameMappingMixin.") + existing_entity_id = self.client.hget(f"{self.key}:name", name_or_ws) + + # Call the parent class's save method + super().save(entity) + + # Update name-to-ID mapping if it's a new entity + if not existing_entity_id: + LOG.debug(f"Creating a new name-to-ID mapping for {name_or_ws} with id {entity.id}") + self.client.hset(f"{self.key}:name", name_or_ws, entity.id) + + def retrieve(self, identifier: str, by_name: bool = False) -> Optional[T]: + """ + Retrieve an entity from the Redis database, either by ID or name. + + Args: + identifier: The ID or name of the entity to retrieve. + by_name: If True, interpret the identifier as a name. If False, interpret it as an ID. + + Returns: + The entity if found, None otherwise. + """ + LOG.debug(f"Retrieving identifier {identifier} in NameMappingMixin.") + if by_name: + # Retrieve the entity ID using the name-to-ID mapping + entity_id = self.client.hget(f"{self.key}:name", identifier) + if entity_id is None: + LOG.debug("Could not retrieve entity id by name-to-ID mapping.") + return None + return super().retrieve(entity_id) + # Use the parent class's retrieve method for ID-based retrieval + return super().retrieve(identifier) + + def delete(self, identifier: str, by_name: bool = False): + """ + Delete an entity from the Redis database, either by ID or name. + + Args: + identifier: The ID or name of the entity to delete. + by_name: If True, interpret the identifier as a name. If False, interpret it as an ID. + """ + id_type = "name" if by_name else "id" + LOG.debug(f"Attempting to delete {self.key} with {id_type} '{identifier}' from Redis...") + + # Retrieve the entity to ensure it exists and get its ID and name + entity = self.retrieve(identifier, by_name=by_name) + if entity is None: + error_class = get_not_found_error_class(self.model_class) + raise error_class(f"{self.key.capitalize()} with {id_type} '{identifier}' not found in the database.") + + # Delete the entity from the name index and Redis + name_or_ws = entity.name if hasattr(entity, "name") else entity.workspace + self.client.hdel(f"{self.key}:name", name_or_ws) + self.client.delete(f"{self.key}:{entity.id}") + + LOG.info(f"Successfully deleted {self.key} with {id_type} '{identifier}'.") diff --git a/merlin/backends/redis/redis_stores.py b/merlin/backends/redis/redis_stores.py new file mode 100644 index 000000000..e8e0d1abf --- /dev/null +++ b/merlin/backends/redis/redis_stores.py @@ -0,0 +1,91 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Redis Store Implementations for Merlin Data Models + +This module provides Redis-backed storage implementations for various Merlin system models. +These store classes offer persistence, retrieval, and management capabilities for core +system entities like studies, runs, and workers. + +Each store implements a consistent interface through inheritance from RedisStoreBase, +with specialized functionality added through mixins (like NameMappingMixin for +name-based lookups). The stores handle serialization/deserialization, CRUD operations, +and maintain appropriate Redis key structures. + +See also: + - merlin.backends.redis.redis_base_store: Base classes and mixins + - merlin.db_scripts.data_models: Data model definitions +""" + +from redis import Redis + +from merlin.backends.redis.redis_store_base import NameMappingMixin, RedisStoreBase +from merlin.db_scripts.data_models import LogicalWorkerModel, PhysicalWorkerModel, RunModel, StudyModel + + +class RedisLogicalWorkerStore(RedisStoreBase[LogicalWorkerModel]): + """ + A Redis-based store for managing [`LogicalWorkerModel`][db_scripts.data_models.LogicalWorkerModel] + objects. + """ + + def __init__(self, client: Redis): + """ + Initialize the `RedisLogicalWorkerStore` with a Redis client. + + Args: + client: A Redis client instance used to interact with the Redis database. + """ + super().__init__(client, "logical_worker", LogicalWorkerModel) + + +class RedisPhysicalWorkerStore(NameMappingMixin, RedisStoreBase[PhysicalWorkerModel]): + """ + A Redis-based store for managing [`PhysicalWorkerModel`][db_scripts.data_models.PhysicalWorkerModel] + objects. + """ + + def __init__(self, client: Redis): + """ + Initialize the `RedisPhysicalWorkerStore` with a Redis client. + + Args: + client: A Redis client instance used to interact with the Redis database. + """ + super().__init__(client, "physical_worker", PhysicalWorkerModel) + + +class RedisRunStore(NameMappingMixin, RedisStoreBase[RunModel]): + """ + A Redis-based store for managing [`RunModel`][db_scripts.data_models.RunModel] + objects. + """ + + def __init__(self, client: Redis): + """ + Initialize the `RedisRunStore` with a Redis client. + + Args: + client: A Redis client instance used to interact with the Redis database. + """ + super().__init__(client, "run", RunModel) + + +class RedisStudyStore(NameMappingMixin, RedisStoreBase[StudyModel]): + """ + A Redis-based store for managing [`StudyModel`][db_scripts.data_models.StudyModel] + objects. + """ + + def __init__(self, client: Redis): + """ + Initialize the `RedisStudyStore` with a Redis client. + + Args: + client: A Redis client instance used to interact with the Redis database. + """ + super().__init__(client, "study", StudyModel) diff --git a/merlin/backends/results_backend.py b/merlin/backends/results_backend.py new file mode 100644 index 000000000..ab85350ca --- /dev/null +++ b/merlin/backends/results_backend.py @@ -0,0 +1,239 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Abstract base class for results backends in the Merlin application. + +This module defines `ResultsBackend`, an abstract base class that specifies +the required interface for backend implementations responsible for persisting +and retrieving data in Merlin. + +The `ResultsBackend` class encapsulates: +- A unified interface for saving, retrieving, and deleting Merlin data models +- Store type routing for specific model categories (study, run, logical worker, physical worker) +- Support for backend-specific configuration such as version reporting and database flushing + +Usage: + This base class is not meant to be instantiated directly. Instead, it should be subclassed + by backend-specific implementations such as `RedisBackend` or `SQLiteBackend`. +""" + +import logging +import uuid +from abc import ABC, abstractmethod +from typing import Dict, List + +from merlin.backends.store_base import StoreBase +from merlin.db_scripts.data_models import BaseDataModel, LogicalWorkerModel, PhysicalWorkerModel, RunModel, StudyModel +from merlin.exceptions import UnsupportedDataModelError + + +LOG = logging.getLogger(__name__) + + +class ResultsBackend(ABC): + """ + Abstract base class for a results backend, which provides methods to save and retrieve + information from a backend database. + + This class defines the interface that must be implemented by any concrete backend, and serves + as the foundation for interacting with various backend types such as Redis, PostgreSQL, etc. + + Attributes: + backend_name (str): The name of the backend (e.g., "redis", "postgresql"). + stores (Dict[str, backends.store_base.StoreBase]): A dictionary of stores that each concrete + implementation of this class will need to define. + + Methods: + get_name: + Retrieve the name of the backend. + + get_version: + Query the backend for the current version. + + get_connection_string: + Retrieve the connection string used to connect to the backend. + + flush_database: + Remove every entry in the database. + + save: + Save an entity (e.g., a study, run, or worker) to the backend database. + + retrieve: + Retrieve an entity from the backend database using its identifier and store type. + + retrieve_all: + Retrieve all entities of a specific type from the backend database. + + delete: + Delete an entity from the backend database using its identifier and store type. + """ + + def __init__(self, backend_name: str, stores: Dict[str, StoreBase]): + """ + Initialize the `ResultsBackend` instance. + + Args: + backend_name: The name of the backend (e.g., "redis"). + """ + self.backend_name: str = backend_name + self.stores: Dict[str, StoreBase] = stores + + def get_name(self) -> str: + """ + Get the name of the backend. + + Returns: + The name of the backend (e.g. redis). + """ + return self.backend_name + + @abstractmethod + def get_version(self) -> str: + """ + Query the backend for the current version. + + Returns: + A string representing the current version of the backend. + """ + raise NotImplementedError("Subclasses of `ResultsBackend` must implement a `get_version` method.") + + def get_connection_string(self) -> str: + """ + Query the backend for the connection string. + + Returns: + A string representing the connection to the backend. + """ + from merlin.config.results_backend import get_connection_string # pylint: disable=import-outside-toplevel + + return get_connection_string(include_password=False) + + @abstractmethod + def flush_database(self): + """ + Remove everything stored in the database. + """ + raise NotImplementedError("Subclasses of `ResultsBackend` must implement a `flush_database` method.") + + def _get_store_by_type(self, store_type: str) -> StoreBase: + """ + Get the appropriate store based on the store type. + + Args: + store_type (str): The type of store. + + Returns: + The corresponding store. + + Raises: + ValueError: If the `store_type` is invalid. + """ + if store_type not in self.stores: + raise ValueError(f"Invalid store type '{store_type}'.") + return self.stores[store_type] + + def _get_store_by_entity(self, entity: BaseDataModel) -> StoreBase: + """ + Get the appropriate store based on the entity type. + + Args: + entity (BaseDataModel): The entity to save. + + Returns: + RedisStore: The corresponding store. + + Raises: + UnsupportedDataModelError: If the entity type is unsupported. + """ + if isinstance(entity, StudyModel): + return self.stores["study"] + if isinstance(entity, RunModel): + return self.stores["run"] + if isinstance(entity, LogicalWorkerModel): + return self.stores["logical_worker"] + if isinstance(entity, PhysicalWorkerModel): + return self.stores["physical_worker"] + raise UnsupportedDataModelError(f"Unsupported data model of type {type(entity)}.") + + def save(self, entity: BaseDataModel): + """ + Save a `BaseDataModel` object to the Redis database. + + Args: + entity (BaseDataModel): An instance of one of `BaseDataModel`'s inherited classes. + + Raises: + UnsupportedDataModelError: If the entity type is unsupported. + """ + store = self._get_store_by_entity(entity) + store.save(entity) + + def retrieve(self, entity_identifier: str, store_type: str) -> BaseDataModel: + """ + Retrieve an object from the appropriate store based on the given query identifier and store type. + + Args: + entity_identifier (str): The identifier used to query the store, either an ID (UUID) or a name. + store_type (str): The type of store to query. Valid options are: + - `study` + - `run` + - `logical_worker` + - `physical_worker` + + Returns: + The object retrieved from the specified store. + """ + LOG.debug(f"Retrieving '{entity_identifier}' from store '{store_type}'.") + store = self._get_store_by_type(store_type) + if store_type in ["study", "physical_worker", "run"]: + try: + uuid.UUID(entity_identifier) + return store.retrieve(entity_identifier) + except ValueError: + return store.retrieve(entity_identifier, by_name=True) + else: + return store.retrieve(entity_identifier) + + def retrieve_all(self, store_type: str) -> List[BaseDataModel]: + """ + Retrieve all objects from the specified store. + + Args: + store_type (str): The type of store to query. Valid options are: + - `study` + - `run` + - `logical_worker` + - `physical_worker` + + Returns: + A list of objects retrieved from the specified store. + """ + store = self._get_store_by_type(store_type) + return store.retrieve_all() + + def delete(self, entity_identifier: str, store_type: str): + """ + Delete an entity from the specified store. + + Args: + entity_identifier (str): The identifier of the entity to delete. + store_type (str): The type of store to query. Valid options are: + - `study` + - `run` + - `logical_worker` + - `physical_worker` + """ + store = self._get_store_by_type(store_type) + if store_type in ["study", "physical_worker"]: + try: + uuid.UUID(entity_identifier) + store.delete(entity_identifier) + except ValueError: + store.delete(entity_identifier, by_name=True) + else: + store.delete(entity_identifier) diff --git a/merlin/backends/sqlite/__init__.py b/merlin/backends/sqlite/__init__.py new file mode 100644 index 000000000..c9e70aa0a --- /dev/null +++ b/merlin/backends/sqlite/__init__.py @@ -0,0 +1,21 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +SQLite-based backend infrastructure for the Merlin application. + +This package provides all components necessary to persist and manage Merlin's core data +entities (such as studies, runs, and workers) using SQLite as the storage backend. +It includes a generic store base, specialized store classes for each entity type, +connection handling logic, and a complete backend implementation conforming to +Merlin's backend interface. + +Modules: + sqlite_backend: Implements the `ResultsBackend` interface using SQLite. + sqlite_connection: Provides a context-managed SQLite connection with safe configuration. + sqlite_store_base: Defines a generic base class for entity stores. + sqlite_stores: Contains concrete SQLite store classes for Merlin models. +""" diff --git a/merlin/backends/sqlite/sqlite_backend.py b/merlin/backends/sqlite/sqlite_backend.py new file mode 100644 index 000000000..4adacfcf2 --- /dev/null +++ b/merlin/backends/sqlite/sqlite_backend.py @@ -0,0 +1,107 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +SQLite backend implementation for the Merlin application. + +This module provides a concrete implementation of the `ResultsBackend` interface using SQLite +as the underlying database. It defines the `SQLiteBackend` class, which manages interactions +with SQLite-specific store classes for different data models, including schema initialization, +CRUD operations, and database flushing. +""" + +import logging + +from merlin.backends.results_backend import ResultsBackend +from merlin.backends.sqlite.sqlite_connection import SQLiteConnection +from merlin.backends.sqlite.sqlite_stores import ( + SQLiteLogicalWorkerStore, + SQLitePhysicalWorkerStore, + SQLiteRunStore, + SQLiteStudyStore, +) + + +LOG = logging.getLogger(__name__) + + +class SQLiteBackend(ResultsBackend): + """ + A SQLite-based implementation of the `ResultsBackend` interface for interacting with a SQLite database. + + Attributes: + backend_name (str): The name of the backend (e.g., "sqlite"). + + Methods: + get_version: + Query SQLite for the current version. + + get_connection_string: + Retrieve the connection string (file path) used to connect to SQLite. + + flush_database: + Remove every entry in the SQLite database by dropping and recreating tables. + + save: + Save a `BaseDataModel` entity to the SQLite database. + + retrieve: + Retrieve an entity from the appropriate store based on the given query identifier and store type. + + retrieve_all: + Retrieve all entities from the specified store. + + delete: + Delete an entity from the specified store. + """ + + def __init__(self, backend_name: str): + """ + Initialize the `SQLiteBackend` instance, setting up the store mappings and tables. + + Args: + backend_name (str): The name of the backend (e.g., "sqlite"). + """ + stores = { + "study": SQLiteStudyStore(), + "run": SQLiteRunStore(), + "logical_worker": SQLiteLogicalWorkerStore(), + "physical_worker": SQLitePhysicalWorkerStore(), + } + + super().__init__(backend_name, stores) + + # Initialize database schema + self._initialize_schema() + + def _initialize_schema(self): + """Initialize the database schema by creating all necessary tables.""" + for store in self.stores.values(): + store.create_table_if_not_exists() + + def get_version(self) -> str: + """ + Query SQLite for the current version. + + Returns: + The SQLite version string. + """ + with SQLiteConnection() as conn: + cursor = conn.execute("SELECT sqlite_version()") + return cursor.fetchone()[0] + + def flush_database(self): + """ + Remove every entry in the SQLite database by dropping and recreating tables. + """ + with SQLiteConnection() as conn: + # Get all table names + cursor = conn.execute("SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'") + tables = [row[0] for row in cursor.fetchall()] + + # Drop all tables + for table in tables: + conn.execute(f"DROP TABLE IF EXISTS {table}") diff --git a/merlin/backends/sqlite/sqlite_connection.py b/merlin/backends/sqlite/sqlite_connection.py new file mode 100644 index 000000000..1d77ebef5 --- /dev/null +++ b/merlin/backends/sqlite/sqlite_connection.py @@ -0,0 +1,96 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +SQLite connection context manager for the Merlin application. + +This module defines the `SQLiteConnection` class, which provides a safe and reusable way to +establish and manage SQLite connections using a context manager. It ensures proper configuration +(e.g., enabling WAL mode and foreign key support), handles compatibility with Python versions, +and guarantees cleanup by closing the connection on exit. +""" + +import logging +import sqlite3 +import sys +from pathlib import Path +from types import TracebackType +from typing import Type + + +LOG = logging.getLogger(__name__) + + +class SQLiteConnection: + """ + Context manager for establishing and safely closing a SQLite database connection. + + This class ensures SQLite connections are created with proper configuration, including: + - WAL mode for better concurrency + - Foreign key constraint enforcement + - Dictionary-style row access via `sqlite3.Row` + - Compatibility with Python versions < 3.12 and ≥ 3.12 regarding autocommit + + Attributes: + conn (sqlite3.Connection): The active SQLite connection used within the context. + + Methods: + __enter__: + Opens and configures the SQLite connection when entering the context. + + __exit__: + Closes the SQLite connection when exiting the context, handling any exceptions. + """ + + def __init__(self): + """Initialize the SQLiteConnection context manager.""" + self.conn: sqlite3.Connection = None + + def __enter__(self) -> sqlite3.Connection: + """ + Enters the runtime context related to this object and creates a sqlite connection. + + Returns: + A sqlite connection. + """ + from merlin.config.results_backend import get_connection_string # pylint: disable=import-outside-toplevel + + # Get the SQLite database path and ensure the path exists + db_path = get_connection_string() + Path(db_path).parent.mkdir(parents=True, exist_ok=True) + + connection_kwargs = {"check_same_thread": False} + if sys.version_info < (3, 12): # Autocommit wasn't added until python 3.12 + connection_kwargs["isolation_level"] = None + else: + connection_kwargs["autocommit"] = True + + # Create SQLite connection with optimized settings + self.conn = sqlite3.connect(db_path, **connection_kwargs) + + # Enable WAL mode for better concurrent access + self.conn.execute("PRAGMA journal_mode=WAL") + # Enable foreign key constraints + self.conn.execute("PRAGMA foreign_keys=ON") + + # This enables name-based access to columns + self.conn.row_factory = sqlite3.Row + + return self.conn + + def __exit__(self, exc_type: Type[Exception], exc_value: Exception, traceback: TracebackType): + """ + Exits the runtime context and performs cleanup. + + This method closes the connection if it's still open. + + Args: + exc_type: The exception type raised, if any. + exc_value: The exception instance raised, if any. + traceback: The traceback object, if an exception was raised. + """ + if self.conn: + self.conn.close() diff --git a/merlin/backends/sqlite/sqlite_store_base.py b/merlin/backends/sqlite/sqlite_store_base.py new file mode 100644 index 000000000..e1b2ddc93 --- /dev/null +++ b/merlin/backends/sqlite/sqlite_store_base.py @@ -0,0 +1,232 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +SQLite-based generic store implementation for Merlin entities. + +This module defines `SQLiteStoreBase`, a generic base class for managing +entity persistence using SQLite as the underlying storage. It provides +core CRUD operations (create, retrieve, update, delete) and dynamic table +creation based on model class field definitions. + +This module is intended to be subclassed by entity-specific store classes +in the Merlin backend architecture. + +See also: + - merlin.backends.store_base: Base class + - merlin.backends.sqlite.sqlite_stores: Concrete store implementations + - merlin.db_scripts.data_models: Data model definitions +""" + +import logging +from datetime import datetime +from typing import Any, Generic, List, Optional, Type + +from merlin.backends.sqlite.sqlite_connection import SQLiteConnection +from merlin.backends.store_base import StoreBase, T +from merlin.backends.utils import deserialize_entity, get_not_found_error_class, serialize_entity + + +LOG = logging.getLogger(__name__) + + +class SQLiteStoreBase(StoreBase[T], Generic[T]): + """ + Base class for SQLite-based stores. + + This class provides common functionality for saving, retrieving, and deleting + entities in a SQLite database. + + Attributes: + table_name (str): The table name used for SQLite entries. + model_class (Type[T]): The model class used for deserialization. + + Methods: + save: Save or update an entity in the database. + retrieve: Retrieve an entity from the database by ID. + retrieve_all: Query the database for all entities of this type. + delete: Delete an entity from the database by ID. + """ + + def __init__(self, table_name: str, model_class: Type[T]): + """ + Initialize the SQLite store with a SQLite connection. + + Args: + table_name: The table name used for SQLite entries. + model_class: The model class used for deserialization. + """ + self.table_name: str = table_name + self.model_class: Type[T] = model_class + self.create_table_if_not_exists() + + def _get_sqlite_type(self, py_type: Any) -> str: + """ + Map Python types to SQLite types. + + Args: + py_type: A Python type hint (e.g., str, int, List[str], etc.) + + Returns: + A string representing the corresponding SQLite column type. + """ + origin_type = getattr(py_type, "__origin__", py_type) + result = "TEXT" # Default fallback + + # Handle generics like List[str], Dict[str, Any], etc. + if origin_type in (list, dict, set): + result = "TEXT" # store as JSON string + elif py_type == str: + result = "TEXT" + elif py_type == int: + result = "INTEGER" + elif py_type == float: + result = "REAL" + elif py_type == bool: + result = "INTEGER" # SQLite uses 0 and 1 for booleans + elif py_type == datetime: + result = "TEXT" # ISO format string + + return result + + def create_table_if_not_exists(self): + """ + Create the table if it doesn't exist. + """ + field_defs = [] + + for field_obj in self.model_class.get_class_fields(): + col_name = field_obj.name + col_type = self._get_sqlite_type(field_obj.type) + field_defs.append(f"{col_name} {col_type}") + + field_defs_str = ", ".join(field_defs) + + with SQLiteConnection() as conn: + conn.execute(f"CREATE TABLE IF NOT EXISTS {self.table_name} ({field_defs_str});") + + def save(self, entity: T): + """ + Save or update an entity in the SQLite database. + + Args: + entity: The entity to save. + """ + # Try to retrieve any existing entity with this ID + existing_data = self.retrieve(entity.id) + + # If the entity already exists, update it + if existing_data: + LOG.debug(f"Attempting to update {self.table_name} with id '{entity.id}'...") + existing_data.update_fields(entity.to_dict()) + serialized_data = serialize_entity(existing_data) + set_str = ", ".join( + f"{field.name} = :{field.name}" for field in self.model_class.get_class_fields() if field.name != "id" + ) + with SQLiteConnection() as conn: + conn.execute( + f""" + UPDATE {self.table_name} + SET {set_str} + WHERE id = :id + """, + serialized_data, + ) + LOG.debug(f"Successfully updated {self.table_name} with id '{entity.id}'.") + # If the entity does not already exist, create it + else: + LOG.debug(f"Creating a {self.table_name} entry in SQLite...") + serialized_data = serialize_entity(entity) + fields = [field.name for field in self.model_class.get_class_fields()] + columns_str = ", ".join(fields) + placeholders_str = ", ".join(f":{name}" for name in fields) + with SQLiteConnection() as conn: + conn.execute( + f""" + INSERT INTO {self.table_name} ({columns_str}) + VALUES ({placeholders_str}) + """, + serialized_data, + ) + LOG.debug(f"Successfully created a {self.table_name} with id '{entity.id}' in SQLite.") + + def retrieve(self, identifier: str, by_name: bool = False) -> Optional[T]: + """ + Retrieve an entity from the SQLite database by ID or name. + + Args: + identifier: The ID or name of the entity to retrieve. + by_name: If True, interpret the identifier as a name. If False, interpret it as an ID. + + Returns: + The entity if found, None otherwise. + """ + LOG.debug(f"Retrieving identifier {identifier} in SQLiteStoreBase.") + + id_type = "name" if by_name else "id" + + with SQLiteConnection() as conn: + cursor = conn.execute(f"SELECT * FROM {self.table_name} WHERE {id_type} = :identifier", {"identifier": identifier}) + row = cursor.fetchone() + + if row is None: + return None + + return deserialize_entity(dict(row), self.model_class) + + def retrieve_all(self) -> List[T]: + """ + Query the SQLite database for all entities of this type. + + Returns: + A list of entities. + """ + entity_type = f"{self.table_name}s" if self.table_name != "study" else "studies" + LOG.info(f"Fetching all {entity_type} from SQLite...") + + with SQLiteConnection() as conn: + cursor = conn.execute(f"SELECT * FROM {self.table_name}") + all_entities = [] + + for row in cursor.fetchall(): + try: + entity = deserialize_entity(dict(row), self.model_class) + if entity: + all_entities.append(entity) + else: + LOG.warning( + f"{self.table_name.capitalize()} with id '{row['id']}' could not be retrieved or does not exist." + ) + except Exception as exc: # pylint: disable=broad-except + LOG.error(f"Error retrieving {self.table_name} with id '{row['id']}': {exc}") + + LOG.info(f"Successfully retrieved {len(all_entities)} {entity_type} from SQLite.") + return all_entities + + def delete(self, identifier: str, by_name: bool = False): + """ + Delete an entity from the SQLite database by ID or name. + + Args: + identifier: The ID or name of the entity to delete. + by_name: If True, interpret the identifier as a name. If False, interpret it as an ID. + """ + id_type = "name" if by_name else "id" + LOG.info(f"Attempting to delete {self.table_name} with {id_type} '{identifier}' from SQLite...") + + entity = self.retrieve(identifier, by_name=by_name) + if entity is None: + error_class = get_not_found_error_class(self.model_class) + raise error_class(f"{self.table_name.capitalize()} with {id_type} '{identifier}' does not exist in the database.") + + # Delete the entity + with SQLiteConnection() as conn: + cursor = conn.execute(f"DELETE FROM {self.table_name} WHERE {id_type} = :identifier", {"identifier": identifier}) + + if cursor.rowcount == 0: + LOG.warning(f"No rows were deleted for {self.table_name} with {id_type} '{identifier}'") + else: + LOG.info(f"Successfully deleted {self.table_name} '{identifier}' from SQLite.") diff --git a/merlin/backends/sqlite/sqlite_stores.py b/merlin/backends/sqlite/sqlite_stores.py new file mode 100644 index 000000000..53d65f9a8 --- /dev/null +++ b/merlin/backends/sqlite/sqlite_stores.py @@ -0,0 +1,65 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +SQLite store implementations for Merlin entity models. + +This module defines concrete `SQLiteStoreBase` subclasses for managing the +persistence of core entity models used in the Merlin application. Each store +is bound to a specific model and SQLite table, providing CRUD operations and +automated table creation. + +See also: + - merlin.backends.sqlite.sqlite_base_store: Base class + - merlin.db_scripts.data_models: Data model definitions +""" + +from merlin.backends.sqlite.sqlite_store_base import SQLiteStoreBase +from merlin.db_scripts.data_models import LogicalWorkerModel, PhysicalWorkerModel, RunModel, StudyModel + + +class SQLiteLogicalWorkerStore(SQLiteStoreBase[LogicalWorkerModel]): + """ + A SQLite-based store for managing [`LogicalWorkerModel`][db_scripts.data_models.LogicalWorkerModel] + objects. + """ + + def __init__(self): + """Initialize the `SQLiteLogicalWorkerStore`.""" + super().__init__("logical_worker", LogicalWorkerModel) + + +class SQLitePhysicalWorkerStore(SQLiteStoreBase[PhysicalWorkerModel]): + """ + A SQLite-based store for managing [`PhysicalWorkerModel`][db_scripts.data_models.PhysicalWorkerModel] + objects. + """ + + def __init__(self): + """Initialize the `SQLitePhysicalWorkerStore`.""" + super().__init__("physical_worker", PhysicalWorkerModel) + + +class SQLiteRunStore(SQLiteStoreBase[RunModel]): + """ + A SQLite-based store for managing [`RunModel`][db_scripts.data_models.RunModel] + objects. + """ + + def __init__(self): + """Initialize the `SQLiteRunStore`.""" + super().__init__("run", RunModel) + + +class SQLiteStudyStore(SQLiteStoreBase[StudyModel]): + """ + A SQLite-based store for managing [`StudyModel`][db_scripts.data_models.StudyModel] + objects. + """ + + def __init__(self): + """Initialize the `SQLiteStudyStore`.""" + super().__init__("study", StudyModel) diff --git a/merlin/backends/store_base.py b/merlin/backends/store_base.py new file mode 100644 index 000000000..9dc9a452d --- /dev/null +++ b/merlin/backends/store_base.py @@ -0,0 +1,78 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module defines the abstract base class for all data store implementations in Merlin. + +This module provides the `StoreBase` class, which outlines the required interface for saving, +retrieving, listing, and deleting entities from a backing data store. All concrete store classes +(e.g., Redis-based stores) must inherit from this class and implement its abstract methods. +""" + +from abc import ABC, abstractmethod +from typing import Generic, List, TypeVar + +from merlin.db_scripts.data_models import BaseDataModel + + +T = TypeVar("T", bound=BaseDataModel) + + +class StoreBase(ABC, Generic[T]): + """ + Base class for all stores supported in Merlin. + + This class defines the methods that are needed for each store in Merlin. + + Methods: + save: Save or update an object in the database. + retrieve: Retrieve an entity from the database by ID. + retrieve_all: Query the database for all entities of this type. + delete: Delete an entity from the database by ID. + """ + + @abstractmethod + def save(self, entity: BaseDataModel): + """ + Save or update an object in the database. + + Args: + entity: The object to save. + """ + raise NotImplementedError("Subclasses of `StoreBase` must implement a `save` method.") + + @abstractmethod + def retrieve(self, identifier: str) -> BaseDataModel: + """ + Retrieve an entity from the database by an identifier. + + Args: + identifier: The identifier (typically ID or name) of the entity to retrieve. + + Returns: + The entity if found, None otherwise. + """ + raise NotImplementedError("Subclasses of `StoreBase` must implement a `retrieve` method.") + + @abstractmethod + def retrieve_all(self) -> List[BaseDataModel]: + """ + Query the database for all entities of this type. + + Returns: + A list of entities. + """ + raise NotImplementedError("Subclasses of `StoreBase` must implement a `retrieve_all` method.") + + @abstractmethod + def delete(self, identifier: str): + """ + Delete an entity from the database by an identifier. + + Args: + identifier: The identifier (typically ID or name) of the entity to delete. + """ + raise NotImplementedError("Subclasses of `StoreBase` must implement a `delete` method.") diff --git a/merlin/backends/utils.py b/merlin/backends/utils.py new file mode 100644 index 000000000..7f3ae9bf2 --- /dev/null +++ b/merlin/backends/utils.py @@ -0,0 +1,137 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Utility functions for backends in the Merlin application. + +These utilities are essential for converting in-memory data models into a persistable format, +ensuring compatibility with backend storage systems, and for consistent error handling +when entity lookups fail. +""" + +import json +import logging +from datetime import datetime +from typing import Dict, Type, TypeVar + +from merlin.db_scripts.data_models import LogicalWorkerModel, PhysicalWorkerModel, RunModel, StudyModel +from merlin.exceptions import RunNotFoundError, StudyNotFoundError, WorkerNotFoundError + + +T = TypeVar("T") + +LOG = logging.getLogger(__name__) + + +def get_not_found_error_class(model_class: Type[T]) -> Exception: + """ + Get the appropriate not found error class based on the model type. + + Args: + model_class: A [`BaseDataModel`][db_scripts.data_models.BaseDataModel] subclass. + + Returns: + The error class to use. + """ + error_map = { + LogicalWorkerModel: WorkerNotFoundError, + PhysicalWorkerModel: WorkerNotFoundError, + RunModel: RunNotFoundError, + StudyModel: StudyNotFoundError, + } + return error_map.get(model_class, Exception) + + +def is_iso_datetime(value: str) -> bool: + """ + Check if a string is in ISO 8601 datetime format. + + Args: + value: The string to check. + + Returns: + True if the string is in ISO 8601 format, False otherwise. + """ + try: + datetime.fromisoformat(value) + return True + except ValueError: + return False + + +def serialize_entity(entity: T) -> Dict[str, str]: + """ + Given a [`BaseDataModel`][db_scripts.data_models.BaseDataModel] instance, + convert it's data into a format that the database can interpret. + + Args: + entity: A [`BaseDataModel`][db_scripts.data_models.BaseDataModel] instance. + + Returns: + A dictionary of information that the database can interpret. + """ + LOG.debug("Deserializing data...") + serialized_data = {} + + for field in entity.get_instance_fields(): + field_value = getattr(entity, field.name) + if isinstance(field_value, set): + # Explicitly mark this as a set so we can properly deserialize it later + serialized_data[field.name] = json.dumps({"__set__": list(field_value)}) + elif isinstance(field_value, (list, dict)): + serialized_data[field.name] = json.dumps(field_value) + elif isinstance(field_value, datetime): + serialized_data[field.name] = field_value.isoformat() + elif field_value is None: + serialized_data[field.name] = "null" + else: + serialized_data[field.name] = str(field_value) + + LOG.debug("Successfully deserialized data.") + return serialized_data + + +def deserialize_entity(data: Dict[str, str], model_class: T) -> T: + """ + Given data that was retrieved, convert it into a data_class instance. + + Args: + data: The data retrieved that we need to deserialize. + model_class: A [`BaseDataModel`][db_scripts.data_models.BaseDataModel] subclass. + + Returns: + A [`BaseDataModel`][db_scripts.data_models.BaseDataModel] instance. + """ + LOG.debug("Deserializing data...") + deserialized_data = {} + + for key, val in data.items(): + if val.startswith("[") or val.startswith("{"): + try: + loaded_val = json.loads(val) + # Check if this is a set that we specially encoded + if isinstance(loaded_val, dict) and "__set__" in loaded_val: + deserialized_data[key] = set(loaded_val["__set__"]) + else: + deserialized_data[key] = loaded_val + except json.JSONDecodeError as e: + LOG.error(f"Failed to deserialize JSON for key {key}: {val}") + LOG.error(f"Error: {str(e)}") + # Use the original string value as fallback + deserialized_data[key] = val + elif val == "null": + deserialized_data[key] = None + elif val in ("True", "False"): + deserialized_data[key] = val == "True" + elif is_iso_datetime(val): + deserialized_data[key] = datetime.fromisoformat(val) + elif val.isdigit(): + deserialized_data[key] = float(val) + else: + deserialized_data[key] = str(val) + + LOG.debug("Successfully deserialized data.") + return model_class.from_dict(deserialized_data) diff --git a/merlin/celery.py b/merlin/celery.py index 81a17c5b2..a4826ac1d 100644 --- a/merlin/celery.py +++ b/merlin/celery.py @@ -1,51 +1,29 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """Updated celery configuration.""" from __future__ import absolute_import, print_function import logging import os -from typing import Dict, Optional, Union +from typing import Any, Dict, List, Optional, Union import billiard import celery import psutil from celery import Celery, states from celery.backends.redis import RedisBackend # noqa: F401 ; Needed for celery patch -from celery.signals import worker_process_init +from celery.signals import celeryd_init, worker_process_init, worker_shutdown import merlin.common.security.encrypt_backend_traffic +from merlin.common.enums import WorkerStatus from merlin.config import broker, celeryconfig, results_backend from merlin.config.configfile import CONFIG from merlin.config.utils import Priority, get_priority +from merlin.db_scripts.merlin_db import MerlinDatabase from merlin.utils import nested_namespace_to_dicts @@ -58,7 +36,8 @@ def patch_celery(): Celery has error callbacks but they do not work properly on chords that are nested within chains. - Credit to this function goes to: https://danidee10.github.io/2019/07/09/celery-chords.html + Credit to this function goes to + [the following post](https://danidee10.github.io/2019/07/09/celery-chords.html). """ def _unpack_chord_result( @@ -84,9 +63,41 @@ def _unpack_chord_result( # This function has to have specific args/return values for celery so ignore pylint -def route_for_task(name, args, kwargs, options, task=None, **kw): # pylint: disable=W0613,R1710 +def route_for_task( + name: str, + args: List[Any], + kwargs: Dict[Any, Any], + options: Dict[Any, Any], + task: celery.Task = None, + **kw: Dict[Any, Any], +) -> Dict[Any, Any]: # pylint: disable=W0613,R1710 """ - Custom task router for queues + Custom task router for Celery queues. + + This function routes tasks to specific queues based on the task name. + If the task name contains a colon, it splits the name to determine the queue. + + Args: + name: The name of the task being routed. + args: The positional arguments passed to the task. + kwargs: The keyword arguments passed to the task. + options: Additional options for the task. + task: The task instance (default is None). + **kw: Additional keyword arguments for THIS function (not the task). + + Returns: + A dictionary specifying the queue to route the task to. + If the task name contains a colon, it returns a dictionary with + the key "queue" set to the queue name. Otherwise, it returns + an empty dictionary. + + Example: + Using a colon in the name will return the string before the colon as the queue: + + ```python + >>> route_for_task("my_queue:my_task") + {"queue": "my_queue"} + ``` """ if ":" in name: queue, _ = name.split(":") @@ -102,21 +113,25 @@ def route_for_task(name, args, kwargs, options, task=None, **kw): # pylint: dis RESULTS_BACKEND_URI: Optional[str] = "" try: BROKER_URI = broker.get_connection_string() - LOG.debug("broker: %s", broker.get_connection_string(include_password=False)) + sanitized_broker_uri = broker.get_connection_string(include_password=False) + LOG.debug(f"Broker connection string: {sanitized_broker_uri}.") BROKER_SSL = broker.get_ssl_config() - LOG.debug("broker_ssl = %s", BROKER_SSL) + LOG.debug(f"Broker SSL {'enabled' if BROKER_SSL else 'disabled'}.") RESULTS_BACKEND_URI = results_backend.get_connection_string() + sanitized_results_backend_uri = results_backend.get_connection_string(include_password=False) + LOG.debug(f"Results backend connection string: {sanitized_results_backend_uri}.") RESULTS_SSL = results_backend.get_ssl_config(celery_check=True) - LOG.debug("results: %s", results_backend.get_connection_string(include_password=False)) - LOG.debug("results: redis_backed_use_ssl = %s", RESULTS_SSL) + LOG.debug(f"Results backend SSL {'enabled' if RESULTS_SSL else 'disabled'}.") except ValueError: # These variables won't be set if running with '--local'. BROKER_URI = None RESULTS_BACKEND_URI = None +app_name = "merlin_test_app" if os.getenv("CELERY_ENV") == "test" else "merlin" + # initialize app with essential properties app: Celery = patch_celery().Celery( - "merlin", + app_name, broker=BROKER_URI, backend=RESULTS_BACKEND_URI, broker_use_ssl=BROKER_SSL, @@ -167,13 +182,76 @@ def route_for_task(name, args, kwargs, options, task=None, **kw): # pylint: dis app.autodiscover_tasks(["merlin.common"]) +@celeryd_init.connect +def handle_worker_startup(sender: str = None, **kwargs): + """ + Store information about each physical worker instance in the database. + + When workers first start up, the `celeryd_init` signal is the first signal + that they receive. This specific function will create a + [`PhysicalWorkerModel`][db_scripts.data_models.PhysicalWorkerModel] and + store it in the database. It does this through the use of the + [`MerlinDatabase`][db_scripts.merlin_db.MerlinDatabase] class. + + Args: + sender (str): The hostname of the worker that was just started + """ + if sender is not None: + LOG.debug(f"Worker {sender} has started.") + options = kwargs.get("options", None) + if options is not None: + try: + # Sender name is of the form celery@worker_name.%hostname + worker_name, host = sender.split("@")[1].split(".%") + merlin_db = MerlinDatabase() + logical_worker = merlin_db.get("logical_worker", worker_name=worker_name, queues=options.get("queues")) + physical_worker = merlin_db.create( + "physical_worker", + name=str(sender), + host=host, + status=WorkerStatus.RUNNING, + logical_worker_id=logical_worker.get_id(), + pid=os.getpid(), + ) + logical_worker.add_physical_worker(physical_worker.get_id()) + # Without this exception catcher, celery does not output any errors that happen here + except Exception as exc: + LOG.error(f"An error occurred when processing handle_worker_startup: {exc}") + else: + LOG.warning("On worker connect could not retrieve worker options from Celery.") + else: + LOG.warning("On worker connect no sender was provided from Celery.") + + +@worker_shutdown.connect +def handle_worker_shutdown(sender: str = None, **kwargs): + """ + Update the database for a worker entry when a worker shuts down. + + Args: + sender (str): The hostname of the worker that was just started + """ + if sender is not None: + LOG.debug(f"Worker {sender} is shutting down.") + merlin_db = MerlinDatabase() + physical_worker = merlin_db.get("physical_worker", str(sender)) + if physical_worker: + physical_worker.set_status(WorkerStatus.STOPPED) + physical_worker.set_pid(None) # Clear the pid + else: + LOG.warning(f"Worker {sender} not found in the database.") + else: + LOG.warning("On worker shutdown no sender was provided from Celery.") + + # Pylint believes the args are unused, I believe they're used after decoration @worker_process_init.connect() -def setup(**kwargs): # pylint: disable=W0613 +def setup(**kwargs: Dict[Any, Any]): # pylint: disable=W0613 """ - Set affinity for the worker on startup (works on toss3 nodes) + Set affinity for the worker on startup (works on toss3 nodes). - :param `**kwargs`: keyword arguments + Args: + **kwargs: Keyword arguments. """ if "CELERY_AFFINITY" in os.environ and int(os.environ["CELERY_AFFINITY"]) > 1: # Number of cpus between workers. diff --git a/merlin/common/__init__.py b/merlin/common/__init__.py index 37cabcad1..3dd89a2a8 100644 --- a/merlin/common/__init__.py +++ b/merlin/common/__init__.py @@ -1,29 +1,26 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +The `common` package provides shared utilities, classes, and logic used across Merlin. +It includes functionality for managing encryption, handling data sampling, working with +enumerations, and defining Celery tasks. + +Subpackages: + - `security/`: Contains functionality for managing encryption and ensuring secure + communication. Includes modules for general encryption logic and encrypting backend traffic. + +Modules: + dumper.py: Provides functionality for dumping information to files. + enums.py: Defines enumerations for interfaces. + sample_index_factory.py: Houses factory methods for creating + [`SampleIndex`][common.sample_index.SampleIndex] objects. + sample_index.py: Implements the logic for managing the sample hierarchy, including + the [`SampleIndex`][common.sample_index.SampleIndex] class. + tasks.py: Defines Celery tasks, breaking down the Directed Acyclic Graph ([`DAG`][study.dag.DAG]) + into smaller tasks that Celery can manage. + util_sampling.py: Contains utility functions for data sampling. +""" diff --git a/merlin/common/abstracts/__init__.py b/merlin/common/abstracts/__init__.py deleted file mode 100644 index 37cabcad1..000000000 --- a/merlin/common/abstracts/__init__.py +++ /dev/null @@ -1,29 +0,0 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### diff --git a/merlin/common/abstracts/enums/__init__.py b/merlin/common/abstracts/enums/__init__.py deleted file mode 100644 index 00d90232b..000000000 --- a/merlin/common/abstracts/enums/__init__.py +++ /dev/null @@ -1,51 +0,0 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### - -"""Package for providing enumerations for interfaces""" -from enum import IntEnum - - -__all__ = ("ReturnCode",) - - -class ReturnCode(IntEnum): - """ - Merlin return codes. - """ - - OK = 0 - ERROR = 1 - RESTART = 100 - SOFT_FAIL = 101 - HARD_FAIL = 102 - DRY_OK = 103 - RETRY = 104 - STOP_WORKERS = 105 - RAISE_ERROR = 106 diff --git a/merlin/common/dumper.py b/merlin/common/dumper.py index 5757b97ec..00cefc487 100644 --- a/merlin/common/dumper.py +++ b/merlin/common/dumper.py @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2022, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2 -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """This file is meant to help dump information to files""" import csv @@ -43,34 +20,53 @@ class Dumper: # pylint: disable=R0903 """ The dumper class is intended to help write information to files. - Currently, the supported file types to dump to are csv and json. - - Example csv usage: - dumper = Dumper("populations.csv") - # Eugene, OR has a population of 175096 - # Livermore, CA has a population of 86803 - population_data = { - "City": ["Eugene", "Livermore"], - "State": ["OR", "CA"], - "Population": [175096, 86803] - } - dumper.write(population_data, "w") - |---> Output will be written to populations.csv - - Example json usage: - dumper = Dumper("populations.json") - population_data = { - "OR": {"Eugene": 175096, "Portland": 641162}, - "CA": {"Livermore": 86803, "San Francisco": 815201} - } - dumper.write(population_data, "w") - |---> Output will be written to populations.json + Currently, the supported dump file types are: csv or json. + + Attributes: + file_name (str): The name of the file to write data to. + file_type (str): The type of the file (either "csv" or "json") determined + from the file name. + + Methods: + write: Writes information to the specified output file based on the file type. + _csv_write: Writes information to a CSV file. + _json_write: Writes information to a JSON file. + + Example: + CSV usage: + ```python + dumper = Dumper("populations.csv") + # Eugene, OR has a population of 175096 + # Livermore, CA has a population of 86803 + population_data = { + "City": ["Eugene", "Livermore"], + "State": ["OR", "CA"], + "Population": [175096, 86803] + } + dumper.write(population_data, "w") # Output will be written to populations.csv + ``` + + Example: + JSON usage: + ```python + dumper = Dumper("populations.json") + population_data = { + "OR": {"Eugene": 175096, "Portland": 641162}, + "CA": {"Livermore": 86803, "San Francisco": 815201} + } + dumper.write(population_data, "w") # Output will be written to populations.json + ``` """ - def __init__(self, file_name): + def __init__(self, file_name: str): """ - Initialize the class and ensure the file is of a supported type. - :param `file_name`: The name of the file to dump to eventually + Initializes the Dumper class and validates the file type. + + Args: + file_name: The name of the file to write data to. + + Raises: + ValueError: If the file type is not supported. Supported types are CSV and JSON. """ supported_types = ["csv", "json"] @@ -87,9 +83,14 @@ def __init__(self, file_name): def write(self, info_to_write: Dict, fmode: str): """ - Write information to an outfile. - :param `info_to_write`: The information you want to write to the output file - :param `fmode`: The file write mode ("w", "a", etc.) + Writes information to the specified output file. + + This method determines the file type and calls the appropriate + method to write the data. + + Args: + info_to_write: The information to write to the output file. + fmode: The file write mode ("w" for write, "a" for append, etc.). """ if self.file_type == "csv": self._csv_write(info_to_write, fmode) @@ -98,10 +99,12 @@ def write(self, info_to_write: Dict, fmode: str): def _csv_write(self, csv_to_dump: Dict[str, List], fmode: str): """ - Write information to a csv file. - :param `csv_to_dump`: The information to write to the csv file. - Dict keys will be the column headers and values will be the column values. - :param `fmode`: The file write mode ("w", "a", etc.) + Writes information to a CSV file. + + Args: + csv_to_dump: The data to write to the CSV file. Keys are column + headers and values are column values. + fmode: The file write mode ("w" for write, "a" for append, etc.). """ # If we have statuses to write, create a csv writer object and write to the csv file with open(self.file_name, fmode) as outfile: @@ -112,9 +115,11 @@ def _csv_write(self, csv_to_dump: Dict[str, List], fmode: str): def _json_write(self, json_to_dump: Dict[str, Dict], fmode: str): """ - Write information to a json file. - :param `json_to_dump`: The information to write to the json file. - :param `fmode`: The file write mode ("w", "a", etc.) + Writes information to a JSON file. + + Args: + json_to_dump: The data to write to the JSON file. + fmode: The file write mode ("w" for write, "a" for append, etc.). """ # Appending to json requires file mode to be r+ for json.load if fmode == "a": @@ -132,11 +137,18 @@ def _json_write(self, json_to_dump: Dict[str, Dict], fmode: str): def dump_handler(dump_file: str, dump_info: Dict): """ - Help handle the process of creating a Dumper object and writing + Handles the process of creating a Dumper object and writing data to an output file. - :param `dump_file`: A filepath to the file we're dumping to - :param `dump_info`: A dict of information that we'll be dumping to `dump_file` + This function checks if the specified dump file exists to determine + the appropriate file write mode (append or write). It then creates + a Dumper object and writes the provided information to the file, + logging the process. + + Args: + dump_file: The filepath to the file where data will be dumped. + dump_info: A dictionary containing the information to be written + to the `dump_file`. """ # Create a dumper object to help us write to dump_file dumper = Dumper(dump_file) diff --git a/merlin/common/enums.py b/merlin/common/enums.py new file mode 100644 index 000000000..a201867b4 --- /dev/null +++ b/merlin/common/enums.py @@ -0,0 +1,59 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +"""Package for providing enumerations for interfaces""" +from enum import Enum, IntEnum + + +__all__ = ("ReturnCode",) + + +class ReturnCode(IntEnum): + """ + Enum for Merlin return codes. + + This class defines various return codes used in the Merlin system to indicate + the status of operations. Each return code corresponds to a specific outcome + of a process. + + Attributes: + OK (int): Indicates a successful operation. Numeric value: 0. + ERROR (int): Indicates a general error occurred. Numeric value: 1. + RESTART (int): Indicates that the process should be restarted. Numeric value: 100. + SOFT_FAIL (int): Indicates a non-critical failure that allows for recovery. Numeric value: 101. + HARD_FAIL (int): Indicates a critical failure that cannot be recovered from. Numeric value: 102. + DRY_OK (int): Indicates a successful operation in a dry run (no changes made). Numeric value: 103. + RETRY (int): Indicates that the operation should be retried. Numeric value: 104. + STOP_WORKERS (int): Indicates that worker processes should be stopped. Numeric value: 105. + RAISE_ERROR (int): Indicates that an error should be raised. Numeric value: 106. + """ + + OK = 0 + ERROR = 1 + RESTART = 100 + SOFT_FAIL = 101 + HARD_FAIL = 102 + DRY_OK = 103 + RETRY = 104 + STOP_WORKERS = 105 + RAISE_ERROR = 106 + + +class WorkerStatus(Enum): + """ + Status of Merlin workers. + + Attributes: + RUNNING (str): Indicates the worker is running. String value: "running". + STALLED (str): Indicates the worker is running but hasn't been processing work. String value: "stalled". + STOPPED (str): Indicates the worker is not running. String value: "stopped". + REBOOTING (str): Indicates the worker is actively restarting itself. String value: "rebooting". + """ + + RUNNING = "running" + STALLED = "stalled" + STOPPED = "stopped" + REBOOTING = "rebooting" diff --git a/merlin/common/openfilelist.py b/merlin/common/openfilelist.py deleted file mode 100644 index 790054c16..000000000 --- a/merlin/common/openfilelist.py +++ /dev/null @@ -1,198 +0,0 @@ -#!/usr/bin/env python - -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### - -""" - OpenFileList - - A synthetic file class that opens a list of files and reads them as if they - were a single file - - SYNOPSIS: - - with OpenFileList(["file1.txt","file2.txt",...]) as f : - print f.read(); - - reads the concatenation of file1.txt, file2.txt, etc. - - file methods supported : - - f.read([bytes]) - f.readlines([bytes]) - f.readline([bytes]) - f.tell() - f.close() - f.__iter__() - - TODO: - implement a seek method - -""" - -# This file is not currently used so we don't care what pylint has to say -# pylint: skip-file - -import copy - - -class OpenFileList: - openwas = open - - def __new__(cls, files, *v, **kw): - if isinstance(files, str): - return open(files, *v, **kw) - return super(OpenFileList, cls).__new__(cls) - - def __init__(self, files, *v, **kw): - self.files = copy.copy(files) - self.argv, self.argkw = (v, kw) - if self.files: - self.fnnow = self.files.pop(0) - self.fnow = open(self.fnnow, *v, **kw) if files else None # noqa - self.atend = False - else: - self.fnnow = self.fnow = None - self.atend = True - self._tell = 0 - - def _errclosed(self): - raise ValueError("I/O operation on closed file") - - def _tonext(self): - if self.fnow is not None: - self._tell += self.fnow.tell() - self.fnow.close() - if self.files: - self.fnnow = self.files.pop(0) - self.fnow = open(self.fnnow, *self.argv, **self.argkw) # noqa - else: - self.fnnow = self.fnow = None - self.atend = True - - def tell(self): - if self.fnow is None: - return self._tell - return self._tell + self.fnow.tell() - - def read(self, n=None): - if self.atend: - return "" - if self.fnow is None: - self._errclosed() - if n is None: - n = 1 << 32 - s = "" - while n and (self.files or self.fnow is not None): - ns = self.fnow.read(n) - if ns: - n -= len(ns) - s += ns - else: - self._tonext() - return s - - def readlines(self, b=None): - if self.atend: - return [] - if self.fnow is None: - self._errclosed() - s = self.read(b) - if not s: - return [] - if not s.endswith("\n"): - ch = "" - while ch != "\n": - ch = self.read(1) - if ch == "" or ch == "\n": - break - s += ch - return s.split("\n") - else: - return s[:-1].split("\n") - - def readline(self, b=None): - if self.atend: - return [] - if self.fnow is None: - return self._errclosed() - if b is None: - s = self.fnow.readline() - else: - s = self.fnow.readline(b) - if not s: - self._tonext() - return s - - def __iter__(self): - while not self.atend: - yield self.readline() - - def close(self): - if self.fnow is not None: - self.fnow.close() - self.files = [] - self.atend = True - self.fnow = self.fnnow = None - - def __enter__(self): - return self - - def __exit__(self, exc_type, exc_value, tback): - self.close() - - -if __name__ == "__main__": - import os - import unittest - import uuid - - import numpy - - class TestOpenFileList(unittest.TestCase): - def test_opener(self): - stride = 5 - fn = [str(uuid.uuid1()) for i in range(3)] - e = numpy.diag(numpy.arange(float(stride) * len(fn))) - - for n, ff in enumerate(fn): # Create files. - with open(ff, "w") as f: - for i in range(stride * n, stride * (n + 1)): - print(" ".join(map(str, e[i])), file=f) - - with OpenFileList(fn) as f: # Load using loadtxt method. - ep = numpy.loadtxt(f) - - for i in fn: - os.unlink(i) # Delete files. - - self.assertTrue(numpy.all(ep == e)) - - unittest.main() diff --git a/merlin/common/opennpylib.py b/merlin/common/opennpylib.py deleted file mode 100644 index ba6e9dcc6..000000000 --- a/merlin/common/opennpylib.py +++ /dev/null @@ -1,391 +0,0 @@ -#!/usr/bin/env python - -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### - -""" -smallnpylib - -A simple library to read the .npy file header and return a dict -containing the header from the .npy file as well as a few other -keys. Also provides the OpenNPY class which is a way of seeking -into .npy files without using the memory mapping functionality -in the load function. Finally, there is the OpenNPYList class -which opens a list of OpenNPY files and allows for random -access among all of them. - - 'shape' shape for the array - 'fortran_order' is it in fortran order (normally false) - 'descr' the dtype description of the array - 'size' total size of the data - 'offset' position in file of start of data - 'rowsize' the total size in bytes of a single row in the file (e.g. sample) - 'itemsize' the size of a single element in the array - - SYNOPSIS : - - d = get_npy_info("myfile.txt"); - print d['offset'] # offset to data in file - print d['rowsize'] # number of bytes per row - - with open("myfile.npy") as f : - f.seek(d['offset']+d['rowsize]*N); - - OpenNPY - - with OpenNPY("myfile.npy") as a : - print a[5] # print row number 5 - print a[1:4] # print rows from 1,2,3 - my_array = a.to_array(); - print len(a) # number of rows in file - for i in a : - print i # print all the rows in a - print a.shape # shape of array - print a.dtype # dtype of array - - with OpenNPYList(["myfile1.npy","myfile2.npy",...]) as a : - print a[5] # print row number 5 - print a[1:4] # print rows from 1,2,3 - my_array = a.to_array(); - print len(a) # number of rows in file - for i in a : - print i # print all the rows in a - print a.shape # shape of array - print a.dtype # dtype of array - -""" -# This file is not currently used so we don't care what pylint has to say -# pylint: skip-file - -from typing import List, Tuple - -import numpy as np - - -try: - unistr = (unicode, str) - npy_magic = "\x93NUMPY" -except NameError: - unistr = str - npy_magic = b"\x93NUMPY" - - -def _get_npy_info2(f): - if isinstance(f, unistr): - f = open(f, "rb") # noqa - magic = f.read(6) # must be .npy file - assert magic == npy_magic # must be .npy file or ELSE - major, _ = list(map(ord, f.read(2))) - if major == 1: - hlen_char = list(map(ord, f.read(2))) - hlen = hlen_char[0] + 256 * hlen_char[1] - elif major == 2: - hlen_char = list(map(ord, f.read(4))) - # fmt: off - hlen = ( - hlen_char[0] - + 256 * hlen_char[1] - + 65536 * hlen_char[2] - + (1 << 24) * hlen_char[3] - ) - # fmt: on - else: - raise Exception("unknown .npy format, e.g. not 1 or 2") - hdr = eval(f.read(hlen)) # TODO remove eval - hdr["dtype"] = np.dtype(hdr["descr"]) - hdr["offset"] = f.tell() # location of data start - hdr["itemsize"] = np.dtype(hdr["descr"]).itemsize - hdr["rowsize"] = hdr["itemsize"] * np.product(hdr["shape"][1:]) - hdr["items"] = np.product(hdr["shape"]) - hdr["size"] = hdr["itemsize"] * hdr["items"] - return f, hdr - - -def _get_npy_info3(f): - if isinstance(f, unistr): - f = open(f, "rb") # noqa - magic = f.read(6) # must be .npy file - assert magic == npy_magic # must be .npy file or ELSE - major, _ = list(f.read(2)) - if major == 1: - hlen_char = list(f.read(2)) - hlen = hlen_char[0] + 256 * hlen_char[1] - elif major == 2: - hlen_char = list(f.read(4)) - # fmt: off - hlen = ( - hlen_char[0] - + 256 * hlen_char[1] - + 65536 * hlen_char[2] - + (1 << 24) * hlen_char[3] - ) - # fmt: on - else: - raise Exception("unknown .npy format, e.g. not 1 or 2") - hdr = eval(f.read(hlen)) # TODO remove eval - hdr["dtype"] = np.dtype(hdr["descr"]) - hdr["offset"] = f.tell() # location of data start - hdr["itemsize"] = np.dtype(hdr["descr"]).itemsize - hdr["rowsize"] = hdr["itemsize"] * np.product(hdr["shape"][1:]) - hdr["items"] = np.product(hdr["shape"]) - hdr["size"] = hdr["itemsize"] * hdr["items"] - return f, hdr - - -def _get_npy_info(f): - d = None - try: - d = _get_npy_info2(f) - except TypeError: - d = _get_npy_info3(f) - - return d - - -def get_npy_info(f): - try: - d = _get_npy_info2(f) - except TypeError: - d = _get_npy_info3(f) - d[0].close() - return d[1] - - -def read_items(f, hdr, idx, n=-1, sep=""): - f.seek(hdr["offset"] + idx * hdr["itemsize"]) - if n < 0: - n = hdr["items"] - idx - n = min(hdr["items"] - idx, n) - print(f"n is {n}") - return np.fromfile(f, dtype=hdr["dtype"], count=n, sep=sep) - - -def read_rows(f, hdr, idx, n=-1, sep=""): - f.seek(hdr["offset"] + idx * hdr["rowsize"]) - if n < 0: - n = hdr["shape"][0] - idx - n = min(hdr["shape"][0] - idx, n) - a = np.fromfile(f, dtype=hdr["dtype"], count=n * hdr["rowsize"] // hdr["itemsize"], sep=sep) - return np.reshape(a, (n,) + hdr["shape"][1:]) - - -def verify_open(func): # A wrapper function used by FileSamples. - """ - :param func: (function) a class instance method that needs to call - _verify_open before doing anything else. - """ - - def wrapper(self, *v, **kw): - self._verify_open() - return func(self, *v, **kw) - - return wrapper - - -class OpenNPY: - def __init__(self, f): - self.hdr = self.f = None - if isinstance(f, unistr): - self.fname = f - else: - self.f = f - self._verify_open() - - def _verify_open(self): - if self.f is None: - self.f, self.hdr = _get_npy_info(self.f if self.f is not None else self.fname) - - @verify_open - def load_header(self, close=True): - if close: - self.close() - return self.hdr - - def close(self): - if self.f is not None: - self.f.close() - self.f = None - - def __del__(self): - self.close() - - def __enter__(self): - return self - - def __exit__(self, exc_type, exc_value, tback): - self.close() - - @verify_open - def _shape(self): - return self.hdr["shape"] - - shape = property(fget=_shape) - - @verify_open - def _dtype(self): - return self.hdr["dtype"] - - dtype = property(fget=_dtype) - - @verify_open - def __getitem__(self, k): - if isinstance(k, int): - return read_rows(self.f, self.hdr, k, 1)[0] - elif isinstance(k, slice): - if k.step == 1 or k.step is None: - return read_rows(self.f, self.hdr, k.start, k.stop - k.start) - else: - return np.asarray( - [read_rows(self.f, self.hdr, _, 1)[0] for _ in range(k.start, k.stop, 1 if k.step is None else k.step)] - ) - - @verify_open - def __len__(self): - return self.hdr["items"] - - @verify_open - def __iter__(self): - for k in range(self.hdr["shape"][0]): - yield self[k] - - @verify_open - def to_array(self): - return read_rows(self.f, self.hdr, 0) - - -class OpenNPYList: - def __init__(self, filename_strs: List[str]): - self.filenames: List[str] = filename_strs - self.files: List[OpenNPY] = [OpenNPY(file_str) for file_str in self.filenames] - i: OpenNPY - for i in self.files: - i.load_header() - self.shapes: List[Tuple[int]] = [openNPY_obj.hdr["shape"] for openNPY_obj in self.files] - k: Tuple[int] - for k in self.shapes[1:]: - # Match subsequent axes shapes. - if k[1:] != self.shapes[0][1:]: - raise AttributeError(f"Mismatch in subsequent axes shapes: {k[1:]} != {self.shapes[0][1:]}") - self.tells: np.ndarray = np.cumsum([arr_shape[0] for arr_shape in self.shapes]) # Tell locations. - self.tells = np.hstack(([0], self.tells)) - - def close(self): - for i in self.files: - i.close() - - def __del__(self): - self.close() - - def __iter__(self): - for i in self.files: - for j in i: - yield j - - def __getitem__(self, k): - if isinstance(k, int): - if k < 0: - k = self.tells[-1] + k # Negative indexing. - if k >= self.tells[-1]: - raise IndexError("index %d is out of bounds" % k) - fno = (self.tells > k).argmax() - return self.files[fno - 1][k - self.tells[fno - 1]] - else: # Slice indexing. - # TODO : Implement a faster version. - return np.asarray([self[_] for _ in np.arange(k.start, k.stop, k.step if k.step is not None else 1)]) - - def to_array(self): - return np.vstack([_.to_array() for _ in self.files]) - - def __len__(self): - return self.tells[-1] - - def __enter__(self): - return self - - def __exit__(self, exc_type, exc_value, tback): - self.close() - - -__all__ = [ - "OpenNPYList", - "OpenNPY", - "unistr", - "read_items", - "read_rows", - "get_npy_info", -] - -if __name__ == "__main__": - import os - import sys - import unittest - import uuid - - if "-h" in sys.argv: - print(__doc__) - sys.exit(1) - - class TestOpenNPY(unittest.TestCase): - def test_seeknpy(self): - e = np.diag(np.arange(5.0)) - try: - fn = unicode(str(uuid.uuid1()) + ".npy") - except NameError: - fn = str(str(uuid.uuid1()) + ".npy") - np.save(fn, e) - with OpenNPY(fn) as a: - ep = np.asarray([_ for _ in a]) - os.unlink(fn) - self.assertTrue((ep == e).all()) - - def test_seeknpylist(self): - e = np.diag(np.arange(5.0)) - fn = str(uuid.uuid1()) + ".npy" - np.save(fn, e) - with OpenNPYList([fn, fn, fn]) as a: - ep = np.asarray([_ for _ in a]) - en = np.asarray(a[5:10]) - en2 = np.asarray(a[11:14]) - en3 = np.asarray(a[1:14]) - en4 = a.to_array() - # test __len__ method - self.assertEqual(len(a), 3 * len(e)) - os.unlink(fn) - # test read slice of whole file - self.assertTrue((en == e).all()) - self.assertTrue((en2 == e[1:4]).all()) # test slice - # test read all - self.assertTrue((ep == np.vstack((e, e, e))).all()) - # test to_array method - self.assertTrue((en4 == np.vstack((e, e, e))).all()) - # test slice read across files - self.assertTrue((en3 == np.vstack((e, e, e))[1:14]).all()) - - unittest.main() diff --git a/merlin/common/sample_index.py b/merlin/common/sample_index.py index e52ded9c7..e7c92d2d2 100644 --- a/merlin/common/sample_index.py +++ b/merlin/common/sample_index.py @@ -1,40 +1,18 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ -The merlin sample_index module, which contains the SampleIndex class. +This module contains the logic for managing the sample hierarchy, including +the implementation of the [`SampleIndex`][common.sample_index.SampleIndex] class. """ import logging import os from contextlib import suppress +from typing import Callable, Dict, Generator, List, Tuple LOG = logging.getLogger(__name__) @@ -42,14 +20,36 @@ MAX_SAMPLE = 99999999999999999999999999999999999 -def new_dir(path): - """Create a new directory at the given path if it does not exist.""" +def new_dir(path: str): + """ + Create a new directory at the specified path if it does not already exist. + + This function attempts to create the directory and suppresses any + OSError that may occur if the directory already exists. + + Args: + path: The path where the new directory should be created. + """ with suppress(OSError): os.makedirs(path) -def uniform_directories(num_samples=MAX_SAMPLE, bundle_size=1, level_max_dirs=100): - """Create a directory hierarchy uniformly stepping up directory sizes.""" +def uniform_directories(num_samples: int = MAX_SAMPLE, bundle_size: int = 1, level_max_dirs: int = 100) -> List[int]: + """ + Create a directory hierarchy with uniformly increasing directory sizes. + + This function generates a list of directory sizes, starting from + the specified `bundle_size` and increasing by a factor of + `level_max_dirs` until the total number of samples is reached. + + Args: + num_samples: The total number of samples to consider. + bundle_size: The initial size of each bundle. + level_max_dirs: The factor by which to increase directory sizes at each level. + + Returns: + A list of integers representing the sizes of directories in the hierarchy. + """ directory_sizes = [bundle_size] while directory_sizes[0] < num_samples: directory_sizes.insert(0, directory_sizes[0] * level_max_dirs) @@ -60,17 +60,80 @@ def uniform_directories(num_samples=MAX_SAMPLE, bundle_size=1, level_max_dirs=10 class SampleIndex: """ - A SampleIndex is the insitu representation of a directory hierarchy where - bundles of samples result files will be stored. Factory methods to produce - full hierarchies are provided, as well as to read an index from one - previously stored on disk. + Represents a hierarchical structure for managing bundles of sample files. + + A [`SampleIndex`][common.sample_index.SampleIndex] serves as an in-situ + representation of a directory hierarchy where bundles of sample result files + are stored. This class provides factory methods to create complete hierarchies + and to read an index from a previously stored file on disk. + + Attributes: + address (str): The full address of this node. + children (Dict[str, SampleIndex]): A dictionary containing the direct children of this node, which are also + of type SampleIndex. + depth (int): (class attribute) A class variable indicating the current depth in the + hierarchy, primarily used for pretty printing. + is_leaf (bool): Returns whether this node is a leaf in the hierarchy. + is_directory (bool): Returns whether this node is a directory (not a leaf). + is_parent_of_leaf (bool): Returns whether this node is the direct parent of a leaf. + is_grandparent_of_leaf (bool): Returns whether this node is the parent of a parent of a leaf. + is_great_grandparent_of_leaf (bool): Returns whether this node is the parent of a grandparent of a leaf. + leafid (int): The unique leaf ID of this node. + max (int): The maximum global sample ID of any child node. + min (int): The minimum global sample ID of any child node. + name (str): The name of this node in the hierarchy. + num_bundles (int): The total number of bundles in this index. + + Methods: + traverse: Yields the full path and associated node for each node meeting a specified condition. + traverse_all: Returns a generator that traverses all nodes in the + [`SampleIndex`][common.sample_index.SampleIndex]. + traverse_bundles: Returns a generator that traverses all bundles (leaves) in the + [`SampleIndex`][common.sample_index.SampleIndex]. + traverse_directories: Returns a generator that traverses all directories in the + [`SampleIndex`][common.sample_index.SampleIndex]. + check_valid_addresses_for_insertion: Validates addresses for insertion into the hierarchy. + __getitem__: Retrieves a child node by its full address. + __setitem__: Sets a child node at a specified full address. + write_directory: Creates a new directory associated with this node. + write_directories: Recursively writes directories for this node and its children. + get_path_to_sample: Retrieves the file path to the bundle file with a specified sample ID. + write_single_sample_index_file: Writes a single sample index file for this node. + write_multiple_sample_index_files: Writes multiple sample index files for this node and its children. + make_directory_string: Creates a delimited string representation of the directories in the index. + __str__: Returns a string representation of the [`SampleIndex`][common.sample_index.SampleIndex], + including its children. """ # Class variable to indicate depth (mostly used for pretty printing). - depth = -1 - - def __init__(self, minid, maxid, children, name, leafid=-1, num_bundles=0, address=""): # pylint: disable=R0913 - """The constructor.""" + depth: int = -1 + + def __init__( + self, + minid: int, + maxid: int, + children: Dict[str, "SampleIndex"], + name: str, + leafid: int = -1, + num_bundles: int = 0, + address: str = "", + ): # pylint: disable=R0913 + """ + Initializes a new instance of the `SampleIndex` class. + + Args: + minid: The minimum global sample ID of any child node. + maxid: The maximum global sample ID of any child node. + children: A dictionary containing the direct children of this node, + where the keys are the children's full addresses and the values are `SampleIndex` instances. + name: The name of this node in the hierarchy. + leafid: The unique leaf ID of this node. + num_bundles: The total number of bundles in this index. + address: The full address of this node. + + Raises: + TypeError: If `children` is not of type `dict`. + """ # The direct children of this node, generally also of type SampleIndex. # A dictionary keyed by the childrens' full addresses. @@ -78,39 +141,56 @@ def __init__(self, minid, maxid, children, name, leafid=-1, num_bundles=0, addre LOG.error("SampleIndex children must be of type dict") raise TypeError - self.children = children - self.name = name # The name of this node. - self.address = str(address) # The address of this node + self.children: Dict[str, "SampleIndex"] = children + self.name: str = name # The name of this node. + self.address: str = str(address) # The address of this node # @NOTE: The following are only valid if no insertions,splits, or # joins have taken place since the last renumber() call, or this index # was constructed with apriori global knowledge of its contents. # The minimum global sample ID of any of the children of this node. - self.min = minid + self.min: int = minid # The maximum global sample ID of any of the children of this node. - self.max = maxid + self.max: int = maxid # The total number of bundles in this index. - self.num_bundles = num_bundles + self.num_bundles: int = num_bundles # The unique leaf ID of this node - self.leafid = leafid + self.leafid: int = leafid @property - def is_leaf(self): - """Returns whether this is a leaf in the graph""" + def is_leaf(self) -> bool: + """ + Indicates whether this node is a leaf in the hierarchy. + + A leaf node is defined as a node that has no children. This property + returns True if the node has no children, and False otherwise. + """ return len(self.children) == 0 @property - def is_directory(self): - """Returns whether this is a directory (not a leaf) in the graph""" + def is_directory(self) -> bool: + """ + Indicates whether this node is a directory (not a leaf). + + A directory node is defined as a node that has one or more children. + This property returns True if the node has children, and False if it + is a leaf node. + """ return len(self.children) > 0 @property - def is_parent_of_leaf(self): - """Returns whether this is the direct parent of a leaf in the graph""" + def is_parent_of_leaf(self) -> bool: + """ + Indicates whether this node is the direct parent of a leaf. + + This property checks if the current node is a directory and if any of + its children are leaf nodes. It returns True if at least one child is + a leaf, and False otherwise. + """ if not self.is_directory: return False for child_val in self.children.values(): @@ -119,8 +199,14 @@ def is_parent_of_leaf(self): return False @property - def is_grandparent_of_leaf(self): - """Returns whether this is the parent of a parent of a leaf in the graph""" + def is_grandparent_of_leaf(self) -> bool: + """ + Indicates whether this node is the parent of a parent of a leaf. + + This property checks if the current node is a directory and if any of + its children are parents of leaf nodes. It returns True if at least + one child is a parent of a leaf, and False otherwise. + """ if not self.is_directory: return False for child_val in self.children.values(): @@ -129,8 +215,14 @@ def is_grandparent_of_leaf(self): return False @property - def is_great_grandparent_of_leaf(self): - """Returns whether this is the parent of a parent of a leaf in the graph""" + def is_great_grandparent_of_leaf(self) -> bool: + """ + Indicates whether this node is the parent of a grandparent of a leaf. + + This property checks if the current node is a directory and if any of + its children are grandparents of leaf nodes. It returns True if at + least one child is a grandparent of a leaf, and False otherwise. + """ if not self.is_directory: return False for child_val in self.children.values(): @@ -138,19 +230,44 @@ def is_great_grandparent_of_leaf(self): return True return False - def traverse(self, path=None, conditional=lambda c: True, bottom_up=True, top_level=True): + def traverse( + self, + path: str = None, + conditional: Callable = lambda c: True, + bottom_up: bool = True, + top_level: bool = True, + ) -> Generator[Tuple[str, "SampleIndex"], None, None]: """ - Yield the full path and associated node for each node that meets the - conditional - param:path: The path to this node. - param:conditional: A lambda that returns a boolean, takes a - SampleIndex as its only argument. - - param:bottom_up: If True, yield leaves of the tree first. Otherwise, - yield top level nodes first. - param:top_level: used to allow filtering of yielded values based off - the conditional. Should only be set to False internally for the - recursive calls. + Traverse the tree structure and yield the full path and associated node + for each node that meets the specified conditional criteria. + + This method allows for flexible traversal of the tree, either from + the top down or bottom up, depending on the `bottom_up` parameter. + Nodes are yielded based on whether they satisfy the provided + conditional function. + + Notes: + - The method uses a "SKIP ME" placeholder to manage the + recursion flow, ensuring that the traversal can skip + non-qualifying nodes without breaking the iteration. + + Args: + path: The current path to this node. If None, it defaults to + the name of the node. + conditional: A function that takes a + [`SampleIndex`][common.sample_index.SampleIndex] instance as its + only argument and returns a boolean. It determines whether a + node should be yielded. + bottom_up: If True, yields leaf nodes first, moving upwards through + the tree. If False, yields top-level nodes first. + top_level: A flag used internally to control filtering of yielded + values based on the conditional. Should only be set to False + for recursive calls. + + Yields: + A tuple containing the full path to the node and the + associated [`SampleIndex`][common.sample_index.SampleIndex] node + that meets the conditional criteria. """ if path is None: path = self.name @@ -171,36 +288,118 @@ def traverse(self, path=None, conditional=lambda c: True, bottom_up=True, top_le if not top_level: yield "SKIP ME" - def traverse_all(self, bottom_up=True): + def traverse_all(self, bottom_up: bool = True) -> Generator[Tuple[str, "SampleIndex"], None, None]: """ - Returns a generator that will traverse all nodes in the SampleIndex. + Traverse all nodes in the [`SampleIndex`][common.sample_index.SampleIndex]. + + This method returns a generator that yields all nodes in the + [`SampleIndex`][common.sample_index.SampleIndex], regardless of their type + (leaf or directory). The traversal order can be controlled by the `bottom_up` + parameter. + + Notes: + This method calls the [`traverse`][common.sample_index.SampleIndex.traverse] + method with a conditional that always returns True, ensuring all nodes are + included. + + Args: + bottom_up: If True, yields leaf nodes first, moving upwards + through the tree. If False, yields top-level nodes first. + + Returns: + A tuple containing the full path to each node and the associated + [`SampleIndex`][common.sample_index.SampleIndex] node. """ return self.traverse(path=self.name, conditional=lambda c: True, bottom_up=bottom_up) - def traverse_bundles(self): + def traverse_bundles(self) -> Generator[Tuple[str, "SampleIndex"], None, None]: """ - Returns a generator that will traverse all Bundles (leaves) in the - SampleIndex. + Traverse all Bundles (leaf nodes) in the [`SampleIndex`][common.sample_index.SampleIndex]. + + This method returns a generator that yields only the leaf nodes + (Bundles) in the [`SampleIndex`][common.sample_index.SampleIndex]. + It filters the nodes based on their type, ensuring that only leaves are returned. + + Notes: + This method calls the [`traverse`][common.sample_index.SampleIndex.traverse] + method with a conditional that checks if a node is a leaf, ensuring only Bundles + are yielded. + + Returns: + A tuple containing the full path to each Bundle and the associated + [`SampleIndex`][common.sample_index.SampleIndex] node. """ return self.traverse(path=self.name, conditional=lambda c: c.is_leaf) - def traverse_directories(self, bottom_up=False): + def traverse_directories(self, bottom_up: bool = False) -> Generator[Tuple[str, "SampleIndex"], None, None]: """ - Returns a generator that will traverse all Directories in the - SampleIndex. + Traverse all Directories in the [`SampleIndex`][common.sample_index.SampleIndex]. + + This method returns a generator that yields all directory nodes + in the [`SampleIndex`][common.sample_index.SampleIndex]. The + traversal order can be controlled by the `bottom_up` parameter. + + Notes: + This method calls the [`traverse`][common.sample_index.SampleIndex.traverse] + method with a conditional that checks if a node is a directory, ensuring only + directories are yielded. + + Args: + bottom_up: If True, yields leaf directories first, moving + upwards through the tree. If False, yields top-level + directories first. + + Yields: + A tuple containing the full path to each Directory and the + associated [`SampleIndex`][common.sample_index.SampleIndex] + node. """ return self.traverse(path=self.name, conditional=lambda c: c.is_directory, bottom_up=bottom_up) @staticmethod - def check_valid_addresses_for_insertion(full_address, sub_tree): + def check_valid_addresses_for_insertion(full_address: str, sub_tree: "SampleIndex"): """ - TODO + Check if the provided address is valid for insertion into the subtree. + + This method traverses all nodes in the given subtree and verifies + that no existing node's address conflicts with the specified + `full_address`. If any node's address starts with the `full_address`, + a TypeError is raised, indicating that the insertion would create + an invalid state. + + Args: + full_address: The full address to be checked for validity before + insertion. + sub_tree: The subtree in which the address will be checked. + + Raises: + TypeError: If any node in the subtree has an address that + conflicts with the `full_address`. """ for _, node in sub_tree.traverse_all(): if node.address[0 : len(full_address)] != full_address: raise TypeError - def __getitem__(self, full_address): + def __getitem__(self, full_address: str) -> "SampleIndex": + """ + Retrieve the subtree associated with the given full address. + + This method allows for accessing nodes in the + [`SampleIndex`][common.sample_index.SampleIndex] using + the full address. If the full address matches the current node's + address, the node itself is returned. Otherwise, it recursively + searches through the children to find the corresponding node. + + Args: + full_address: The full address of the node to retrieve. + + Returns: + The node associated with the specified full address. + + Raises: + KeyError: If no node with the specified full address exists + in the [`SampleIndex`][common.sample_index.SampleIndex]. + """ if full_address == self.address: return self for child_val in list(self.children.values()): @@ -208,7 +407,28 @@ def __getitem__(self, full_address): return child_val[full_address] raise KeyError - def __setitem__(self, full_address, sub_tree): + def __setitem__(self, full_address: str, sub_tree: "SampleIndex"): + """ + Set or replace the subtree associated with the given full address. + + This method allows for inserting or updating a node in the + [`SampleIndex`][common.sample_index.SampleIndex]. If the full + address matches the current node's address, a KeyError is raised + to prevent self-assignment. The method searches through the children + to find the appropriate location for insertion. If a node already + exists at the specified address, it will be replaced after validating + the insertion. + + Args: + full_address: The full address of the node to set or replace. + sub_tree: The subtree to insert or replace at the specified + address. + + Raises: + KeyError: If the full address matches the current node's + address or if no matching child node is found for + insertion. + """ if full_address == self.address: # This should never happen. raise KeyError @@ -231,23 +451,68 @@ def __setitem__(self, full_address, sub_tree): return raise KeyError - def write_directory(self, path): - """Creates a new directory associated with this node in the graph.""" + def write_directory(self, path: str): + """ + Create a new directory associated with this node in the graph. + + This method checks if the current node is a directory and, if so, + creates a new directory at the specified path using the node's + name. The directory is created using the + [`new_dir`][common.sample_index.new_dir] function. + + Args: + path: The base path where the new directory will be created. + """ if self.is_directory: new_dir(os.path.join(path, self.name)) - def write_directories(self, path="."): + def write_directories(self, path: str = "."): """ - Creates the directory tree associated with this node and its children. + Create the directory tree associated with this node and its children. + + This method initiates the creation of the directory structure + starting from the current node. It first creates the directory for + the current node and then recursively creates directories for all + child nodes. The base path can be specified, and the directories + will be created relative to this path. + + Args: + path: The base path where the directory tree will be created. + Defaults to the current directory ("."), meaning the + directories will be created in the current working + directory. """ self.write_directory(path) for child_val in list(self.children.values()): child_val.write_directories(os.path.join(path, self.name)) - def get_path_to_sample(self, sample_id): + def get_path_to_sample(self, sample_id: int) -> str: """ - Retrieves the file path to the bundle file with the sample_id of - interest. Note this only works when the global numbering is known. + Retrieve the file path to the bundle file associated with the specified + sample ID. + + This method constructs the path to the bundle file by traversing the + directory structure represented by the current node and its children. + It checks each child node to determine if the provided `sample_id` + falls within the range defined by the child's `min` and `max` attributes. + If a matching child is found, the method recursively calls itself to + build the complete path. + + Notes: + - This method assumes that the global numbering system is known + and that the `min` and `max` attributes of child nodes are + correctly defined. + - If no matching child is found, the method will return the + current node's name as the base path. + + Args: + sample_id: The identifier of the sample for which the file path is + to be retrieved. This ID must correspond to a sample within the + known global numbering system. + + Returns: + The constructed file path to the bundle file associated + with the specified `sample_id`. """ path = self.name for child_val in self.children.values(): @@ -255,8 +520,32 @@ def get_path_to_sample(self, sample_id): path = os.path.join(path, child_val.get_path_to_sample(sample_id)) return path - def write_single_sample_index_file(self, path): - """Writes the index file associated with this node.""" + def write_single_sample_index_file(self, path: str) -> str: + """ + Write the index file associated with this node. + + This method creates an index file named "sample_index.txt" in the + specified directory path. The index file contains information about + the child nodes of the current node. For each child, it records + whether the child is a leaf or a directory, along with its address, + name, and the range of sample IDs it covers. + + Notes: + - This method will only execute if the current node is identified + as a directory + ([`self.is_directory`][common.sample_index.SampleIndex.is_directory]). + - The index file format includes lines for each child in the + following format: + - For leaf nodes: `BUNDLE:

\tname:\tSAMPLES:[, )` + - For directory nodes: `DIR:
\tname:\tSAMPLES:[, )` + + Args: + path: The base path where the index file will be created. + + Returns: + The full path to the created index file if the current node is + a directory; otherwise, returns None. + """ if not self.is_directory: return None @@ -273,10 +562,23 @@ def write_single_sample_index_file(self, path): ) return fname - def write_multiple_sample_index_files(self, path="."): + def write_multiple_sample_index_files(self, path: str = ".") -> List[str]: """ - Write index files that couple with location in directory hierarchy, - contain necessary info to create a new index. + Write index files that correspond to the location in the directory hierarchy. + + This method generates index files for the current node and all its + children, allowing for a structured representation of the sample + indices in the directory hierarchy. It first writes a single index + file for the current node and then recursively writes index files + for each child node. + + Args: + path: The base path where the index files will be created. + Defaults to the current directory ("."), meaning the index + files will be created in the current working directory. + + Returns: + A list of file paths to the created index files. """ filepath = self.write_single_sample_index_file(path) filepaths = [] @@ -286,18 +588,29 @@ def write_multiple_sample_index_files(self, path="."): filepaths += child_val.write_multiple_sample_index_files(os.path.join(path, self.name)) return filepaths - def make_directory_string(self, delimiter=" ", just_leaf_directories=True): + def make_directory_string(self, delimiter: str = " ", just_leaf_directories: bool = True) -> str: """ - Make a string that is a delimited list of the directories in the - index. - - :param delimiter: the characters used to separate the directories - :param just_leaf_directories: A boolean on whether just to return the - leaf (bottom) directories - :returns: A string representation of the directories - e.g. - "0/0 0/1 0/2 1/0 1/1 1/2" - + Create a delimited string representation of the directories in the index. + + This method generates a string that lists the directories in the + current index, separated by the specified delimiter. The user can + choose to include only the leaf directories or all directories. + + Notes: + - The method utilizes the + [`traverse_directories`][common.sample_index.SampleIndex.traverse_directories] + function to retrieve the directory paths and their corresponding nodes. + + Args: + delimiter: The characters used to separate the directories in the + resulting string. + just_leaf_directories: If True, only leaf directories (the bottom-level + directories) will be included in the output. If False, all directories + will be included. + + Returns: + A string representation of the directories, formatted as a delimited + list. For example: "0/0 0/1 0/2 1/0 1/1 1/2". """ # fmt: off if just_leaf_directories: @@ -309,8 +622,28 @@ def make_directory_string(self, delimiter=" ", just_leaf_directories=True): # fmt: on return delimiter.join([path for path, _ in self.traverse_directories()]) - def __str__(self): - """String representation.""" + def __str__(self) -> str: + """ + Return a string representation of the + [`SampleIndex`][common.sample_index.SampleIndex] object. + + This method provides a formatted string that represents the current + node in the sample index, including its address, type (BUNDLE or + DIRECTORY), and relevant attributes such as minimum and maximum + sample IDs. If the node is a directory, it recursively includes + the string representations of its child nodes. + + Notes: + - The method uses a class variable `depth` to manage indentation + levels for nested directories, enhancing readability. + - The output format varies depending on whether the node is a + leaf or a directory. + + Returns: + A formatted string representation of the + [`SampleIndex`][common.sample_index.SampleIndex] object, + including its children if applicable. + """ SampleIndex.depth = SampleIndex.depth + 1 if self.is_leaf: result = (" " * SampleIndex.depth) + f"{self.address}: BUNDLE {self.leafid} MIN {self.min} MAX {self.max}\n" diff --git a/merlin/common/sample_index_factory.py b/merlin/common/sample_index_factory.py index 75de4676b..f9cbacd7c 100644 --- a/merlin/common/sample_index_factory.py +++ b/merlin/common/sample_index_factory.py @@ -1,36 +1,14 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ -SampleIndex factory methods +This module houses [`SampleIndex`][common.sample_index.SampleIndex] factory methods. """ +from typing import List + from parse import parse from merlin.common.sample_index import MAX_SAMPLE, SampleIndex @@ -43,27 +21,38 @@ def create_hierarchy( - num_samples, - bundle_size, - directory_sizes=None, - root=".", - start_sample_id=0, - start_bundle_id=0, - address="", - n_digits=1, -): + num_samples: int, + bundle_size: int, + directory_sizes: List[int] = None, + root: str = ".", + start_sample_id: int = 0, + start_bundle_id: int = 0, + address: str = "", + n_digits: int = 1, +) -> SampleIndex: """ - SampleIndex Hierarchy Factory method. Wraps - create_hierarchy_from_max_sample, which is a max_sample-based API, not a - numSample-based API like this method. - - :param num_samples: The total number of samples. - :bundle_size: The max number of samples a bundle file is responsible for. - :directory_sizes: The number of samples each directory is responsible - for - a list, one value for each level in the directory hierarchy. - :root: The root path of this index. Defaults to ".". - :start_sample_id: The start of the sample count. Defaults to 0. - :n_digits: The number of digits to pad the directories with + Factory method to create a [`SampleIndex`][common.sample_index.SampleIndex] + hierarchy based on the number of samples. + + This method wraps the + [`create_hierarchy_from_max_sample`][common.sample_index_factory.create_hierarchy_from_max_sample] + function, which operates on a maximum sample basis rather than a total + sample count. + + Args: + num_samples (int): The total number of samples. + bundle_size (int): The maximum number of samples a bundle file can handle. + directory_sizes (List[int]): A list specifying the number of samples each directory + is responsible for. + root (str): The root path of the index. + start_sample_id (int): The starting sample ID. + start_bundle_id (int): The starting bundle ID. + address (str): An optional address prefix for the hierarchy. + n_digits (int): The number of digits to pad the directory names. + + Returns: + (common.sample_index.SampleIndex): The root [`SampleIndex`][common.sample_index.SampleIndex] + object representing the hierarchy. """ if directory_sizes is None: directory_sizes = [] @@ -80,29 +69,37 @@ def create_hierarchy( def create_hierarchy_from_max_sample( - max_sample, - bundle_size, - directory_sizes=None, - root=".", - start_bundle_id=0, - min_sample=0, - address="", - n_digits=1, -): - """ " - Construct the SampleIndex based off the total number of samples and the - chunking size at each depth in the hierarchy. - - This method will add new SampleIndex objects as this SampleIndex's - children if directory_sizes is not the empty set. - - :param max_sample: The max Sample ID this hierarchy is responsible for. - :bundle_size: The max number of samples a bundle file is responsible for. - :directory_sizes: The number of samples each directory is responsible - for - a list, one value for each level in the directory hierarchy. - :bundle_id: The current bundle_id count. - :min_sample: The start of the sample count. - :n_digits: The number of digits to pad the directories with + max_sample: int, + bundle_size: int, + directory_sizes: List[int] = None, + root: str = ".", + start_bundle_id: int = 0, + min_sample: int = 0, + address: str = "", + n_digits: int = 1, +) -> SampleIndex: + """ + Constructs a [`SampleIndex`][common.sample_index.SampleIndex] hierarchy based on + the maximum sample ID and chunking size at each depth. + + This method adds new [`SampleIndex`][common.sample_index.SampleIndex] objects as + children if `directory_sizes` is provided. + + Args: + max_sample: The maximum Sample ID this hierarchy is responsible for. + bundle_size: The maximum number of samples a bundle file can handle. + directory_sizes: A list specifying the number of samples each directory + is responsible for. + root: The root path of this index. + start_bundle_id: The starting bundle ID. + min_sample: The starting sample ID. + address: An optional address prefix for the hierarchy. + n_digits: The number of digits to pad the directory names. + + Returns: + (common.sample_index.SampleIndex): The root + [`SampleIndex`][common.sample_index.SampleIndex] object representing + the constructed hierarchy. """ if directory_sizes is None: directory_sizes = [] @@ -158,9 +155,23 @@ def create_hierarchy_from_max_sample( return SampleIndex(min_sample, max_sample, children, root, num_bundles=num_bundles, address=address) -def read_hierarchy(path): +def read_hierarchy(path: str) -> SampleIndex: """ - TODO + Reads a hierarchy from a specified path and constructs a + [`SampleIndex`][common.sample_index.SampleIndex]. + + This function reads a file named "sample_index.txt" in the given path, + parsing its contents to create a hierarchical structure of + [`SampleIndex`][common.sample_index.SampleIndex] objects based on the + information found in the file. + + Args: + path: The directory path where the sample index file is located. + + Returns: + (common.sample_index.SampleIndex): The root + [`SampleIndex`][common.sample_index.SampleIndex] object representing + the hierarchy read from the file. """ children = {} min_sample = MAX_SAMPLE diff --git a/merlin/common/security/__init__.py b/merlin/common/security/__init__.py index 37cabcad1..5fbc0ecb5 100644 --- a/merlin/common/security/__init__.py +++ b/merlin/common/security/__init__.py @@ -1,29 +1,14 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +The `security` package contains functionality for managing encryption within Merlin, +ensuring secure communication and data protection. + +Modules: + encrypt.py: Handles general encryption logic. + encrypt_backend_traffic.py: Provides functions for encrypting backend traffic. +""" diff --git a/merlin/common/security/encrypt.py b/merlin/common/security/encrypt.py index e38b0ebbd..b514be06e 100644 --- a/merlin/common/security/encrypt.py +++ b/merlin/common/security/encrypt.py @@ -1,32 +1,8 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """This module handles encryption logic""" @@ -45,8 +21,19 @@ LOG = logging.getLogger(__name__) -def _get_key_path(): - """Loads the redis encryption key path from the file described in config.""" +def _get_key_path() -> str: + """ + Loads the path to the Redis encryption key from the configuration. + + If the path is not specified in the configuration, it defaults to + "~/.merlin/encrypt_data_key". + + Returns: + A string representing the absolute path to the encryption key file. + + Raises: + ValueError: If there is an issue retrieving the key path from the configuration. + """ try: key_filepath = CONFIG.results_backend.encryption_key except AttributeError: @@ -58,8 +45,13 @@ def _get_key_path(): return os.path.abspath(os.path.expanduser(key_filepath)) -def _gen_key(key_path): - """generates an encryption key and writes it to the given key_path""" +def _gen_key(key_path: str): + """ + Generates a new encryption key and writes it to the specified key path. + + Args: + key_path: The path where the encryption key will be stored. + """ key = Fernet.generate_key() parent_dir = os.path.dirname(os.path.normpath(key_path)) if not os.path.isdir(parent_dir): @@ -68,9 +60,20 @@ def _gen_key(key_path): f.write(key) -def _get_key(): - """get a valid encryption key. Loads from CONFIG.results_backend.encryption_key if possible, - initializes that key if it does not yet exist.""" +def _get_key() -> bytes: + """ + Retrieves a valid encryption key. + + This function attempts to load the key from the path specified in the + configuration. If the key does not exist, it generates a new key and + saves it to the specified path. + + Returns: + The encryption key in bytes format. + + Raises: + IOError: If there is an issue reading the key file or generating a new key. + """ key_path = _get_key_path() try: with open(key_path, "rb") as f: @@ -87,9 +90,15 @@ def _get_key(): return key -def encrypt(payload): +def encrypt(payload: bytes) -> bytes: """ - TODO + Encrypts the given payload using a Fernet key. + + Args: + payload: The data to be encrypted. Must be in bytes format. + + Returns: + The encrypted data in bytes format. """ key = _get_key() f = Fernet(key) @@ -97,9 +106,15 @@ def encrypt(payload): return f.encrypt(payload) -def decrypt(payload): +def decrypt(payload: bytes) -> bytes: """ - TODO + Decrypts the given payload using a Fernet key. + + Args: + payload: The encrypted data to be decrypted. Must be in bytes format. + + Returns: + The decrypted data in bytes format. """ key = _get_key() f = Fernet(key) @@ -109,7 +124,9 @@ def decrypt(payload): def init_key(): """ - Initialize the key to disk on import to prevent race conditions later on, or at least drastically reduce - the number of corner cases where they could appear. + Initializes the Fernet key and stores it on disk. + + This function is called on import to prevent race conditions later on, + or at least drastically reduce the number of corner cases where they could appear. """ Fernet(_get_key()) diff --git a/merlin/common/security/encrypt_backend_traffic.py b/merlin/common/security/encrypt_backend_traffic.py index 23bdaacf8..ebfb08939 100644 --- a/merlin/common/security/encrypt_backend_traffic.py +++ b/merlin/common/security/encrypt_backend_traffic.py @@ -1,37 +1,16 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ Functions for encrypting backend traffic. """ +from typing import Any + import celery.backends.base +from celery.backends.base import Backend from merlin.common.security import encrypt @@ -44,25 +23,47 @@ old_decode = celery.backends.base.Backend.decode -def _encrypt_encode(*args, **kwargs): +def _encrypt_encode(*args, **kwargs) -> bytes: """ - Intercept all celery.backends.Backend.encode calls and encrypt them after - encoding + Intercepts calls to the encode method of the Celery backend and encrypts + the encoded data. + + This function wraps the original encode method, encrypting the result + after encoding. + + Returns: + The encrypted encoded data in bytes format. """ return encrypt.encrypt(old_encode(*args, **kwargs)) -def _decrypt_decode(self, payload): +def _decrypt_decode(self: Backend, payload: bytes) -> Any: """ - Intercept all celery.backends.Backend.decode calls and decrypt them before - decoding. + Intercepts calls to the decode method of the Celery backend and decrypts + the payload before decoding. + + This function wraps the original decode method, decrypting the payload + prior to decoding. + + Args: + self: The instance of the backend from which the decode method is called. + payload: The encrypted data to be decrypted. + + Returns: + The decoded data after decryption. Can be any format. """ return old_decode(self, encrypt.decrypt(payload)) def set_backend_funcs(): """ - Set the encode / decode to our own encrypt_encode / encrypt_decode. + Sets the encode and decode methods of the Celery backend to custom + implementations that handle encryption and decryption. + + This function replaces the default encode and decode methods with + `_encrypt_encode` and `_decrypt_decode`, respectively, ensuring that + all data processed by the Celery backend is encrypted and decrypted + appropriately. """ celery.backends.base.Backend.encode = _encrypt_encode celery.backends.base.Backend.decode = _decrypt_decode diff --git a/merlin/common/tasks.py b/merlin/common/tasks.py index 736d92225..fe04eafbf 100644 --- a/merlin/common/tasks.py +++ b/merlin/common/tasks.py @@ -1,69 +1,66 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### - -"""Test tasks.""" +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module contains Celery task definitions. + +The purpose of this module is to convert the Directed Acyclic Graph +([`DAG`][study.dag.DAG]) provided by Maestro into smaller tasks that +Celery can manage. +""" from __future__ import absolute_import, unicode_literals import json import logging import os -from typing import Any, Dict, List, Optional +from typing import Any, Callable, Dict, List, Optional # Need to disable an overwrite warning here since celery has an exception that we need that directly # overwrites a python built-in exception -from celery import chain, chord, group, shared_task, signature -from celery.exceptions import MaxRetriesExceededError, OperationalError, TimeoutError # pylint: disable=W0622 +from celery import Signature, Task, chain, chord, group, shared_task, signature +from celery.exceptions import MaxRetriesExceededError +from celery.exceptions import OperationalError as CeleryOperationalError +from celery.exceptions import TimeoutError as CeleryTimeoutError +from celery.result import AsyncResult from filelock import FileLock, Timeout +from kombu.exceptions import OperationalError as KombuOperationalError from redis.exceptions import TimeoutError as RedisTimeoutError -from merlin.common.abstracts.enums import ReturnCode -from merlin.common.sample_index import uniform_directories +from merlin.common.enums import ReturnCode +from merlin.common.sample_index import SampleIndex, uniform_directories from merlin.common.sample_index_factory import create_hierarchy from merlin.config.utils import Priority, get_priority +from merlin.db_scripts.merlin_db import MerlinDatabase from merlin.exceptions import HardFailException, InvalidChainException, RestartException, RetryException from merlin.router import stop_workers from merlin.spec.expansion import parameter_substitutions_for_cmd, parameter_substitutions_for_sample +from merlin.study.dag import DAG from merlin.study.status import read_status, status_conflict_handler +from merlin.study.step import Step +from merlin.study.study import MerlinStudy from merlin.utils import dict_deep_merge retry_exceptions = ( + # Python Built-in Exceptions IOError, OSError, AttributeError, TimeoutError, - OperationalError, - RetryException, - RestartException, FileNotFoundError, + # Celery Exceptions + CeleryOperationalError, + CeleryTimeoutError, + # Kombu Exceptions + KombuOperationalError, + # Redis Exceptions RedisTimeoutError, + # Merlin Exceptions + RetryException, + RestartException, ) LOG = logging.getLogger(__name__) @@ -84,20 +81,42 @@ retry_backoff=True, priority=get_priority(Priority.HIGH), ) -def merlin_step(self, *args: Any, **kwargs: Any) -> Optional[ReturnCode]: # noqa: C901 pylint: disable=R0912,R0915 +def merlin_step(self: Task, *args: Any, **kwargs: Any) -> ReturnCode: # noqa: C901 pylint: disable=R0912,R0915 """ - Executes a Merlin Step - :param args: The arguments, one of which should be an instance of Step - :param kwargs: The optional keyword arguments that describe adapter_config and - the next step in the chain, if there is one. - - Example kwargs dict: - {"adapter_config": {'type':'local'}, - "next_in_chain": } # merlin_step will be added to the current chord - # with next_in_chain as an argument + Executes a Merlin step. + + This task executes a step in the Merlin workflow, handling various + outcomes such as success, retries, and failures. It can also manage + chaining to the next step in the workflow. + + Notes: + - If the step has already been completed, it will be skipped. + + Args: + self: The current task instance. + *args: Positional arguments, one of which should be an instance + of [`Step`][study.step.Step]. + **kwargs: Optional keyword arguments that include:\n + - adapter_config (`Dict`): Configuration for the adapter, + defaulting to `{'type': 'local'}`. + - next_in_chain ([`Step`][study.step.Step]): The next step in + the workflow chain, if applicable.\n + Example kwargs dict where `merlin_step` will be added to the + current chord with `next_in_chain` as an argument:\n + ``` + { + "adapter_config": { + 'type': 'local' + }, + "next_in_chain": + } + ``` + + Returns: + (common.enums.ReturnCode): The result of the step + execution, which can indicate success, various failure modes, + or a request to retry. """ - from merlin.study.step import Step # pylint: disable=C0415 - step: Optional[Step] = None LOG.debug(f"args is {len(args)} long") @@ -204,15 +223,29 @@ def merlin_step(self, *args: Any, **kwargs: Any) -> Optional[ReturnCode]: # noq return None -def is_chain_expandable(chain_, labels): +def is_chain_expandable(chain_: List[Step], labels: List[str]) -> bool: """ - Returns whether to expand the steps in the given chain. - A chain_ is expandable if all the steps are expandable. - It is not expandable if none of the steps are expandable. - If neither expandable nor not expandable, we raise an InvalidChainException. - :param chain_: A list of Step objects representing chain of dependent steps. - :param labels: The labels - + Determine if the steps in the given chain are expandable. + + A chain is considered expandable if all steps within the chain require + expansion. Conversely, if none of the steps require expansion, the chain + is not expandable. If there is a mix of steps that require expansion and + those that do not, an `InvalidChainException` is raised, indicating that + the chain is incompatible. + + Args: + chain_ (List[study.step.Step]): A list of [`Step`][study.step.Step] + objects representing a chain of dependent steps. + labels: The labels associated with the steps in the chain, used to + determine if expansion is needed. + + Returns: + True if all steps in the chain are expandable, False if none are + expandable. + + Raises: + InvalidChainException: If there is a mix of steps that require + expansion and those that do not, indicating an incompatible chain. """ array_of_bools = [step.check_if_expansion_needed(labels) for step in chain_] @@ -235,11 +268,19 @@ def is_chain_expandable(chain_, labels): return needs_expansion -def prepare_chain_workspace(sample_index, chain_): +def prepare_chain_workspace(sample_index: SampleIndex, chain_: List[Step]): """ - Prepares a user's workspace for each step in the given chain. - :param chain_: A list of Step objects representing chain of dependent steps. - :param labels: The labels + Prepares a user's workspace for each step in the given chain of dependent steps. + + This function iterates through a list of [`Step`][study.step.Step] objects and + prepares the necessary workspace for each step by creating directories and writing + sample index files. + + Args: + sample_index (common.sample_index.SampleIndex): An object that manages sample + indexing and workspace preparation. + chain_ (List[study.step.Step]): A list of [`Step`][study.step.Step] objects + representing a chain of dependent steps. Each step's workspace will be prepared. """ # TODO: figure out faster way to create these directories (probably using # yet another task) @@ -261,25 +302,36 @@ def prepare_chain_workspace(sample_index, chain_): priority=get_priority(Priority.LOW), ) def add_merlin_expanded_chain_to_chord( # pylint: disable=R0913,R0914 - self, - task_type, - chain_, - samples, - labels, - sample_index, - adapter_config, - min_sample_id, + self: Task, + task_type: Signature, + chain_: List[Step], + samples: List[Any], + labels: List[str], + sample_index: SampleIndex, + adapter_config: Dict, + min_sample_id: int, ): """ - Expands tasks in a chain, then adds the expanded tasks to the current chord. - :param self: The current task. - :param task_type: The celery task signature type the new tasks should be. - :param chain_: The list of tasks to expand. - :param samples: The sample values to use for each new task. - :param labels: The sample labels. - :param sample_index: The sample index that contains the directory structure for tasks. - :param adapter_config: The adapter config. - :param min_sample_id: offset to use for the sample_index. + Expand tasks in a chain and add the expanded tasks to the current chord. + + This Celery task recursively expands a chain of tasks based on provided + sample values and their corresponding labels. The expanded tasks are + configured with specific parameters and added to the current chord for + execution. The function handles both the expansion of tasks and the + management of task dependencies. + + Args: + self: The current task instance. + task_type: The Celery task signature type for the new tasks to be + created. + chain_ (List[study.step.Step]): A list of tasks to expand into a chain. + samples: The sample values to use for each new task. + labels: The sample labels corresponding to the samples. + sample_index (common.sample_index.SampleIndex): The sample index that + contains the directory structure for tasks. + adapter_config: Configuration settings for the adapter used in task + execution. + min_sample_id: An offset to use for the sample index. """ num_samples = len(samples) # Use the index to get a path to each sample @@ -368,13 +420,27 @@ def add_merlin_expanded_chain_to_chord( # pylint: disable=R0913,R0914 return ReturnCode.OK -def add_simple_chain_to_chord(self, task_type, chain_, adapter_config): +def add_simple_chain_to_chord(self: Task, task_type: Signature, chain_: List[Step], adapter_config: Dict): """ - Adds a chain of tasks to the current chord. - :param self: The current task. - :param task_type: The celery task signature type the new tasks should be. - :param chain_: The list of tasks to expand. - :param adapter_config: The adapter config. + Add a chain of tasks to the current chord for execution. + + This function takes a list of tasks, modifies their signatures based on + provided parameters, and adds them to the current chord. Each task in the + chain is transformed into a new task signature with specific configurations + such as queue and task ID. + + This function takes a list of steps and creates signatures based on the + parameters they provide, such as queue and workspace. It then adds these + signatures to the current chord for later execution. + + Args: + self: The current task instance invoking this method. + task_type: The Celery task signature type that the new tasks should be + based on. + chain_ (List[study.step.Step]): A list of tasks to expand into a chain. + Each task should provide necessary parameters for signature creation. + adapter_config: Configuration settings for the adapter used in task + execution. """ LOG.debug(f"simple chain with {chain_}") all_chains = [] @@ -394,18 +460,20 @@ def add_simple_chain_to_chord(self, task_type, chain_, adapter_config): launch_chain(self, chain_1d) -def launch_chain(self: "Task", chain_1d: List["Signature"], condense_sig: "Signature" = None): # noqa: F821 +def launch_chain(self: Task, chain_1d: List[Signature], condense_sig: Signature = None): """ - Given a 1D chain, appropriately launch the signatures it contains. - If this is a local run, launch the signatures instantly. - Otherwise, there's two cases: - a. The chain is dealing with samples (i.e. we'll need to condense status files) - so create a new chord and add it to the current chord - b. The chain is NOT dealing with samples so we can just add the signatures to the current chord - - :param `self`: The current task - :param `chain_1d`: A 1-dimensional list of signatures to launch - :param `condense_sig`: A signature for condensing the status files. None if condensing isn't needed. + Launch a 1D chain of task signatures appropriately based on the execution context. + + This function handles the launching of a list of task signatures in a + one-dimensional chain. The behavior varies depending on whether the + execution is local or remote, and whether the tasks involve sample + processing that requires condensing status files. + + Args: + self: The current task instance invoking this method. + chain_1d: A one-dimensional list of task signatures to be launched. + condense_sig: A signature for condensing the status files after task execution. + If None, condensing is not required. """ # If there's nothing in the chain then we won't have to launch anything so check that first if chain_1d: @@ -426,11 +494,28 @@ def launch_chain(self: "Task", chain_1d: List["Signature"], condense_sig: "Signa self.add_to_chord(sig, lazy=False) -def get_1d_chain(all_chains: List[List["Signature"]]) -> List["Signature"]: # noqa: F821 +def get_1d_chain(all_chains: List[List[Signature]]) -> List[Signature]: """ - Convert a 2D list of chains into a 1D list. - :param all_chains: Two-dimensional list of chains [chain_length][number_of_chains] - :returns: A one-dimensional list representing a chain of tasks + Convert a 2D list of task chains into a 1D list of task signatures. + + This function takes a two-dimensional list of task signatures, where each + inner list represents a parallel group of tasks. It transforms this structure + into a one-dimensional list suitable for creating a linear chain of tasks. + If there is only one chain, it returns that chain directly. If there are + multiple chains, it sets up dependencies between tasks to ensure proper + execution order. + + Notes: + - The function processes the chains in reverse order to correctly + set up the dependencies before adding them to the final list. + + Args: + all_chains: A two-dimensional list of task signatures, where each inner + list represents a group of tasks that can be executed in parallel. + + Returns: + A one-dimensional list of task signatures representing a chain of tasks, + with dependencies set up for proper execution order. """ chain_steps = [] if len(all_chains) == 1: @@ -465,17 +550,37 @@ def get_1d_chain(all_chains: List[List["Signature"]]) -> List["Signature"]: # n return chain_steps -def gather_statuses( - sample_index: "SampleIndex", workspace: str, condensed_workspace: str, files_to_remove: List[str] # noqa: F821 -) -> Dict: +def gather_statuses(sample_index: SampleIndex, workspace: str, condensed_workspace: str, files_to_remove: List[str]) -> Dict: """ - Traverse the sample index and gather all of the statuses into one. - - :param `sample_index`: A SampleIndex object to track this specific sample hierarchy - :param `workspace`: The full workspace path to the step we're condensing for - :param `condensed_workspace`: A shortened version of `workspace` that's saved in the status files - :param `files_to_remove`: An empty list that we'll add filepaths to that need removed - :returns: A dict of condensed statuses + Traverse the sample index and gather all statuses into a single dictionary. + + This function iterates through the provided + [`SampleIndex`][common.sample_index.SampleIndex] object, + reading status files from each sample's workspace. It condenses + the statuses into a single dictionary while tracking which files + need to be removed after condensing. The function ensures that + only completed statuses are included in the condensed output. + + Args: + sample_index (common.sample_index.SampleIndex): A + [`SampleIndex`][common.sample_index.SampleIndex] object + representing the specific sample hierarchy to traverse. + workspace: The full path to the workspace for the step being + condensed. + condensed_workspace: A shortened version of the workspace + path that will be used in the status files. + files_to_remove: A list that will be populated with file paths + of status files that need to be removed after condensing. + + Returns: + A dictionary containing the condensed statuses gathered + from the status files. + + Raises: + TimeoutError: If a timeout occurs while reading a status file, + triggering a restart of the task. + FileNotFoundError: If a status file is not found during the + condensing process. """ LOG.info(f"Gathering statuses to condense for '{condensed_workspace}'") condensed_statuses = {} @@ -520,15 +625,38 @@ def gather_statuses( retry_backoff=True, priority=get_priority(Priority.LOW), ) -def condense_status_files(self, *args: Any, **kwargs: Any) -> ReturnCode: # pylint: disable=R0914,W0613 +def condense_status_files(self: Task, *args: Any, **kwargs: Any) -> ReturnCode: # pylint: disable=R0914,W0613 """ - After a section of the sample tree has finished, condense the status files. - - kwargs should look like so: - kwargs = { - "sample_index": SampleIndex Object, - "workspace": str representing the step's workspace - } + Condenses status files after a section of the sample tree has completed processing. + + This task gathers status information from a specified + [`SampleIndex`][common.sample_index.SampleIndex] and condenses it into a single + JSON file. It handles potential race conditions by using a file lock during + the write operation. If the condensed status file already exists, it merges + the new statuses with the existing ones. + + Notes: + - The task will remove the original status files after condensing them + into the JSON file. + + Args: + self: The current task instance. + *args: Additional positional arguments (not used in this task). + **kwargs: Keyword arguments containing:\n + - `sample_index` ([`SampleIndex`][common.sample_index.SampleIndex]): + The [`SampleIndex`][common.sample_index.SampleIndex] object used + for gathering statuses. + - `workspace` (str): The workspace path for the step. + - `condensed_workspace` (str): The workspace path for the + condensed status. + + Returns: + (common.enums.ReturnCode): A [`ReturnCode.OK`][common.enums.ReturnCode] + message if the operation was successful. None, otherwise. + + Raises: + TimeoutError: If the file lock cannot be acquired within the + specified timeout period, which triggers a task restart. """ # Get the sample index object that we'll use for condensing sample_index = kwargs.pop("sample_index", None) @@ -592,26 +720,40 @@ def condense_status_files(self, *args: Any, **kwargs: Any) -> ReturnCode: # pyl priority=get_priority(Priority.LOW), ) def expand_tasks_with_samples( # pylint: disable=R0913,R0914 - self, - dag, - chain_, - samples, - labels, - task_type, - adapter_config, - level_max_dirs, + self: Task, + dag: DAG, + chain_: List[str], + samples: List[List[str]], + labels: List[str], + task_type: Callable, + adapter_config: Dict, + level_max_dirs: int, ): """ - Generate a group of celery chains of tasks from a chain of task names, using merlin - samples and labels to do variable substitution. - - :param dag : A Merlin DAG. - :param chain_ : The list of task names to expand into a celery group of celery chains. - :param samples : The list of lists of merlin sample values to do substitution for. - :labels : A list of strings containing the label associated with each column in the samples. - :task_type : The celery task type to create. Currently always merlin_step. - :adapter_config : A dictionary used for configuring maestro script adapters. - :level_max_dirs : The max number of directories per level in the sample hierarchy. + Expands a chain of task names into a group of Celery chains, using samples + and labels for variable substitution. + + This task determines whether the provided chain of tasks requires + expansion based on the structure of the Directed Acyclic Graph ([`DAG`][study.dag.DAG]), + samples, and labels. If expansion is needed, it generates and queues new tasks + for each range of samples. Otherwise, it queues a simple chain task. + + Args: + self: The current task instance. + dag (study.dag.DAG): A Merlin Directed Acyclic Graph + ([`DAG`][study.dag.DAG]) representing the workflow. + chain_: A list of task names to be expanded into a + Celery group of chains. + samples: A list of lists containing Merlin sample values for + variable substitution. + labels: A list of strings representing the labels associated + with each column in the samples. + task_type: The Celery task type to create, currently expected + to be [`merlin_step`][common.tasks.merlin_step]. + adapter_config: A configuration dictionary for Maestro + script adapters. + level_max_dirs: The maximum number of directories allowed per + level in the sample hierarchy. """ LOG.debug(f"expand_tasks_with_samples called with chain,{chain_}\n") # Figure out how many directories there are, make a glob string @@ -703,15 +845,19 @@ def expand_tasks_with_samples( # pylint: disable=R0913,R0914 name="merlin:shutdown_workers", priority=get_priority(Priority.HIGH), ) -def shutdown_workers(self, shutdown_queues): # pylint: disable=W0613 +def shutdown_workers(self: Task, shutdown_queues: List[str]): # pylint: disable=W0613 """ - This task issues a call to shutdown workers. + Initiates the shutdown of Celery workers. - It wraps the stop_celery_workers call as a task. - It is acknolwedged right away, so that it will not be requeued when - executed by a worker. + This task wraps the [`stop_celery_workers`][study.celeryadapter.stop_celery_workers] + function, allowing for the graceful shutdown of specified Celery worker queues. It is + acknowledged immediately upon execution, ensuring that it will not be requeued, even + if executed by a worker. - :param: shutdown_queues: The specific queues to shutdown (list) + Args: + self: The current task instance. + shutdown_queues: A list of specific queues to shut down. If None, all queues will + be shut down. """ if shutdown_queues is not None: LOG.warning(f"Shutting down workers in queues {shutdown_queues}!") @@ -728,26 +874,80 @@ def shutdown_workers(self, shutdown_queues): # pylint: disable=W0613 name="merlin:chordfinisher", priority=get_priority(Priority.LOW), ) -def chordfinisher(*args, **kwargs): # pylint: disable=W0613 - """. - It turns out that chain(group,group) in celery does not execute one group - after another, but executes the groups as if they were independent from - one another. To get a sync point between groups, we use this method as a - callback to enforce sync points for chords so we can declare chains of groups - dynamically. +def chordfinisher(*args: List, **kwargs: Dict) -> str: # pylint: disable=W0613 + """ + Synchronization callback for Celery chords. + + This function serves as a synchronization point between groups of tasks + in a Celery workflow. In Celery, using `chain(group, group)` does not + guarantee that the second group will execute only after the first group + has completed. Instead, both groups are executed independently. + + To enforce a synchronization point between these groups, this function + is used as a callback in a chord. It allows for the declaration of chains + of groups dynamically, ensuring that subsequent tasks wait for the + completion of all tasks in the preceding groups. + + Args: + *args: Variable length argument list. Needed by Celery. + **kwargs: Arbitrary keyword arguments. Needed by Celery. + + Returns: + A constant string "SYNC" indicating the synchronization point + has been reached. """ return "SYNC" +@shared_task( + autoretry_for=retry_exceptions, + retry_backoff=True, + name="merlin:mark_run_as_complete", + priority=get_priority(Priority.LOW), +) +def mark_run_as_complete(study_workspace: str) -> str: + """ + Mark this run as complete and save that to the database. + + Args: + study_workspace: The output workspace for this run. + + Returns: + A string denoting that this run has completed. + """ + merlin_db = MerlinDatabase() + run_entity = merlin_db.get("run", study_workspace) + run_entity.run_complete = True + run_entity.save() + return "Run Completed" + + @shared_task( autoretry_for=retry_exceptions, retry_backoff=True, name="merlin:queue_merlin_study", priority=get_priority(Priority.LOW), ) -def queue_merlin_study(study, adapter): +def queue_merlin_study(study: MerlinStudy, adapter: Dict) -> AsyncResult: """ - Launch a chain of tasks based off of a MerlinStudy. + Launch a chain of tasks based on a MerlinStudy. + + This Celery task initiates a series of tasks derived from a + [`MerlinStudy`][study.study.MerlinStudy] object. It processes + the study's Directed Acyclic Graph ([`DAG`][study.dag.DAG]) + to group tasks and convert them into a chain of Celery tasks + for execution. + + Args: + study: The study object containing samples, sample labels, + and the Directed Acyclic Graph ([`DAG`][study.dag.DAG]) + structure that defines the task dependencies. + adapter: An adapter object used to facilitate interactions with + the study's data or processing logic. + + Returns: + An instance representing the asynchronous result of the task chain, + allowing for tracking and management of the task's execution. """ samples = study.samples sample_labels = study.sample_labels @@ -777,5 +977,14 @@ def queue_merlin_study(study, adapter): ) for chain_group in groups_of_chains[1:] ) + + # Append the final task that marks the run as complete + final_task = mark_run_as_complete.si(study.workspace).set( + queue=egraph.step( + groups_of_chains[-1][-1][-1] # Use the task queue from the final step to execute this task + ).get_task_queue() + ) + celery_dag = celery_dag | final_task + LOG.info("Launching tasks.") return celery_dag.delay(None) diff --git a/merlin/common/util_sampling.py b/merlin/common/util_sampling.py index 21c97c709..4b57f5bcd 100644 --- a/merlin/common/util_sampling.py +++ b/merlin/common/util_sampling.py @@ -1,70 +1,59 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ Utility functions for sampling. """ +from typing import List, Tuple, Union import numpy as np # TODO should we move this to merlin-spellbook? -def scale_samples(samples_norm, limits, limits_norm=(0, 1), do_log=False): - """Scale samples to new limits, either log10 or linearly. +def scale_samples( + samples_norm: np.ndarray, + limits: List[Tuple[int, int]], + limits_norm: Tuple[int, int] = (0, 1), + do_log: Union[bool, List[bool]] = False, +) -> np.ndarray: + """ + Scale samples to new limits, either logarithmically or linearly. + + This function transforms normalized samples to specified limits, + allowing for both linear and logarithmic scaling based on the + provided parameters. Args: - samples_norm (ndarray): The normalized samples to scale, - with dimensions (nsamples,ndims). - limits (list of tuples): A list of (min, max) for the various - dimensions. Length of list is ndims. - limits_norm (tuple of floats, optional): The (min, max) from which - samples_norm were drawn. Defaults to (0,1). - do_log (boolean or list of booleans, optional): Whether - to log10 scale each dimension. Either a single boolean or - a list of length ndims, for each dimension. - Defaults to ndims*[False]. + samples_norm: The normalized samples to scale, with dimensions + (nsamples, ndims). + limits: A list of (min, max) tuples for the various dimensions. + The length of the list must match the number of dimensions (ndims). + limits_norm: The (min, max) values from which `samples_norm` were + derived. Defaults to (0, 1). + do_log: Indicates whether to apply log10 scaling to each dimension. + This can be a single boolean or a list of length ndims. Defaults + to a list of `ndims` containing `False`. Returns: - ndarray: The scaled samples. + The scaled samples, with the same shape as `samples_norm`. - Note: - We follow the sklearn convention of requiring samples to be - given as an (nsamples, ndims) array. + Raises: + ValueError: If `samples_norm` does not have two dimensions. - To transform 1-D arrays: - - >>> samples = samples.reshape((-1,1)) # ndims = 1 - >>> samples = samples.reshape((1,-1)) # nsamples = 1 + Notes: + - The function follows the sklearn convention, requiring + samples to be provided as an (nsamples, ndims) array. + - To transform 1-D arrays, reshape them accordingly: + ```python + >>> samples = samples.reshape((-1, 1)) # ndims = 1 + >>> samples = samples.reshape((1, -1)) # nsamples = 1 + ``` Example: - + ```python >>> # Turn 0:1 samples into -1:1 >>> import numpy as np >>> norm_values = np.linspace(0,1,5).reshape((-1,1)) @@ -83,6 +72,7 @@ def scale_samples(samples_norm, limits, limits_norm=(0, 1), do_log=False): [ 1.00000000e+02] [ 1.00000000e+03] [ 1.00000000e+04]] + ``` """ norms = np.asarray(samples_norm) if len(norms.shape) != 2: diff --git a/merlin/config/__init__.py b/merlin/config/__init__.py index f1451b4c4..65f96dec6 100644 --- a/merlin/config/__init__.py +++ b/merlin/config/__init__.py @@ -1,35 +1,24 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ Used to store the application configuration. + +The `config` package provides functionality for managing and configuring various aspects +of the Merlin application, including broker settings, results backends, Celery configurations, +and application-level settings. It serves as the central hub for loading, processing, and +utilizing configuration data defined in the `app.yaml` file and other related resources. + +Modules: + broker.py: Manages broker configurations and connection strings for messaging systems. + celeryconfig.py: Contains default Celery configuration settings for Merlin. + configfile.py: Handles the loading and processing of application configuration files + and SSL-related settings. + results_backend.py: Configures connection strings and SSL settings for results backends. + utils.py: Provides utilities for broker priority handling and validation. """ from copy import copy from types import SimpleNamespace @@ -46,9 +35,29 @@ class Config: # pylint: disable=R0903 The Config class, meant to store all Merlin config settings in one place. Regardless of the config data loading method, this class is meant to standardize config data retrieval throughout all parts of Merlin. + + Attributes: + celery (Optional[SimpleNamespace]): A namespace containing Celery configuration settings. + broker (Optional[SimpleNamespace]): A namespace containing broker configuration settings. + results_backend (Optional[SimpleNamespace]): A namespace containing results backend configuration settings. + + Methods: + __copy__: Creates a shallow copy of the Config instance. + __str__: Returns a formatted string representation of the Config instance. + load_app_into_namespaces: Converts the provided configuration dictionary into namespaces + and assigns them to the Config instance's attributes. """ - def __init__(self, app_dict): + def __init__(self, app_dict: Dict): + """ + Initializes the Config instance with configuration data from a dictionary. + + Args: + app_dict: A dictionary containing configuration data for the application. + The dictionary may include keys such as "celery", "broker", and "results_backend", + each of which is converted into a `SimpleNamespace` and assigned to the corresponding + attribute of the Config instance. + """ # I think this ends up a SimpleNamespace from load_app_into_namespaces, but it seems like it should be typed as # the app var in celery.py, as celery.app.base.Celery self.celery: Optional[SimpleNamespace] @@ -56,9 +65,12 @@ def __init__(self, app_dict): self.results_backend: Optional[SimpleNamespace] self.load_app_into_namespaces(app_dict) - def __copy__(self): + def __copy__(self) -> "Config": """ - A magic method to allow this class to be copied with copy(instance_of_Config). + Creates a shallow copy of the Config instance. + + Returns: + A new Config instance with copied `celery`, `broker`, and `results_backend` attributes. """ cls = self.__class__ result = cls.__new__(cls) @@ -70,9 +82,12 @@ def __copy__(self): result.__dict__.update(copied_attrs) return result - def __str__(self): + def __str__(self) -> str: """ - A magic method so we can print the CONFIG class. + Returns a formatted string representation of the Config instance. + + Returns: + str: A string containing the values of the `celery`, `broker`, and `results_backend` attributes. """ formatted_str = "config:" attrs = {"celery": self.celery, "broker": self.broker, "results_backend": self.results_backend} @@ -85,9 +100,13 @@ def __str__(self): formatted_str += f"\n {name}:\n None" return formatted_str - def load_app_into_namespaces(self, app_dict: Dict) -> None: + def load_app_into_namespaces(self, app_dict: Dict): """ - Makes the application dictionary into a namespace, sets the attributes of the Config from the namespace values. + Converts the provided application dictionary into namespaces and assigns them + to the Config instance's attributes. + + Args: + app_dict: A dictionary containing configuration data for the application. """ fields: List[str] = ["celery", "broker", "results_backend"] for field in fields: diff --git a/merlin/config/broker.py b/merlin/config/broker.py index 6145691a5..512fa337b 100644 --- a/merlin/config/broker.py +++ b/merlin/config/broker.py @@ -1,34 +1,18 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### - -"""Logic for configuring the celery broker.""" +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module provides utility functions and constants to manage broker configurations and connection strings +for various messaging systems, including RabbitMQ and Redis. It supports multiple connection protocols +and configurations, such as SSL, Unix sockets, and password inclusion. + +The module defines constants for supported brokers and connection string templates, along with functions +to construct and retrieve connection strings and SSL configurations based on settings defined in the +`app.yaml` configuration file. +""" from __future__ import print_function import getpass @@ -37,16 +21,11 @@ import ssl from os.path import expanduser from typing import Dict, List, Optional, Union +from urllib.parse import quote from merlin.config.configfile import CONFIG, get_ssl_entries -try: - from urllib import quote -except ImportError: - from urllib.parse import quote - - LOG: logging.Logger = logging.getLogger(__name__) BROKERS: List[str] = ["rabbitmq", "redis", "rediss", "redis+socket", "amqps", "amqp"] @@ -56,19 +35,41 @@ USER = getpass.getuser() -def read_file(filepath): - "Safe file read from filepath" +def read_file(filepath: str) -> str: + """ + Safely reads the first line from a file and returns it with special characters URL-encoded. + + Args: + filepath (str): The path to the file to be read. + + Returns: + The first line of the file, stripped of leading and trailing whitespace, + with special characters URL-encoded. + """ with open(filepath, "r") as f: # pylint: disable=C0103 line = f.readline().strip() return quote(line, safe="") -def get_rabbit_connection(include_password, conn="amqps"): +def get_rabbit_connection(include_password: bool, conn: str = "amqps") -> str: """ - Given the path to the directory where the broker configurations are stored - setup and return the RabbitMQ connection string. + Constructs and returns a RabbitMQ connection string based on broker configurations. - :param include_password : Format the connection for ouput by setting this True + This function reads broker configurations (such as server, port, username, password, and vhost) + and formats them into a RabbitMQ connection string. Optionally, the password can be included + in the connection string if `include_password` is set to `True`. + + Args: + include_password (bool): Whether to include the password in the connection string. + conn (str, optional): The connection protocol to use. Defaults to "amqps". + Supported values are "amqp" and "amqps". + + Returns: + A formatted RabbitMQ connection string. + + Raises: + ValueError: If the password file path is not provided in the broker configuration, or if + the password file does not exist or cannot be read. """ LOG.debug(f"Broker: connection = {conn}") @@ -83,7 +84,7 @@ def get_rabbit_connection(include_password, conn="amqps"): try: password_filepath = CONFIG.broker.password - LOG.debug(f"Broker: password filepath = {password_filepath}") + LOG.debug("Broker: password file path has been configured.") password_filepath = os.path.abspath(expanduser(password_filepath)) except (AttributeError, KeyError) as exc: raise ValueError("Broker: No password provided for RabbitMQ") from exc @@ -119,10 +120,17 @@ def get_rabbit_connection(include_password, conn="amqps"): return RABBITMQ_CONNECTION.format(**rabbitmq_config) -def get_redissock_connection(): +def get_redissock_connection() -> str: """ - Given the path to the directory where the broker configurations are stored - setup and return the redis+socket connection string. + Constructs and returns a Redis connection string using a Unix socket. + + This function retrieves broker configurations, such as the database number (`db_num`) and + the Unix socket file path (`path`), and formats them into a Redis connection string. + + If the database number is not specified in the configuration, it defaults to `0`. + + Returns: + A formatted Redis connection string using a Unix socket. """ try: db_num = CONFIG.broker.db_num @@ -137,12 +145,20 @@ def get_redissock_connection(): # flake8 complains this function is too complex, we don't gain much nesting any of this as a separate function, # however, cyclomatic complexity examination is off to get around this -def get_redis_connection(include_password, use_ssl=False): # noqa C901 +def get_redis_connection(include_password: bool, use_ssl: bool = False) -> str: # noqa C901 """ - Return the redis or rediss specific connection + Constructs and returns a Redis connection string, optionally using SSL and including a password. + + This function retrieves broker configurations (such as server, port, username, password, and database number) + and formats them into a Redis connection string. The connection can be configured to use SSL (`rediss` protocol) + and optionally include the password in the connection string. + + Args: + include_password (bool): Whether to include the password in the connection string. + use_ssl (bool, optional): Whether to use the `rediss` protocol (SSL). - :param include_password : Format the connection for ouput by setting this True - :param use_ssl : Flag to use rediss output + Returns: + A formatted Redis connection string. """ server = CONFIG.broker.server LOG.debug(f"Broker: server = {server}") @@ -184,15 +200,22 @@ def get_redis_connection(include_password, use_ssl=False): # noqa C901 return f"{urlbase}://{spass}{server}:{port}/{db_num}" -def get_connection_string(include_password=True): +def get_connection_string(include_password: bool = True) -> str: """ - Return the connection string based on the configuration specified in the - `app.yaml` config file. + Constructs and returns a connection string based on the broker configuration. - If the url variable is present, return that as the connection string. + This function retrieves the connection string from the `CONFIG.broker.url` if available. + Otherwise, it determines the connection string based on the broker name specified in the + configuration file (`app.yaml`). If the broker name is not supported, a `ValueError` is raised. - :param include_password : The connection can be formatted for output by - setting this to True + Args: + include_password (bool): Whether to include the password in the connection string. + + Returns: + A formatted connection string based on the broker configuration. + + Raises: + ValueError: If the broker name is not supported. """ try: return CONFIG.broker.url @@ -210,7 +233,22 @@ def get_connection_string(include_password=True): return _sort_valid_broker(broker, include_password) -def _sort_valid_broker(broker, include_password): +def _sort_valid_broker(broker: str, include_password: bool) -> str: + """ + Determines and returns the appropriate connection string for a given broker. + + This function selects the connection string generation method based on the broker type + provided as input. Supported brokers include RabbitMQ (`amqp` or `amqps`), Redis (`redis`), + Redis over SSL (`rediss`), and Redis over a socket (`redis+socket`). + + Args: + broker (str): The name of the broker. Must be one of the supported broker types: + `rabbitmq`, `amqps`, `amqp`, `redis+socket`, `redis`, or `rediss`. + include_password (bool): Whether to include the password in the connection string. + + Returns: + A formatted connection string for the specified broker. + """ if broker in ("rabbitmq", "amqps"): return get_rabbit_connection(include_password, conn="amqps") @@ -229,11 +267,17 @@ def _sort_valid_broker(broker, include_password): def get_ssl_config() -> Union[bool, Dict[str, Union[str, ssl.VerifyMode]]]: """ - Return the ssl config based on the configuration specified in the - `app.yaml` config file. + Retrieves the SSL configuration for the broker based on the settings in the `app.yaml` configuration file. + + This function determines whether SSL should be used for the broker connection and, if applicable, + returns the SSL configuration details. If the broker does not require SSL or is unsupported, + the function returns `False`. - :return: Returns either False if no ssl - :rtype: Union[bool, Dict[str, Union[str, ssl.VerifyMode]]] + Returns: + This returns either:\n + - `False` if SSL is not required or the broker is unsupported. + - A dictionary containing SSL configuration details if SSL is required. + The dictionary may include keys such as certificate paths and verification modes. """ broker: Union[bool, str] = "" try: diff --git a/merlin/config/celeryconfig.py b/merlin/config/celeryconfig.py index ff006c739..19254d747 100644 --- a/merlin/config/celeryconfig.py +++ b/merlin/config/celeryconfig.py @@ -1,58 +1,49 @@ -""" -Default celery configuration for merlin -""" +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +"""This module houses the default Celery configuration settings for Merlin.""" from merlin.log_formatter import FORMATS DICT = { + # -------- SERIALIZER SETTINGS -------- "task_serializer": "pickle", "accept_content": ["pickle"], "result_serializer": "pickle", + # ----------- TASK SETTINGS ----------- "task_acks_late": True, "task_reject_on_worker_lost": True, "task_publish_retry_policy": { "interval_start": 10, "interval_step": 10, - "interval_max": 60, + "interval_max": 300, }, - "redis_max_connections": 100000, + "task_default_queue": "merlin", + # ---------- BROKER SETTINGS ---------- "broker_transport_options": { "visibility_timeout": 60 * 60 * 24, "max_connections": 100, + "socket_timeout": 300, + "retry_policy": { + "timeout": 600, + }, }, + "broker_connection_timeout": 60, "broker_pool_limit": 0, - "task_default_queue": "merlin", + # --------- BACKEND SETTINGS ---------- + "result_backend_always_retry": True, + "result_backend_max_retries": 20, + # ---------- REDIS SETTINGS ----------- + "redis_max_connections": 100000, + "redis_retry_on_timeout": True, + "redis_socket_connect_timeout": 300, + "redis_socket_timeout": 300, + "redis_socket_keepalive": True, + # ---------- WORKER SETTINGS ---------- "worker_log_color": True, "worker_log_format": FORMATS["DEFAULT"], "worker_task_log_format": FORMATS["WORKER"], diff --git a/merlin/config/config_filepaths.py b/merlin/config/config_filepaths.py new file mode 100644 index 000000000..ae64f2dad --- /dev/null +++ b/merlin/config/config_filepaths.py @@ -0,0 +1,18 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module stores constants representing file paths that will be needed for +Merlin's configuration. +""" + +import os + + +APP_FILENAME: str = "app.yaml" +USER_HOME: str = os.path.expanduser("~") +MERLIN_HOME: str = os.path.join(USER_HOME, ".merlin") +CONFIG_PATH_FILE: str = os.path.join(MERLIN_HOME, "config_path.txt") diff --git a/merlin/config/configfile.py b/merlin/config/configfile.py index 010641a3f..ece97bbd0 100644 --- a/merlin/config/configfile.py +++ b/merlin/config/configfile.py @@ -1,36 +1,16 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ -This module handles the logic for the Merlin config files for setting up all -configurations. +This module provides functionality for managing and loading application configuration files, +default settings, and SSL-related configurations. It includes utilities for locating, +reading, and processing configuration files, as well as handling SSL certificates and protocols +for various server types. + +It houses the `CONFIG` object that's used throughout Merlin's codebase. """ import getpass import logging @@ -39,24 +19,48 @@ from typing import Dict, Optional, Union from merlin.config import Config +from merlin.config.config_filepaths import APP_FILENAME, CONFIG_PATH_FILE, MERLIN_HOME from merlin.utils import load_yaml LOG: logging.Logger = logging.getLogger(__name__) -APP_FILENAME: str = "app.yaml" CONFIG: Optional[Config] = None +IS_LOCAL_MODE: bool = False + + +def set_local_mode(enable: bool = True): + """ + Sets Merlin to run in local mode, which doesn't require a configuration file. + + Args: + enable (bool): True to enable local mode, False to disable it. + """ + global IS_LOCAL_MODE # pylint: disable=global-statement + IS_LOCAL_MODE = enable + if enable: + LOG.info("Running Merlin in local mode (no configuration file required)") + + +def is_local_mode() -> bool: + """ + Checks if Merlin is running in local mode. -USER_HOME: str = os.path.expanduser("~") -MERLIN_HOME: str = os.path.join(USER_HOME, ".merlin") + Returns: + True if running in local mode, False otherwise. + """ + return IS_LOCAL_MODE -def load_config(filepath): +def load_config(filepath: str) -> Dict: """ - Given the path to the merlin YAML config file, read the file and return - a dictionary of the contents. + Reads a Merlin YAML configuration file and returns its contents as a dictionary. - :param filepath : Read a yaml file given by filepath + Args: + filepath (str): The path to the YAML configuration file. + + Returns: + A dictionary containing the contents of the YAML file or None if file doesn't exist. """ if not os.path.isfile(filepath): LOG.info(f"No app config file at {filepath}") @@ -65,21 +69,44 @@ def load_config(filepath): return load_yaml(filepath) -def find_config_file(path=None): +def find_config_file(path: str = None) -> str: """ - Given a dir path, find and return the path to the merlin application - config file. + Locate the Merlin application configuration file (`app.yaml`). + + This function searches for the configuration file based on a given directory or, + if no directory is provided, uses a fallback sequence: + 1. Check for `app.yaml` in the current working directory. + 2. Check if `CONFIG_PATH_FILE` exists and points to a valid config file. + 3. Check for `app.yaml` in the `MERLIN_HOME` directory. - :param path : The path to search for the app.yaml file + If a `path` is explicitly provided, the function checks only that directory + for `app.yaml`. + + Args: + path (str, optional): A specific directory to look for `app.yaml`. + + Returns: + The full path to the `app.yaml` file if found, otherwise `None`. """ + # Fallback to default logic if path is None: + # Check current working directory for an active config path local_app = os.path.join(os.getcwd(), APP_FILENAME) - path_app = os.path.join(MERLIN_HOME, APP_FILENAME) - if os.path.isfile(local_app): return local_app + + # Check the config_path.txt file for the active config path + if os.path.isfile(CONFIG_PATH_FILE): + with open(CONFIG_PATH_FILE, "r") as f: + config_path = f.read().strip() + if os.path.isfile(config_path): + return config_path + + # Check the Merlin home directory for an active config path + path_app = os.path.join(MERLIN_HOME, APP_FILENAME) if os.path.isfile(path_app): return path_app + return None app_path = os.path.join(path, APP_FILENAME) @@ -89,15 +116,23 @@ def find_config_file(path=None): return None -def load_default_user_names(config): +def set_username_and_vhost(config: Dict): """ - Load broker.username and broker.vhost defaults if they are not present in - the current configuration. Doing this here prevents other areas that rely - on config from needing to know that those fields could not be defined by - the user. + Ensures that `broker.username` and `broker.vhost` default values are set in the + configuration if they are not already defined. - :param config : The namespace config object + This function checks the `config` object for the presence of `broker.username` + and `broker.vhost`. If either is missing, it sets their default values using + the current system username. This prevents other parts of the code from having + to handle missing values for these fields. + + Args: + config (Dict): The configuration object containing the `broker` namespace. """ + # Ensure broker key exists + if "broker" not in config: + config["broker"] = {} + try: config["broker"]["username"] except KeyError: @@ -110,29 +145,77 @@ def load_default_user_names(config): config["broker"]["vhost"] = vhost -def get_config(path: Optional[str]) -> Dict: +def get_default_config() -> Dict: """ - Load a merlin configuration file and return a dictionary of the - configurations. + Creates a minimal default configuration for local mode. - :param [Optional[str]] path : The path to search for the config file. - :return: the config file to coordinate brokers/results backend/task manager." - :rtype: A Dict with all the config data. + Returns: + Dict: A configuration dictionary with essential default values. """ - filepath: Optional[str] = find_config_file(path) + default_config = { + "broker": { + "username": "user", + "vhost": "vhost", + "server": "localhost", + "name": "rabbitmq", # Default broker name + "port": 5672, # Default RabbitMQ port + "protocol": "amqp", # Default protocol + }, + "celery": {"omit_queue_tag": False, "queue_tag": "[merlin]_", "override": None}, + "results_backend": { + "name": "sqlite", # Default results backend + "port": 1234, + }, + } + return default_config + + +def get_config(path: Optional[str] = None) -> Dict: + """ + Loads a Merlin configuration file and returns a dictionary containing the configuration data. + + This function locates the configuration file using the provided `path` or default search locations, + loads the configuration data, and applies default values where necessary. + + Args: + path (str, optional): The directory path to search for the configuration file. + If `None`, default search paths are used. + + Returns: + A dictionary containing all the configuration data, including broker, + results backend, and task manager settings. + + Raises: + ValueError: If the configuration file cannot be found and it's not a local run. + """ + if is_local_mode(): + LOG.info("Using default configuration (local mode)") + config = get_default_config() + load_defaults(config) + return config + filepath: Optional[str] = find_config_file(path) if filepath is None: raise ValueError( - f"Cannot find a merlin config file! Run 'merlin config' and edit the file '{MERLIN_HOME}/{APP_FILENAME}'" + "Cannot find a merlin config file! Run 'merlin config create' and edit the file " + f"'{os.path.join(MERLIN_HOME, APP_FILENAME)}'" ) - config: Dict = load_config(filepath) load_defaults(config) return config -def load_default_celery(config): - """Creates the celery default configuration""" +def load_default_celery(config: Dict): + """ + Initializes the default Celery configuration within the provided configuration object. + + This function ensures that the `celery` section of the configuration exists and sets + default values for specific Celery-related settings if they are not already defined. + These defaults include `omit_queue_tag`, `queue_tag`, and `override`. + + Args: + config (Dict): The configuration object where the Celery settings will be initialized. + """ try: config["celery"] except KeyError: @@ -151,23 +234,52 @@ def load_default_celery(config): config["celery"]["override"] = None -def load_defaults(config): - """Loads default configuration values""" - load_default_user_names(config) +def load_defaults(config: Dict): + """ + Loads default configuration values into the provided configuration dictionary. + + This function initializes default values for various configuration sections, + including user-related settings and Celery-specific settings, by calling + `set_username_and_vhost` and `load_default_celery`. + + Args: + config (Dict): The configuration dictionary to be updated with default values. + """ + set_username_and_vhost(config) load_default_celery(config) -def is_debug(): +def is_debug() -> bool: """ - Check for MERLIN_DEBUG in environment to set a debugging flag + Determines whether the application is running in debug mode. + + This function checks the environment variable `MERLIN_DEBUG`. If the variable + exists and its value is set to `1`, debug mode is enabled. + + Returns: + True if `MERLIN_DEBUG` is set to `1` in the environment, otherwise False. """ if "MERLIN_DEBUG" in os.environ and int(os.environ["MERLIN_DEBUG"]) == 1: return True return False -def default_config_info(): - """Return information about Merlin's default configurations.""" +def default_config_info() -> Dict: + """ + Returns information about Merlin's default configurations. + + This function gathers and returns key details about the current configuration + of the Merlin application, including the location of the configuration file, + debug mode status, the Merlin home directory, and whether the Merlin home + directory exists. + + Returns: + A dictionary containing the following keys:\n + - `config_file` (str): Path to the Merlin configuration file. + - `is_debug` (bool): Whether debug mode is enabled. + - `merlin_home` (str): Path to the Merlin home directory. + - `merlin_home_exists` (bool): True if the Merlin home directory exists, otherwise False. + """ return { "config_file": find_config_file(), "is_debug": is_debug(), @@ -176,14 +288,22 @@ def default_config_info(): } -def get_cert_file(server_type, config, cert_name, cert_path): +def get_cert_file(server_type: str, config: Config, cert_name: str, cert_path: str) -> str: """ - Check if a ssl certificate file is present in the config + Determines the SSL certificate file for a given server configuration. + + This function checks if an SSL certificate file is specified in the server configuration. + If the file does not exist, it attempts to locate the certificate in an optional + certificate path. If the certificate file cannot be found, an error is logged. + + Args: + server_type (str): The type of server (e.g., Broker, Results Backend) for logging purposes. + config (config.Config): The server configuration object containing certificate details. + cert_name (str): The name of the certificate attribute in the configuration. + cert_path (str): An optional directory path to search for the certificate file. - :param server_type : The server type for output (Broker, Results Backend) - :param config : The server config - :param cert_name : The argument in cert argument name - :param cert_path : The optional cert path + Returns: + The absolute path to the certificate file if found, otherwise `None`. """ cert_file = None try: @@ -208,14 +328,25 @@ def get_ssl_entries( server_type: str, server_name: str, server_config: Config, cert_path: str ) -> Dict[str, Union[str, ssl.VerifyMode]]: """ - Check if a ssl certificate file is present in the config - - :param [str] server_type : The server type - :param [str] server_name : The server name for output - :param [Config] server_config : The server config - :param [str] cert_path : The optional cert path - :return : The data needed to manage an ssl certification. - :rtype : A Dict. + Retrieves SSL configuration entries for a given server. + + This function checks for SSL certificate files and other SSL-related settings + in the server configuration. It builds and returns a dictionary containing + the necessary data to manage SSL certificates and protocols for the server. + + Args: + server_type (str): The type of server (e.g., Broker, Results Backend) for logging purposes. + server_name (str): The name of the server, used for output and mapping SSL configurations. + server_config (config.Config): The server configuration object containing SSL settings. + cert_path (str): An optional directory path to search for certificate files. + + Returns: + A dictionary containing SSL configuration entries, including:\n + - `keyfile` (str): Path to the SSL key file, if present. + - `certfile` (str): Path to the SSL certificate file, if present. + - `ca_certs` (str): Path to the CA certificates file, if present. + - `cert_reqs` (ssl.VerifyMode): SSL certificate requirements (e.g., `CERT_REQUIRED`, `CERT_OPTIONAL`, `CERT_NONE`). + - `ssl_protocol` (str): SSL protocol used, if specified. """ server_ssl: Dict[str, Union[str, ssl.VerifyMode]] = {} @@ -259,11 +390,22 @@ def get_ssl_entries( return server_ssl -def process_ssl_map(server_name: str) -> Optional[Dict[str, str]]: +def process_ssl_map(server_name: str) -> Dict[str, str]: """ - Process a special map for rediss and mysql. + Processes and returns a mapping of SSL-related configuration keys + specific to certain server types (e.g., Redis and MySQL). + + This function generates a dictionary mapping standard SSL configuration keys + (e.g., `keyfile`, `certfile`, `ca_certs`, `cert_reqs`) to server-specific key names + required by Redis (`rediss`) or MySQL server configurations. - :param server_name : The server name for output + Args: + server_name (str): The name of the server (e.g., "rediss", "mysql") used to determine + the appropriate SSL key mappings. + + Returns: + A dictionary containing the SSL key mappings for the given server type. Returns an empty + dictionary if the server type is not `rediss` or `mysql`. """ ssl_map: Dict[str, str] = {} # The redis server requires key names with ssl_ @@ -284,11 +426,20 @@ def process_ssl_map(server_name: str) -> Optional[Dict[str, str]]: def merge_sslmap(server_ssl: Dict[str, Union[str, ssl.VerifyMode]], ssl_map: Dict[str, str]) -> Dict: """ - The different servers have different key var expectations, this updates the keys of the ssl_server dict with keys from - the ssl_map if using rediss or mysql. + Updates the keys of the `server_ssl` dictionary based on the `ssl_map` for specific server types. + + This function modifies the `server_ssl` dictionary by replacing its keys with the corresponding + keys from the `ssl_map` when the server type requires specialized key names (e.g., `rediss` or `mysql`). + If a key in `server_ssl` is not found in `ssl_map`, it remains unchanged. - : param server_ssl : the dict constructed in get_ssl_entries, here updated with keys from ssl_map - : param ssl_map : the dict holding special key:value pairs for rediss and mysql + Args: + server_ssl (Dict[str, Union[str, ssl.VerifyMode]]): The dictionary constructed in `get_ssl_entries`, + containing SSL configuration entries such as `keyfile`, `certfile`, `ca_certs`, and `cert_reqs`. + ssl_map (Dict[str, str]): A dictionary mapping standard SSL keys to server-specific keys. + + Returns: + A new dictionary with updated keys based on the `ssl_map`. Keys not present in `ssl_map` + remain unchanged. """ new_server_ssl: Dict[str, Union[str, ssl.VerifyMode]] = {} @@ -301,5 +452,32 @@ def merge_sslmap(server_ssl: Dict[str, Union[str, ssl.VerifyMode]], ssl_map: Dic return new_server_ssl -app_config: Dict = get_config(None) -CONFIG = Config(app_config) +def initialize_config(path: Optional[str] = None, local_mode: bool = False) -> Config: + """ + Initializes and returns the Merlin configuration. + + This function can be used to explicitly initialize the configuration when needed, + rather than relying on the module-level CONFIG constant. + + Args: + path (Optional[str]): Path to look for configuration file + local_mode (bool): Whether to use local mode (no config file required) + + Returns: + The initialized configuration object + """ + if local_mode: + set_local_mode(True) + + global CONFIG # pylint: disable=global-statement + + try: + app_config = get_config(path) + CONFIG = Config(app_config) + except ValueError as e: + LOG.warning(f"Error loading configuration: {e}. Falling back to default configuration.") + # Fallback to default config + CONFIG = Config(get_default_config()) + + +initialize_config() diff --git a/merlin/config/merlin_config_manager.py b/merlin/config/merlin_config_manager.py new file mode 100644 index 000000000..9839b4021 --- /dev/null +++ b/merlin/config/merlin_config_manager.py @@ -0,0 +1,227 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module provides the `MerlinConfigManager` class, which is responsible for managing +Merlin configuration files. It allows users to initialize, create, and update configuration +files for the Merlin application, including broker and results backend settings. +""" + +import logging +import os +import shutil +from argparse import Namespace +from importlib import resources +from typing import Dict, List, Union + +import yaml + +from merlin.config.config_filepaths import CONFIG_PATH_FILE +from merlin.exceptions import MerlinInvalidTaskServerError + + +LOG = logging.getLogger(__name__) + + +class MerlinConfigManager: + """ + A class to manage the configuration of the Merlin application. + + This class provides functionality to initialize, create, and update configuration files + for the Merlin application. It supports updating broker and results backend settings + for Redis and RabbitMQ, and allows users to switch between multiple configurations + using a `config_path.txt` file. + + Attributes: + args (Namespace): Parsed command-line arguments. + output_dir (str): The directory where configuration files are stored. + config_file (str): The path to the current configuration file. + + Methods: + save_config_path: + Saves the path to the configuration file in `config_path.txt`. + + create_template_config: + Creates a template configuration file based on user input. + + update_broker: + Updates the broker section of the configuration file. + + update_backend: + Updates the results backend section of the configuration file. + + update_redis_config: + Updates Redis-specific configuration fields for broker or backend. + + update_rabbitmq_config: + Updates RabbitMQ-specific configuration fields for broker. + + update_config: + Generic method to update configuration fields based on user input. + """ + + def __init__(self, args: Namespace): + """ + Initialize the configuration manager. + + Args: + args: Parsed command-line arguments. + """ + self.args = args + self.config_file = getattr(args, "config_file", None) + if not self.config_file: # This should never be reached because of argparse defaults + raise ValueError("No config file given to MerlinConfigManager.") + + self.config_file = os.path.abspath(self.config_file) + LOG.debug(f"MerlinConfigManager successfully initialized with config file: {self.config_file}") + + def save_config_path(self): + """ + Save the path to the configuration file in `config_path.txt`. + """ + if not os.path.isfile(self.config_file): + raise FileNotFoundError(f"Cannot set config path. File does not exist: '{self.config_file}'") + + test_config_path = os.path.abspath(os.path.join(os.path.dirname(self.config_file), "config_path.txt")) + config_path_file = test_config_path if self.args.test else CONFIG_PATH_FILE + with open(config_path_file, "w") as f: + f.write(self.config_file) + LOG.info(f"Configuration path saved to '{config_path_file}'.") + + def create_template_config(self): + """ + Create a template configuration file. + """ + LOG.info("Creating config ...") + + if self.args.task_server != "celery": + raise MerlinInvalidTaskServerError("Only celery can be configured currently.") + + template_config = "app_redis.yaml" if self.args.broker == "redis" else "app.yaml" + + with resources.path("merlin.data.celery", template_config) as template_config_file: + self._create_config(template_config_file) + + def _create_config(self, template_config_file: str): + """ + Internal method to create the Celery configuration. + + Args: + template_config_file (str): Path to the template configuration file. + """ + # Create the configuration file if it doesn't already exist + if not os.path.isfile(self.config_file): + os.makedirs(os.path.dirname(self.config_file), exist_ok=True) + shutil.copy(template_config_file, self.config_file) + + # Check to make sure the copy worked + if not os.path.isfile(self.config_file): + LOG.error(f"Cannot create config file '{self.config_file}'.") + else: + LOG.info(f"The file '{self.config_file}' is ready to be edited for your system.") + # Log a message if the configuration file already exists + else: + LOG.info(f"The config file already exists, '{self.config_file}'.") + + from merlin.common.security import encrypt # pylint: disable=import-outside-toplevel + + encrypt.init_key() + + def update_broker(self): + """ + Update the broker section of the app.yaml file. + """ + LOG.info(f"Updating broker settings in '{self.config_file}'...") + + with open(self.config_file, "r") as app_yaml_file: + config = yaml.safe_load(app_yaml_file) + + broker_config = config.get("broker", {}) + if self.args.type == "redis": + self.update_redis_config(broker_config) + elif self.args.type == "rabbitmq": + self.update_rabbitmq_config(broker_config) + else: + LOG.error("Invalid broker type. Use 'redis' or 'rabbitmq'.") + return + + config["broker"] = broker_config + + with open(self.config_file, "w") as app_yaml_file: + yaml.dump(config, app_yaml_file, default_flow_style=False) + + LOG.info("Broker settings successfully updated. Check your new connection with `merlin info`.") + + def update_backend(self): + """ + Update the results backend section of the app.yaml file. + """ + LOG.info(f"Updating results backend settings in '{self.config_file}'...") + + with open(self.config_file, "r") as app_yaml_file: + config = yaml.safe_load(app_yaml_file) + + backend_config = config.get("results_backend", {}) + if self.args.type == "redis": + self.update_redis_config(backend_config) + else: + LOG.error("Invalid backend type. Use 'redis'.") + return + + config["results_backend"] = backend_config + + with open(self.config_file, "w") as app_yaml_file: + yaml.dump(config, app_yaml_file, default_flow_style=False) + + LOG.info("Results backend settings successfully updated. Check your new connection with `merlin info`.") + + def update_redis_config(self, config: Dict[str, Union[str, int]]): + """ + Update the Redis-specific configuration for either broker or backend. + + Args: + config: The configuration dictionary to update. + """ + LOG.warning("Redis does not use the 'username' or 'vhost' arguments. Ignoring these if provided.") + config["name"] = "rediss" + config["username"] = "" # Redis doesn't use a username + + required_fields = ["password_file", "server", "port", "db_num", "cert_reqs"] + self.update_config(config, required_fields) + + def update_rabbitmq_config(self, config: Dict[str, Union[str, int]]): + """ + Update the RabbitMQ-specific broker configuration. + + Args: + config: The configuration dictionary to update. + """ + LOG.warning("RabbitMQ does not use the 'db_num' argument.") + config["name"] = "rabbitmq" + + required_fields = ["username", "password_file", "server", "port", "vhost", "cert_reqs"] + self.update_config(config, required_fields) + + def update_config(self, config: Dict[str, Union[str, int]], required_fields: List[str]): + """ + Generic function to update configuration fields based on provided arguments. + + Args: + config: The configuration dictionary to update. + required_fields: List of required field names for the configuration. + """ + for field in required_fields: + value = getattr(self.args, field, None) + if value is not None: + if field == "password_file": + field = "password" + LOG.info("Updating password field.") + else: + try: + LOG.info(f"Updating {field} value from '{config[field]}' to '{value}'.") + except KeyError: # This should only be hit in the test suite + LOG.info(f"Adding new field '{field}' with value '{value}' to config.") + config[field] = value if field != "port" else int(value) diff --git a/merlin/config/results_backend.py b/merlin/config/results_backend.py index ae58f0340..61934c745 100644 --- a/merlin/config/results_backend.py +++ b/merlin/config/results_backend.py @@ -1,50 +1,26 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ -This module contains the logic for configuring the Celery results backend. +This module provides functionality for managing and configuring connection strings +and SSL settings for various results backends, including MySQL, Redis, Rediss, and SQLite. +The module relies on the application's configuration file (`app.yaml`) to determine backend +settings and certificate paths. """ from __future__ import print_function import logging import os +from typing import Dict +from urllib.parse import quote +from merlin.config.config_filepaths import MERLIN_HOME from merlin.config.configfile import CONFIG, get_ssl_entries -try: - from urllib import quote -except ImportError: - from urllib.parse import quote - - LOG = logging.getLogger(__name__) BACKENDS = ["sqlite", "mysql", "redis", "rediss", "none"] @@ -68,20 +44,30 @@ ) # fmt: on - -SQLITE_CONNECTION_STRING = "db+sqlite:///results.db" +SQLITE_CONNECTION_STRING = os.path.join(MERLIN_HOME, "merlin.db") -def get_backend_password(password_file, certs_path=None): +def get_backend_password(password_file: str, certs_path: str = None) -> str: """ - Check for password in file. - If the password is not found in the given password_file, - then the certs_path will be searched for the file, - if this file cannot be found, the password value will - be returned. - - :param password_file : The file path for the password - :param certs_path : The path for ssl certificates and passwords + Retrieves the backend password from a specified file or returns the provided password value. + + This function attempts to locate the password file in several locations: + + 1. The default Merlin directory (`~/.merlin`). + 2. The path specified by `password_file`. + 3. A directory specified by `certs_path` (if provided). + + If the password file is found, the password is read from the file. If the file cannot be + found, the value of `password_file` is treated as the password itself and returned. + + Args: + password_file (str): The file path or value for the password. If this is not a valid + file path, it is treated as the password itself. + certs_path (str, optional): An optional directory path where SSL certificates and + password files may be located. + + Returns: + The backend password, either retrieved from the file or the provided value. """ password = None @@ -104,8 +90,12 @@ def get_backend_password(password_file, certs_path=None): line = f.readline().strip() password = quote(line, safe="") - LOG.debug(f"Results backend: aux password path (certs_path) = {certs_path}") - LOG.debug(f"Results backend: password_filepath = {password_filepath}") + LOG.debug( + "Results backend: certs_path was provided and used in password resolution." + if certs_path + else "Results backend: certs_path was not provided." + ) + LOG.debug("Password resolution: using file." if password_filepath else "Password resolution: using direct value.") return password @@ -113,13 +103,19 @@ def get_backend_password(password_file, certs_path=None): # flake8 complains about cyclomatic complexity because of all the try-excepts, # this isn't so complicated it can't be followed and tucking things in functions # would make it less readable, so complexity evaluation is off -def get_redis(certs_path=None, include_password=True, ssl=False): # noqa C901 +def get_redis(certs_path: str = None, include_password: bool = True, ssl: bool = False) -> str: # noqa C901 """ - Return the redis or rediss specific connection + Constructs and returns a Redis or Rediss connection URL based on the provided parameters and configuration. - :param certs_path : The path for ssl certificates and passwords - :param include_password : Format the connection for ouput by setting this True - :param ssl : Flag to use rediss output + Args: + certs_path (str, optional): The path to SSL certificates and password files. + include_password (bool, optional): Whether to include the password in the connection URL. + If True, the password will be included; otherwise, it will be masked. + ssl (bool, optional): Flag indicating whether to use SSL for the connection (Rediss). + If True, the connection URL will use the "rediss" protocol; otherwise, it will use "redis". + + Returns: + A Redis or Rediss connection URL formatted based on the provided parameters and configuration. """ server = CONFIG.results_backend.server password_file = "" @@ -130,13 +126,13 @@ def get_redis(certs_path=None, include_password=True, ssl=False): # noqa C901 port = CONFIG.results_backend.port except (KeyError, AttributeError): port = 6379 - LOG.debug(f"Results backend: redis using default port = {port}") + LOG.debug("Results backend: using default Redis port.") try: db_num = CONFIG.results_backend.db_num except (KeyError, AttributeError): db_num = 0 - LOG.debug(f"Results backend: redis using default db_num = {db_num}") + LOG.debug("Results backend: using default Redis database number.") try: username = CONFIG.results_backend.username @@ -156,22 +152,29 @@ def get_redis(certs_path=None, include_password=True, ssl=False): # noqa C901 spass = f"{username}:******@" except (KeyError, AttributeError): spass = "" - LOG.debug(f"Results backend: redis using default password = {spass}") + LOG.debug("Results backend: no Redis password configured in backend config.") - LOG.debug(f"Results backend: password_file = {password_file}") - LOG.debug(f"Results backend: server = {server}") - LOG.debug(f"Results backend: certs_path = {certs_path}") + LOG.debug( + f"Results backend: {'password file specified in config' if password_file else 'no password file specified; using direct value'}." + ) + LOG.debug(f"Results backend: certs_path was {'provided' if certs_path else 'not provided'}.") + LOG.debug(f"Results backend: Redis server address {'configured' if server else 'not found in config'}.") return f"{urlbase}://{spass}{server}:{port}/{db_num}" -def get_mysql_config(certs_path, mysql_certs): +def get_mysql_config(certs_path: str, mysql_certs: Dict) -> Dict: """ - Determine if all the information for connecting MySQL as the Celery - results backend exists. + Determines whether all required information for connecting to MySQL as the Celery + results backend is available, and returns the MySQL SSL configuration or certificate paths. + + Args: + certs_path (str): The path to the directory containing SSL certificates and password files. + mysql_certs (Dict): A dictionary mapping certificate keys (e.g., 'cert', 'key', 'ca') + to their expected filenames. - :param certs_path : The path for ssl certificates and passwords - :param mysql_certs : The dict of mysql certificates + Returns: + A dictionary containing the paths to the required MySQL certificates if they exist. """ mysql_ssl = get_ssl_config(celery_check=False) if mysql_ssl: @@ -192,13 +195,25 @@ def get_mysql_config(certs_path, mysql_certs): return certs -def get_mysql(certs_path=None, mysql_certs=None, include_password=True): +def get_mysql(certs_path: str = None, mysql_certs: Dict = None, include_password: bool = True) -> str: """ - Returns the formatted MySQL connection string. - - :param certs_path : The path for ssl certificates and passwords - :param mysql_certs : The dict of mysql certificates - :param include_password : Format the connection for ouput by setting this True + Constructs and returns a formatted MySQL connection string based on the provided parameters + and application configuration. + + Args: + certs_path (str, optional): The path to the directory containing SSL certificates and password files. + mysql_certs (dict, optional): A dictionary mapping MySQL certificate keys (e.g., 'ssl_key', 'ssl_cert', 'ssl_ca') + to their expected filenames. If this is None, it uses the default `MYSQL_CONFIG_FILENAMES`. + include_password (bool, optional): Whether to include the password in the connection string. + If True, the password will be included; otherwise, it will be masked. + + Returns: + A formatted MySQL connection string. + + Raises: + TypeError: \n + - If the `server` configuration is missing or invalid. + - If the MySQL connection information cannot be set due to missing certificates or configuration. """ dbname = CONFIG.results_backend.dbname password_file = CONFIG.results_backend.password @@ -208,10 +223,12 @@ def get_mysql(certs_path=None, mysql_certs=None, include_password=True): # eventually be configured to use a logger. This logic should also # eventually be decoupled so we can print debug messages similar to our # Python debugging messages. - LOG.debug(f"Results backend: dbname = {dbname}") - LOG.debug(f"Results backend: password_file = {password_file}") - LOG.debug(f"Results backend: server = {server}") - LOG.debug(f"Results backend: certs_path = {certs_path}") + LOG.debug(f"Results backend: database name is {'configured' if dbname else 'missing'}.") + LOG.debug( + f"Results backend: password file {'specified in configuration' if password_file else 'not specified in configuration; using direct value'}." + ) + LOG.debug(f"Results backend: server address is {'configured' if server else 'missing'}.") + LOG.debug(f"Results backend: certs_path was {'provided' if certs_path else 'not provided'}.") if not server: msg = f"Results backend: server {server} does not have a configuration" @@ -245,15 +262,23 @@ def get_mysql(certs_path=None, mysql_certs=None, include_password=True): return MYSQL_CONNECTION_STRING.format(**mysql_config) -def get_connection_string(include_password=True): +def get_connection_string(include_password: bool = True) -> str: """ - Given the package configuration determine what results backend to use and - return the connection string. + Determines the appropriate results backend to use based on the package configuration + and returns the corresponding connection string. + + If a URL is explicitly defined in the configuration (`CONFIG.results_backend.url`), + it is returned as the connection string. + + Args: + include_password (bool, optional): Whether to include the password in the connection string. + If True, the password will be included; otherwise, it will be masked. - If the url variable is present, return that as the connection string. + Returns: + The connection string for the configured results backend. - :param config_path : The path for ssl certificates and passwords - :param include_password : Format the connection for ouput by setting this True + Raises: + ValueError: If the specified results backend in the configuration is not supported. """ try: return CONFIG.results_backend.url @@ -278,7 +303,23 @@ def get_connection_string(include_password=True): return _resolve_backend_string(backend, certs_path, include_password) -def _resolve_backend_string(backend, certs_path, include_password): +def _resolve_backend_string(backend: str, certs_path: str, include_password: bool) -> str: + """ + Resolves and returns the connection string for the specified results backend. + + Based on the backend type provided, this function delegates the connection string + generation to the appropriate helper function or returns a predefined connection string. + + Args: + backend (str): The name of the results backend (e.g., "mysql", "sqlite", "redis", "rediss"). + certs_path (str): The path to SSL certificates and password files, used for certain backends + (e.g., MySQL and Redis). + include_password (bool): Whether to include the password in the connection string. + If True, the password will be included; otherwise, it will be masked. + + Returns: + The connection string for the specified backend, or `None` if the backend is unsupported. + """ if "mysql" in backend: return get_mysql(certs_path=certs_path, include_password=include_password) @@ -294,12 +335,22 @@ def _resolve_backend_string(backend, certs_path, include_password): return None -def get_ssl_config(celery_check=False): +def get_ssl_config(celery_check: bool = False) -> bool: """ - Return the ssl config based on the configuration specified in the - `app.yaml` config file. + Retrieves the SSL configuration for the results backend based on the settings + specified in the `app.yaml` configuration file. + + This function determines whether SSL should be enabled for the results backend + and returns the appropriate configuration. It supports various backend types + such as MySQL, Redis, and Rediss. + + Args: + celery_check (bool, optional): If True, the function returns the SSL settings + specifically for configuring Celery. - :param celery_check : Return the proper results ssl setting when configuring celery + Returns: + The SSL configuration for the results backend. Returns `True` if SSL is enabled, + `False` otherwise. """ results_backend = "" try: diff --git a/merlin/config/utils.py b/merlin/config/utils.py index 090398187..61c39c225 100644 --- a/merlin/config/utils.py +++ b/merlin/config/utils.py @@ -1,33 +1,15 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### -"""This module contains priority handling""" +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module provides utility functions and classes for handling broker priorities +and determining configurations for supported brokers such as RabbitMQ and Redis. +It includes functionality for mapping priority levels to integer values based on +the broker type and validating broker configurations. +""" import enum from typing import Dict @@ -36,7 +18,18 @@ class Priority(enum.Enum): - """Enumerated Priorities""" + """ + Enumerated Priorities. + + This enumeration defines the different priority levels that can be used + for message handling with brokers. + + Attributes: + HIGH (int): Represents the highest priority level. Numeric value: 1. + MID (int): Represents the medium priority level. Numeric value: 2. + LOW (int): Represents the lowest priority level. Numeric value: 3. + RETRY (int): Represents the priority level for retrying messages. Numeric value: 4. + """ HIGH = 1 MID = 2 @@ -44,22 +37,54 @@ class Priority(enum.Enum): RETRY = 4 -def is_rabbit_broker(broker: str) -> bool: - """Check if the broker is a rabbit server""" - return broker in ["rabbitmq", "amqps", "amqp"] +def is_rabbit_broker(broker_name: str) -> bool: + """ + Check if the given broker is a RabbitMQ server. + + This function checks whether the provided broker name matches any of the + RabbitMQ-related broker types. + + Args: + broker_name: The name of the broker to check. + + Returns: + True if the broker is a RabbitMQ server, False otherwise. + """ + return broker_name in ["rabbitmq", "amqps", "amqp"] + + +def is_redis_broker(broker_name: str) -> bool: + """ + Check if the given broker is a Redis server. + This function checks whether the provided broker name matches any of the + Redis-related broker types. -def is_redis_broker(broker: str) -> bool: - """Check if the broker is a redis server""" - return broker in ["redis", "rediss", "redis+socket"] + Args: + broker_name: The name of the broker to check. + + Returns: + True if the broker is a Redis server, False otherwise. + """ + return broker_name in ["redis", "rediss", "redis+socket"] def determine_priority_map(broker_name: str) -> Dict[Priority, int]: """ - Returns the priority mapping for the given broker name. + Determine the priority mapping for the given broker name. + + This function returns a mapping of [`Priority`][config.utils.Priority] + enum values to integer priority levels based on the type of broker provided. - :param broker_name: The name of the broker that we need the priority map for - :returns: The priority map associated with `broker_name` + Args: + broker_name: The name of the broker for which to determine the priority map. + + Returns: + (Dict[config.utils.Priority, int]): A dictionary mapping + [`Priority`][config.utils.Priority] enum values to integer levels. + + Raises: + ValueError: If the broker name is not supported. """ if is_rabbit_broker(broker_name): return {Priority.LOW: 1, Priority.MID: 5, Priority.HIGH: 9, Priority.RETRY: 10} @@ -71,11 +96,24 @@ def determine_priority_map(broker_name: str) -> Dict[Priority, int]: def get_priority(priority: Priority) -> int: """ - Gets the priority level as an integer based on the broker. - For a rabbit broker a low priority is 1 and high is 10. For redis it's the opposite. + Get the integer priority level for a given [`Priority`][config.utils.Priority] + enum value. + + This function determines the priority level as an integer based on the + broker configuration. For RabbitMQ brokers, lower numbers represent lower + priorities, while for Redis brokers, higher numbers represent lower + priorities. + + Args: + priority (config.utils.Priority): The [`Priority`][config.utils.Priority] + enum value for which to get the integer level. + + Returns: + The integer priority level corresponding to the given [`Priority`][config.utils.Priority]. - :param priority: The priority value that we want - :returns: The priority value as an integer + Raises: + ValueError: If the provided `priority` is invalid or not part of the + [`Priority`][config.utils.Priority] enum. """ priority_err_msg = f"Invalid priority: {priority}" try: diff --git a/merlin/data/celery/__init__.py b/merlin/data/celery/__init__.py index 37cabcad1..3232b50b9 100644 --- a/merlin/data/celery/__init__.py +++ b/merlin/data/celery/__init__.py @@ -1,29 +1,5 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## diff --git a/merlin/data/celery/app.yaml b/merlin/data/celery/app.yaml index 4dd05515b..466294cf5 100644 --- a/merlin/data/celery/app.yaml +++ b/merlin/data/celery/app.yaml @@ -1,40 +1,39 @@ -celery: - # see Celery configuration options - # https://docs.celeryproject.org/en/stable/userguide/configuration.html - override: - visibility_timeout: 86400 - broker: - # can be redis, redis+sock, or rabbitmq name: rabbitmq - #username: # defaults to your username unless changed here - password: ~/.merlin/jackalope-password - # server URL - server: jackalope.llnl.gov - - ### for rabbitmq, redis+sock connections ### - #vhost: # defaults to your username unless changed here - - ### for redis+sock connections ### - #socketname: the socket name your redis connection can be found on. - #path: The path to the socket. - - ### for redis connections ### - #port: The port number redis is listening on (default 6379) - #db_num: The data base number to connect to. - + # This is the value stored in rabbitmq-user + username: + # 1. Create a rabbit.pass file + # 2. Copy the value of rabbitmq-password to it + # 3. Put the path to that file here + # e.g. password: ~/.merlin/rabbit.pass + password: + # This is the value stored in service-host + server: + # This is the value stored in service-port + port: + # This is the value stored in rabbitmq-host + vhost: + # Leave this as none + cert_reqs: none results_backend: - # must be redis - name: redis - dbname: mlsi - username: mlsi - # name of file where redis password is stored. - password: redis.pass - server: jackalope.llnl.gov - # merlin will generate this key if it does not exist yet, - # and will use it to encrypt all data over the wire to - # your redis server. + name: rediss + # The username here must remain blank + username: '' + # 1. Create a redis.pass file + # 2. Copy the value of database-password to it + # 3. Put the path to that file here + # e.g. password: ~/.merlin/redis.pass + password: + # This is the value stored in service-host + server: + # This should be automatically generated for you with merlin config encryption_key: ~/.merlin/encrypt_data_key - port: 6379 - db_num: 0 \ No newline at end of file + # This is the value stored in service-port + port: + db_num: 0 + cert_reqs: none + +celery: + overrides: + \ No newline at end of file diff --git a/merlin/data/celery/app_redis.yaml b/merlin/data/celery/app_redis.yaml index d65666b8b..bfdc73a14 100644 --- a/merlin/data/celery/app_redis.yaml +++ b/merlin/data/celery/app_redis.yaml @@ -1,44 +1,40 @@ -celery: - # see Celery configuration options - # https://docs.celeryproject.org/en/stable/userguide/configuration.html - override: - visibility_timeout: 86400 - broker: - # can be redis, redis+sock, or rabbitmq - name: redis - - # username: # defaults to your username unless changed here - # password: ~/.merlin/jackalope-password - - # server URL - server: localhost - - ### for rabbitmq, redis+sock connections ### - #vhost: # defaults to your username unless changed here - - ### for redis+sock connections ### - #socketname: the socket name your redis connection can be found on. - #path: The path to the socket. - - ### for redis connections ### - port: 6379 + name: rediss + # The username here must remain blank + username: '' + # 1. Create a redis.pass file + # 2. Copy the value of database-password to it + # 3. Put the path to that file here     + # Note: this must be an absolute path + # e.g. password: /.merlin/redis.pass + password: + # This is the value stored in service-host + server: + # This is the value stored in service-port + port: db_num: 0 + cert_reqs: none - +# This is the exact same setup as broker but we add the encryption key file results_backend: - # must be redis - name: redis - # dbname: mlsi - # username: mlsi - # name of file where redis password is stored. - # password: redis.pass - server: localhost - # merlin will generate this key if it does not exist yet, - # and will use it to encrypt all data over the wire to - # your redis server. - # encryption_key: ~/.merlin/encrypt_data_key - port: 6379 + name: rediss + # The username here must remain blank + username: '' + # 1. Create a redis.pass file + # 2. Copy the value of database-password to it + # 3. Put the path to that file here + # Note: this must be an absolute path + # e.g. password: /.merlin/redis.pass + password: + # This is the value stored in service-host + server: + # This should be automatically generated for you with merlin config + encryption_key: ~/.merlin/encrypt_data_key + # This is the value stored in service-port + port: db_num: 0 + cert_reqs: none - +celery: + overrides: + \ No newline at end of file diff --git a/merlin/data/celery/app_test.yaml b/merlin/data/celery/app_test.yaml deleted file mode 100644 index 798a863b1..000000000 --- a/merlin/data/celery/app_test.yaml +++ /dev/null @@ -1,44 +0,0 @@ -celery: - # see Celery configuration options - # https://docs.celeryproject.org/en/stable/userguide/configuration.html - override: - visibility_timeout: 86400 - -broker: - # can be redis, redis+sock, or rabbitmq - name: rabbitmq - #username: # defaults to your username unless changed here - username: $(RABBITMQ_USER) - # password: - password: $(RABBITMQ_PASS) - # server URL - server: $(RABBITMQ_PORT) - - - - ### for rabbitmq, redis+sock connections ### - #vhost: # defaults to your username unless changed here - - ### for redis+sock connections ### - #socketname: the socket name your redis connection can be found on. - #path: The path to the socket. - - ### for redis connections ### - #port: The port number redis is listening on (default 6379) - #db_num: The data base number to connect to. - - -results_backend: - # must be redis - name: redis - dbname: mlsi - username: mlsi - # name of file where redis password is stored. - password: redis.pass - server: jackalope.llnl.gov - # merlin will generate this key if it does not exist yet, - # and will use it to encrypt all data over the wire to - # your redis server. - encryption_key: ~/.merlin/encrypt_data_key - port: 6379 - db_num: 0 \ No newline at end of file diff --git a/merlin/db_scripts/__init__.py b/merlin/db_scripts/__init__.py new file mode 100644 index 000000000..c03bdfb41 --- /dev/null +++ b/merlin/db_scripts/__init__.py @@ -0,0 +1,30 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +The `db_scripts` package provides the core database infrastructure for the Merlin system. + +This package defines the models, entities, and management logic required to persist, retrieve, +and manipulate data stored in Merlin's database. It encapsulates both low-level data models and +high-level entity managers, offering a structured, maintainable interface for all database +interactions within the system. + +Subpackages: + - `entities/`: Contains entity classes that wrap database models and expose higher-level + operations such as `save`, `delete`, and `reload`. + - `entity_managers/`: Provides manager classes that orchestrate CRUD operations across + entity types, with support for inter-entity references and cleanup routines. + +Modules: + data_models.py: Defines the dataclasses used to represent raw records in the database, + such as [`StudyModel`][db_scripts.data_models.StudyModel], [`RunModel`][db_scripts.data_models.RunModel], + etc. + db_commands.py: Exposes database-related commands intended for external use (e.g., CLI or scripts), + allowing users to interact with Merlin's stored data. + merlin_db.py: Contains the [`MerlinDatabase`][db_scripts.merlin_db.MerlinDatabase] class, which + aggregates all entity managers and acts as the central access point for database operations + across the system. +""" diff --git a/merlin/db_scripts/data_models.py b/merlin/db_scripts/data_models.py new file mode 100644 index 000000000..88aa9cbdc --- /dev/null +++ b/merlin/db_scripts/data_models.py @@ -0,0 +1,456 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module houses dataclasses that define the format of the data +that's stored in Merlin's database. +""" + +import hashlib +import json +import logging +import os +import uuid +from abc import ABC, abstractmethod +from dataclasses import Field, asdict, dataclass, field +from dataclasses import fields as dataclass_fields +from datetime import datetime +from typing import Dict, List, Set, Tuple, Type, TypeVar + +from filelock import FileLock + +from merlin.common.enums import WorkerStatus + + +LOG = logging.getLogger("merlin") +T = TypeVar("T", bound="BaseDataModel") + + +@dataclass +class BaseDataModel(ABC): + """ + A base class for dataclasses that provides common serialization, deserialization, and + update functionality, with support for additional data. + + This class is designed to be extended by other dataclasses and includes methods for + converting instances to and from dictionaries or JSON, managing fields, and updating + field values with validation. + + Attributes: + additional_data: A dictionary to store any extra data not explicitly defined + as fields in the dataclass. + fields_allowed_to_be_updated: A list of field names that are allowed to be updated. + Must be defined in subclasses. + + Methods: + to_dict: + Convert the dataclass instance to a dictionary. + + to_json: + Serialize the dataclass instance to a JSON string. + + from_dict (classmethod): + Create an instance of the dataclass from a dictionary. + + from_json (classmethod): + Create an instance of the dataclass from a JSON string. + + dump_to_json_file: + Dump the data of this dataclass to a JSON file. + + load_from_json_file (classmethod): + Load the data stored in a JSON file to this dataclass. + + fields: + Retrieve the fields associated with this dataclass instance or class. + + fields (classmethod): + Retrieve the fields associated with the dataclass class itself. + + update_fields: + Update the fields of the dataclass based on a given dictionary of updates. + """ + + additional_data: Dict = field(default_factory=dict) + + def to_dict(self) -> Dict: + """ + Convert the dataclass to a dictionary. + + Returns: + The dataclass as a dictionary. + """ + return asdict(self) + + def to_json(self) -> str: + """ + Serialize the dataclass to a JSON string. + + Returns: + The dataclass as a JSON string. + """ + return json.dumps(self.to_dict()) + + @classmethod + def from_dict(cls: Type[T], data: Dict) -> T: + """ + Create an instance of the dataclass from a dictionary. + + Args: + data: A dictionary to turn into an instance of this dataclass. + + Returns: + An instance of the dataclass that called this. + """ + return cls(**data) + + @classmethod + def from_json(cls: Type[T], json_str: str) -> T: + """ + Create an instance of the dataclass from a JSON string. + + Args: + json_str: A JSON string to turn into an instance of this dataclass. + + Returns: + An instance of the dataclass that called this. + """ + data = json.loads(json_str) + return cls.from_dict(data) + + def dump_to_json_file(self, filepath: str): + """ + Dump the data of this dataclass to a JSON file. + + Args: + filepath: The path to the JSON file where the data will be written. + + Raises: + ValueError: If the `filepath` is not provided or is invalid. + """ + if not filepath: + raise ValueError("A valid file path must be provided.") + + # Ensure the directory for the file exists + os.makedirs(os.path.dirname(filepath), exist_ok=True) + + # Create a lock file alongside the target JSON file + lock_file = f"{filepath}.lock" + with FileLock(lock_file): # pylint: disable=abstract-class-instantiated + # Write the data to the JSON file + temp_filepath = f"{filepath}.tmp" # Use a temporary file for atomic writes + with open(temp_filepath, "w") as json_file: + json.dump(self.to_dict(), json_file, indent=4) + + # Replace the temporary file with the target file + os.replace(temp_filepath, filepath) + + LOG.debug(f"Data successfully dumped to {filepath}.") + + @classmethod + def load_from_json_file(cls: Type[T], filepath: str) -> T: + """ + Load the data stored in a JSON file to this dataclass. + + Args: + filepath: The path to the JSON file where the data is located. + + Raises: + ValueError: If the `filepath` is not provided or is invalid. + """ + if not filepath or not os.path.exists(filepath): + raise ValueError("A valid file path must be provided.") + + # Create a lock file alongside the target JSON file + lock_file = f"{filepath}.lock" + with FileLock(lock_file): # pylint: disable=abstract-class-instantiated + with open(filepath, "r") as json_file: + # Parse the JSON data into a dictionary + data = json.load(json_file) + + # Use from_dict to create an instance of the dataclass + return cls.from_dict(data) + + def get_instance_fields(self) -> Tuple[Field]: + """ + Get the fields associated with this instance. Added this method so that the dataclass.fields + doesn't have to be imported each time you want this info. + + Returns: + A tuple of dataclass.Field objects representing the fields in this data class. + """ + return dataclass_fields(self) + + @classmethod + def get_class_fields(cls) -> Tuple[Field]: + """ + Get the fields associated with this object. Added this method so that the dataclass.fields + doesn't have to be imported each time you want this info. + + Returns: + A tuple of dataclass.Field objects representing the fields in this data class. + """ + return dataclass_fields(cls) + + @property + @abstractmethod + def fields_allowed_to_be_updated(self) -> List[str]: + """ + A property to be overridden in subclasses to define which fields are allowed to be updated. + + Returns: + A list of fields that are allowed to be updated in this class. + """ + + def update_fields(self, updates: Dict): + """ + Given a dictionary of updates to be made to this data class, loop through the updates + applying them when valid. + + Args: + updates: A dictionary of updates to be made to this data class. + """ + # Iterate through the updates + for field_name, new_value in updates.items(): + if field_name == "id": + continue + + if hasattr(self, field_name): + if getattr(self, field_name) == new_value: # Not an update so skip + continue + + if field_name in self.fields_allowed_to_be_updated: + # Update the allowed field + setattr(self, field_name, new_value) + else: + # Log a warning for unauthorized updates + LOG.warning(f"Field '{field_name}' is not allowed to be updated. Ignoring the change.") + else: + # Log a warning if the field doesn't exist explicitly + LOG.warning( + f"Field '{field_name}' does not explicitly exist in the object. Adding it to the 'additional_data' field." + ) + self.additional_data[field_name] = new_value + + +@dataclass +class StudyModel(BaseDataModel): + """ + A dataclass to store all of the information for a study. + + Attributes: + additional_data (Dict): For any extra data not explicitly defined. + fields_allowed_to_be_updated (List[str]): A list of field names that are + allowed to be updated. + id (str): The unique ID for the study. + name (str): The name of the study. + runs (List[str]): A list of runs associated with this study. + """ + + id: str = field(default_factory=lambda: str(uuid.uuid4())) # pylint: disable=invalid-name + name: str = None + runs: List[str] = field(default_factory=list) + + @property + def fields_allowed_to_be_updated(self) -> List[str]: + """ + Define the fields that are allowed to be updated for a `StudyModel` object. + + Returns: + A list of fields that are allowed to be updated in this class. + """ + return ["runs"] + + +@dataclass +class RunModel(BaseDataModel): # pylint: disable=too-many-instance-attributes + """ + A dataclass to store all of the information for a run. + + Attributes: + additional_data (Dict): For any extra data not explicitly defined. + child (str): The ID of the child run (if any). + fields_allowed_to_be_updated (List[str]): A list of field names that are allowed + to be updated. + id (str): The unique ID for the run. + parameters (Dict): The parameters used in this run. + parent (str): The ID of the parent run (if any). + queues (List[str]): The task queues used for this run. + run_complete (bool): Wether the run is complete. + samples (Dict): The samples used in this run. + steps (List[str]): A list of unique step IDs that are executed in this run. + Each ID will correspond to a `StepInfo` entry. + study_id (str): The unique ID of the study this run is associated with. + Corresponds with a `StudyModel` entry. + workers (List[str]): A list of worker ids executing tasks for this run. Each ID + will correspond with a `LogicalWorkerModel` entry. + workspace (str): The path to the output workspace. + """ + + id: str = field(default_factory=lambda: str(uuid.uuid4())) # pylint: disable=invalid-name + study_id: str = None + workspace: str = None + steps: List[str] = field(default_factory=list) # TODO NOT YET IMPLEMENTED + queues: List[str] = field(default_factory=list) + workers: List[str] = field(default_factory=list) + parent: str = None # TODO NOT YET IMPLEMENTED; do we even have a good way that this and `child` can be set? + child: str = None # TODO NOT YET IMPLEMENTED + run_complete: bool = False + parameters: Dict = field(default_factory=dict) # TODO NOT YET IMPLEMENTED + samples: Dict = field(default_factory=dict) # TODO NOT YET IMPLEMENTED + + @property + def fields_allowed_to_be_updated(self) -> List[str]: + """ + Define the fields that are allowed to be updated for a `RunModel` object. + + Returns: + A list of fields that are allowed to be updated in this class. + """ + return ["parent", "child", "run_complete", "additional_data", "workers"] + + +@dataclass +class LogicalWorkerModel(BaseDataModel): + """ + Represents a high-level definition of a Celery worker, as defined by the user. + + Logical workers are abstract representations of workers that define their behavior + and configuration, such as the queues they listen to and their name. They are unique + based on their name and queues, and do not correspond directly to any running process. + Instead, they serve as templates or logical definitions from which physical workers + are created. + + Note: + Logical workers are abstract and do not represent actual running processes. They are + used to define worker behavior and configuration at a high level, while physical workers + represent the actual running instances of these logical definitions. + + Attributes: + additional_data (Dict): For any extra data not explicitly defined. + fields_allowed_to_be_updated (List[str]): A list of field names that are + allowed to be updated. + id (str): A unique identifier for the logical worker. Defaults to a UUID string. + name (str): The name of the logical worker. + physical_workers (List[str]): A list of unique IDs of the physical worker instances + created from this logical instance. Corresponds with + [`PhyiscalWorkerModel`][db_scripts.data_models.PhysicalWorkerModel] entries. + queues (List[str]): A list of task queues the worker is listening to. + runs (List[str]): A list of unique IDs of the runs using this worker. + Corresponds with [`RunModel`][db_scripts.data_models.RunModel] entries. + """ + + name: str = None + queues: Set[str] = field(default_factory=set) + id: str = None # pylint: disable=invalid-name + runs: List[str] = field(default_factory=list) + physical_workers: List[str] = field(default_factory=list) + + def __post_init__(self): + """ + Generate and save a UUID based on the values of `name` and `queues`, to help ensure that + each logical worker is unique to these values. + + Raises: + TypeError: When name or queues are not provided to the constructor. + """ + if self.name is None or not self.queues: + raise TypeError("The `name` and `queues` arguments of LogicalWorkerModel are required.") + + generated_id = self.generate_id(self.name, self.queues) + if self.id != generated_id: + if self.id is not None: + LOG.warning(f"ID '{self.id}' for LogicalWorkerModel was provided but it will be overwritten.") + self.id = generated_id + + @classmethod + def generate_id(cls, name: str, queues: List[str]) -> uuid.UUID: + """ + Generate a UUID based on the values of `name` and `queues`. + + Args: + name: The name of the logical worker. + queues: The queues that the logical worker is assigned. + + Returns: + A UUID based on the values of `name` and `queues`. + """ + unique_string = f"{name}:{','.join(sorted(queues))}" + hex_string = hashlib.md5(unique_string.encode("UTF-8")).hexdigest() + return str(uuid.UUID(hex=hex_string)) + + @property + def fields_allowed_to_be_updated(self) -> List[str]: + """ + Define the fields that are allowed to be updated for a `LogicalWorkerModel` object. + + Returns: + A list of fields that are allowed to be updated in this class. + """ + return ["runs", "physical_workers"] + + +@dataclass +class PhysicalWorkerModel(BaseDataModel): # pylint: disable=too-many-instance-attributes + """ + Represents a running instance of a Celery worker, created from a logical worker definition. + + Physical workers are the actual implementations of logical workers, running as processes on a host machine. + They are responsible for executing tasks defined in the queues specified by their corresponding logical worker. + Each physical worker is uniquely identified and includes runtime-specific details such as its PID, status, and + heartbeat timestamp. + + Attributes: + additional_data (Dict): For any extra data not explicitly defined. + args (Dict): A dictionary of arguments used to configure the worker. + fields_allowed_to_be_updated (List[str]): A list of field names that are + allowed to be updated. + heartbeat_timestamp (datetime): The last time the worker sent a heartbeat signal. + host (str): The hostname or IP address of the machine running the worker. + id (str): A unique identifier for the physical worker. Defaults to a UUID string. + latest_start_time (datetime): The timestamp when the worker process was last started. + launch_cmd (str): The command used to launch the worker process. + logical_worker_id (str): The ID of the logical worker that this was created from. + name (str): The name of the physical worker. + pid (str): The process ID (PID) of the worker process. + restart_count (int): The number of times this worker has been restarted. + status (WorkerStatus): The current status of the worker (e.g., running, stopped). + """ + + id: str = field(default_factory=lambda: str(uuid.uuid4())) # pylint: disable=invalid-name + logical_worker_id: str = None + name: str = None # Will be of the form celery@worker_name.hostname + launch_cmd: str = None + args: Dict = field(default_factory=dict) + pid: str = None + status: WorkerStatus = WorkerStatus.STOPPED + heartbeat_timestamp: datetime = field(default_factory=datetime.now) + latest_start_time: datetime = field(default_factory=datetime.now) + host: str = None + restart_count: int = 0 + + @property + def fields_allowed_to_be_updated(self) -> List[str]: + """ + Define the fields that are allowed to be updated for a `PhysicalWorkerModel` object. + + Returns: + A list of fields that are allowed to be updated in this class. + """ + return [ + "launch_cmd", + "args", + "pid", + "status", + "heartbeat_timestamp", + "latest_start_time", + "restart_count", + ] + + +# TODO create a StepInfo class to store information about a step +# - Can probably link this to status +# - Each step should have entries for parameters/samples but only those that are actually used in the step diff --git a/merlin/db_scripts/db_commands.py b/merlin/db_scripts/db_commands.py new file mode 100644 index 000000000..45231325e --- /dev/null +++ b/merlin/db_scripts/db_commands.py @@ -0,0 +1,171 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module acts as an interface for users to interact with Merlin's +database. +""" + +import logging +from argparse import Namespace +from typing import Any, List + +from merlin.db_scripts.merlin_db import MerlinDatabase + + +LOG = logging.getLogger("merlin") + + +def database_info(): + """ + Print information about the database to the console. + """ + merlin_db = MerlinDatabase() + db_studies = merlin_db.get_all("study") + db_runs = merlin_db.get_all("run") + db_logical_workers = merlin_db.get_all("logical_worker") + db_physical_workers = merlin_db.get_all("physical_worker") + + print("Merlin Database Information") + print("---------------------------") + print("General Information:") + print(f"- Database Type: {merlin_db.get_db_type()}") + print(f"- Database Version: {merlin_db.get_db_version()}") + print(f"- Connection String: {merlin_db.get_connection_string()}") + + print() + print("Studies:") + print(f"- Total: {len(db_studies)}") + # TODO add something about recent studies that looks like so: + # - Recent Studies: + # 1. Study ID: 123, Name: "Experiment A" + # 2. Study ID: 124, Name: "Experiment B" + # 3. Study ID: 125, Name: "Experiment C" + # (and 9 more studies) + + print() + print("Runs:") + print(f"- Total: {len(db_runs)}") + # TODO add something about recent runs that looks like so: + # - Recent Runs: + # 1. Run ID: 456, Workspace: "/path/to/workspace" + # 2. Run ID: 457, Workspace: "/path/to/workspace" + # 3. Run ID: 458, Workspace: "/path/to/workspace" + # (and 42 more runs) + + print() + print("Logical Workers:") + print(f"- Total: {len(db_logical_workers)}") + + print() + print("Physical Workers:") + print(f"- Total: {len(db_physical_workers)}") + + print() + + +def database_get(args: Namespace): + """ + Handles the delegation of get operations to Merlin's database. + + Args: + args: Parsed CLI arguments from the user. + """ + merlin_db = MerlinDatabase() + + def print_items(items: List[Any], empty_message: str): + """ + Prints a list of items or logs a message if the list is empty. + + Args: + items: List of items to print. + empty_message: Message to log if the list is empty. + """ + if items: + for item in items: + print(item) + else: + LOG.info(empty_message) + + def get_and_print(entity_type: str, identifiers: List[str]): + """ + Get entities by type and identifiers, then print them. + + Args: + entity_type: The entity type (study, run, logical_worker, etc.). + identifiers: List of identifiers for fetching. + """ + items = [merlin_db.get(entity_type, identifier) for identifier in identifiers] + print_items(items, f"No {entity_type.replace('_', ' ')}s found for the given identifiers.") + + def get_all_and_print(entity_type: str): + """ + Get all entities of a given type and print them. + + Args: + entity_type: The entity type. + """ + items = merlin_db.get_all(entity_type) + entity_type_str = "studies" if entity_type == "study" else f"{entity_type.replace('_', ' ')}s" + print_items(items, f"No {entity_type_str} found in the database.") + + operations = { + "study": lambda: get_and_print("study", args.study), + "run": lambda: get_and_print("run", args.run), + "logical-worker": lambda: get_and_print("logical_worker", args.worker), + "physical-worker": lambda: get_and_print("physical_worker", args.worker), + "all-studies": lambda: get_all_and_print("study"), + "all-runs": lambda: get_all_and_print("run"), + "all-logical-workers": lambda: get_all_and_print("logical_worker"), + "all-physical-workers": lambda: get_all_and_print("physical_worker"), + "everything": lambda: print_items(merlin_db.get_everything(), "Nothing found in the database."), + } + + operation = operations.get(args.get_type) + if operation: + operation() + else: + LOG.error("No valid get option provided.") + + +def database_delete(args: Namespace): + """ + Handles the delegation of delete operations to Merlin's database. + + Args: + args: Parsed CLI arguments from the user. + """ + merlin_db = MerlinDatabase() + + def delete_entities(entity_type: str, identifiers: List[str], **kwargs): + """ + Delete a list of entities by type and identifier. + + Args: + entity_type: The entity type (study, run, etc.). + identifiers: The identifiers of the entities to delete. + kwargs: Additional keyword args for the delete call. + """ + for identifier in identifiers: + merlin_db.delete(entity_type, identifier, **kwargs) + + operations = { + "study": lambda: delete_entities("study", args.study, remove_associated_runs=not args.keep_associated_runs), + "run": lambda: delete_entities("run", args.run), + "logical-worker": lambda: delete_entities("logical_worker", args.worker), + "physical-worker": lambda: delete_entities("physical_worker", args.worker), + "all-studies": lambda: merlin_db.delete_all("study", remove_associated_runs=not args.keep_associated_runs), + "all-runs": lambda: merlin_db.delete_all("run"), + "all-logical-workers": lambda: merlin_db.delete_all("logical_worker"), + "all-physical-workers": lambda: merlin_db.delete_all("physical_worker"), + "everything": lambda: merlin_db.delete_everything(force=args.force), + } + + operation = operations.get(args.delete_type) + if operation: + operation() + else: + LOG.error("No valid delete option provided.") diff --git a/merlin/db_scripts/entities/__init__.py b/merlin/db_scripts/entities/__init__.py new file mode 100644 index 000000000..f38dd4d70 --- /dev/null +++ b/merlin/db_scripts/entities/__init__.py @@ -0,0 +1,32 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +The `entities` package defines database entity classes used throughout Merlin for managing core +components such as studies, runs, and workers. These classes provide a structured interface for +interacting with persisted data, ensuring consistency and maintainability. + +At the heart of this package is the abstract base class `DatabaseEntity`, which outlines the +standard methods that all database-backed entities must implement, including save, delete, and +reload operations. + +Subpackages: + - `mixins/`: Contains mixin classes for entities that help reduce shared code. + +Modules: + db_entity.py: Defines the abstract base class [`DatabaseEntity`][db_scripts.entities.db_entity.DatabaseEntity], + which provides a common interface for all database entity classes. + logical_worker_entity.py: Implements the + [`LogicalWorkerEntity`][db_scripts.entities.logical_worker_entity.LogicalWorkerEntity] + class, representing logical workers and their associated operations. + physical_worker_entity.py: Defines the + [`PhysicalWorkerEntity`][db_scripts.entities.physical_worker_entity.PhysicalWorkerEntity] + class for managing physical workers stored in the database. + run_entity.py: Implements the [`RunEntity`][db_scripts.entities.run_entity.RunEntity] class, + which encapsulates database operations related to run records. + study_entity.py: Defines the [`StudyEntity`][db_scripts.entities.study_entity.StudyEntity] class + for handling study-related database interactions. +""" diff --git a/merlin/db_scripts/entities/db_entity.py b/merlin/db_scripts/entities/db_entity.py new file mode 100644 index 000000000..47122de83 --- /dev/null +++ b/merlin/db_scripts/entities/db_entity.py @@ -0,0 +1,219 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module defines the `DatabaseEntity` abstract base class, which serves as a common interface +for interacting with database entities such as studies, runs, and workers. The `DatabaseEntity` +class provides a standardized structure for managing these entities, including methods for +saving, deleting, reloading, and retrieving additional metadata. + +Classes that inherit from `DatabaseEntity` must implement the abstract methods defined in the base +class, ensuring consistency across different types of database entities. This abstraction reduces +code duplication and promotes maintainability by centralizing shared functionality. +""" + +import logging +from abc import ABC, abstractmethod +from typing import Dict, Generic, Type, TypeVar + +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.data_models import BaseDataModel +from merlin.exceptions import EntityNotFoundError, RunNotFoundError, StudyNotFoundError, WorkerNotFoundError + + +# Type variable for entity models +T = TypeVar("T", bound=BaseDataModel) +# Type variable for entity classes +E = TypeVar("E", bound="DatabaseEntity") + +LOG = logging.getLogger(__name__) + + +class DatabaseEntity(Generic[T], ABC): + """ + Abstract base class for database entities such as runs, studies, and workers. + + This class defines the common interface and behavior for interacting with + database entities, including saving, deleting, and reloading data. + + Attributes: + entity_info (T): The data model containing information about the entity. + backend (backends.results_backend.ResultsBackend): The backend instance used + to interact with the database. + + Methods: + __repr__: + Provide a string representation of the entity. + + __str__: + Provide a human-readable string representation of the entity. + + reload_data: + Reload the latest data for this entity from the database. + + get_id: + Get the unique ID for this entity. + + get_additional_data: + Get any additional data saved to this entity. + + save: + Save the current state of this entity to the database. + + _post_save_hook: + Hook called after saving the entity to the database. Subclasses can override + this method to add additional behavior. + + load: + Load an entity from the database by its ID. + + delete: + Delete an entity from the database by its ID. + + entity_type: + Get the type of this entity for database operations. + + _get_entity_type: + Get the entity type for database operations (used internally). + """ + + # Mapping of entity types to their error classes + _error_classes = { + "logical_worker": WorkerNotFoundError, + "physical_worker": WorkerNotFoundError, + "run": RunNotFoundError, + "study": StudyNotFoundError, + } + + def __init__(self, entity_info: T, backend: ResultsBackend): + """ + Initialize a `DatabaseEntity` instance. + + Args: + entity_info (T): The data model containing information about the entity. + backend (backends.results_backend.ResultsBackend): The backend instance used to + interact with the database. + """ + self.entity_info: T = entity_info + self.backend: ResultsBackend = backend + + @abstractmethod + def __repr__(self) -> str: + """ + Provide a string representation of the entity. + """ + raise NotImplementedError("Subclasses of `DatabaseEntity` must implement a `__repr__` method.") + + @abstractmethod + def __str__(self) -> str: + """ + Provide a human-readable string representation of the entity. + """ + raise NotImplementedError("Subclasses of `DatabaseEntity` must implement a `__str__` method.") + + @property + def entity_type(self) -> str: + """ + Get the type of this entity for database operations. + + Returns: + The type name as a string. + """ + return self._get_entity_type() + + @classmethod + def _get_entity_type(cls) -> str: + """ + Get the entity type for database operations. + + Returns: + The type name as a string. + """ + # Default implementation based on class name + # Can be overridden by subclasses if needed + return cls.__name__.lower().replace("entity", "") + + def reload_data(self): + """ + Reload the latest data for this entity from the database. + + Raises: + (exceptions.EntityNotFoundError): If an entry for this entity was not found in the database. + """ + entity_id = self.get_id() + updated_entity_info = self.backend.retrieve(entity_id, self.entity_type) + if not updated_entity_info: + error_class = self._error_classes.get(self.entity_type, EntityNotFoundError) + raise error_class(f"{self.entity_type.capitalize()} with ID {entity_id} not found in the database.") + self.entity_info = updated_entity_info + + def get_id(self) -> str: + """ + Get the unique ID for this entity. + + Returns: + The unique ID for this entity. + """ + return self.entity_info.id + + def get_additional_data(self) -> Dict: + """ + Get any additional data saved to this entity. + + Returns: + A dictionary of additional data. + """ + self.reload_data() + return self.entity_info.additional_data + + def save(self): + """ + Save the current state of this entity to the database. + """ + self.backend.save(self.entity_info) + self._post_save_hook() + + def _post_save_hook(self): + """ + Hook called after saving the entity to the database. + Subclasses can override to add additional behavior. + """ + + @classmethod + def load(cls: Type[E], entity_identifier: str, backend: ResultsBackend) -> E: + """ + Load an entity from the database by its ID. + + Args: + entity_identifier: The ID of the entity to load. + backend (backends.results_backend.ResultsBackend): A `ResultsBackend` instance. + + Returns: + An instance of the entity. + + Raises: + (exceptions.EntityNotFoundError): If an entry for the entity was not found in the database. + """ + entity_info = backend.retrieve(entity_identifier, cls._get_entity_type()) + if not entity_info: + error_class = cls._error_classes.get(cls._get_entity_type(), EntityNotFoundError) + raise error_class(f"{cls._get_entity_type().capitalize()} with ID {entity_identifier} not found in the database.") + + return cls(entity_info, backend) + + @classmethod + def delete(cls, entity_identifier: str, backend: ResultsBackend): + """ + Delete an entity from the database by its ID. + + Args: + entity_identifier: The ID of the entity to delete. + backend (backends.results_backend.ResultsBackend): A ResultsBackend instance. + """ + entity_type = cls._get_entity_type() + LOG.debug(f"Deleting {entity_type} with ID '{entity_identifier}' from the database...") + backend.delete(entity_identifier, entity_type) + LOG.info(f"{entity_type.capitalize()} with ID '{entity_identifier}' has been successfully deleted.") diff --git a/merlin/db_scripts/entities/logical_worker_entity.py b/merlin/db_scripts/entities/logical_worker_entity.py new file mode 100644 index 000000000..759267aa7 --- /dev/null +++ b/merlin/db_scripts/entities/logical_worker_entity.py @@ -0,0 +1,176 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Module for managing database entities related to logical workers. + +This module defines the `LogicalWorkerEntity` class, which extends the abstract base class +[`DatabaseEntity`][db_scripts.entities.db_entity.DatabaseEntity], to encapsulate logical-worker-specific +operations and behaviors. +""" + +import logging +from typing import List + +from merlin.db_scripts.data_models import LogicalWorkerModel +from merlin.db_scripts.entities.db_entity import DatabaseEntity +from merlin.db_scripts.entities.mixins.name import NameMixin +from merlin.db_scripts.entities.mixins.queue_management import QueueManagementMixin +from merlin.db_scripts.entities.mixins.run_management import RunManagementMixin +from merlin.db_scripts.entities.physical_worker_entity import PhysicalWorkerEntity + + +LOG = logging.getLogger("merlin") + + +class LogicalWorkerEntity(DatabaseEntity[LogicalWorkerModel], RunManagementMixin, QueueManagementMixin, NameMixin): + """ + A class representing a logical worker in the database. + + This class provides methods to interact with and manage a logical worker's data, including + retrieving, adding, and removing run IDs and physical worker IDs from their respective lists, + as well as saving or deleting the logical worker itself from the database. + + Attributes: + entity_info (db_scripts.data_models.LogicalWorkerModel): An instance of the `LogicalWorkerModel` + class containing the logical worker's metadata. + backend (backends.results_backend.ResultsBackend): An instance of the `ResultsBackend` + class used to interact with the database. + + Methods: + __repr__: + Provide a string representation of the `LogicalWorkerEntity` instance. + + __str__: + Provide a human-readable string representation of the `LogicalWorkerEntity` instance. + + reload_data: + Reload the latest data for this logical worker from the database. + + get_id: + Retrieve the unique ID of the logical worker. _Implementation found in + [`DatabaseEntity.get_id`][db_scripts.entities.db_entity.DatabaseEntity.get_id]._ + + get_additional_data: + Retrieve any additional metadata associated with the logical worker. _Implementation found in + [`DatabaseEntity.get_additional_data`][db_scripts.entities.db_entity.DatabaseEntity.get_additional_data]._ + + get_name: + Retrieve the name of the logical worker. + + get_runs: + Retrieve the IDs of the runs using this logical worker. + + add_run: + Add a run ID to the list of runs. + + remove_run: + Remove a run ID from the list of runs. + + get_physical_workers: + Retrieve the IDs of the physical workers created from this logical worker. + + add_physical_worker: + Add a physical worker ID to the list of physical workers. + + remove_physical_worker: + Remove a physical worker ID from the list of physical workers. + + get_queues: + Retrieve the list of queues that this worker is assigned to. + + save: + Save the current state of the logical worker to the database. + + load: + (classmethod) Load a `LogicalWorkerEntity` instance from the database by its ID. + + delete: + (classmethod) Delete a logical worker from the database by its ID. + """ + + @classmethod + def _get_entity_type(cls) -> str: + return "logical_worker" + + def __repr__(self) -> str: + """ + Provide a string representation of the `LogicalWorkerEntity` instance. + + Returns: + A human-readable string representation of the `LogicalWorkerEntity` instance. + """ + return ( + f"LogicalWorkerEntity(" + f"id={self.get_id()}, " + f"name={self.get_name()}, " + f"runs={self.get_runs()}, " + f"queues={self.get_queues()}, " + f"physical_workers={self.get_physical_workers()}, " + f"additional_data={self.get_additional_data()}, " + f"backend={self.backend.get_name()})" + ) + + def __str__(self) -> str: + """ + Provide a string representation of the `LogicalWorkerEntity` instance. + + Returns: + A human-readable string representation of the `LogicalWorkerEntity` instance. + """ + worker_id = self.get_id() + physical_workers = [ + PhysicalWorkerEntity.load(physical_worker_id, self.backend) for physical_worker_id in self.get_physical_workers() + ] + physical_worker_str = "" + if physical_workers: + for physical_worker in physical_workers: + physical_worker_str += f" - ID: {physical_worker.get_id()}\n Name: {physical_worker.get_name()}\n" + else: + physical_worker_str = " No physical workers found.\n" + return ( + f"Logical Worker with ID {worker_id}\n" + f"------------{'-' * len(worker_id)}\n" + f"Name: {self.get_name()}\n" + f"Runs:\n{self.construct_run_string()}" + f"Queues: {self.get_queues()}\n" + f"Physical Workers:\n{physical_worker_str}" + f"Additional Data: {self.get_additional_data()}\n\n" + ) + + def get_physical_workers(self) -> List[str]: + """ + Get the physical instances of this logical worker. + + Returns: + A list of physical worker IDs. + """ + self.reload_data() + return self.entity_info.physical_workers + + def add_physical_worker(self, physical_worker_id: str): + """ + Add a new physical worker id to the list of physical workers. + + Args: + physical_worker_id: The id of the physical worker to add. + """ + self.entity_info.physical_workers.append(physical_worker_id) + self.save() + + def remove_physical_worker(self, physical_worker_id: str): + """ + Remove a physical worker id from the list of physical workers. + + Does *not* delete a [`PhysicalWorkerEntity`][db_scripts.entities.physical_worker_entity.PhysicalWorkerEntity] + from the database. This will only remove the physical worker's id from the list in this worker. + + Args: + physical_worker_id: The ID of the physical worker to remove. + """ + self.reload_data() + self.entity_info.physical_workers.remove(physical_worker_id) + self.save() diff --git a/merlin/db_scripts/entities/mixins/__init__.py b/merlin/db_scripts/entities/mixins/__init__.py new file mode 100644 index 000000000..0793e419d --- /dev/null +++ b/merlin/db_scripts/entities/mixins/__init__.py @@ -0,0 +1,25 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +The `mixins` package provides reusable mixin classes that encapsulate common behaviors +shared across multiple entity types in the Merlin system. + +These mixins are designed to be composed into larger classes (e.g., entity models or +entity managers) without enforcing inheritance from a shared base, offering lightweight +extensions to class behavior. + +Modules: + name.py: Defines the [`NameMixin`][db_scripts.entities.mixins.name.NameMixin], which provides name-based + access and utility methods for entities with a `name` attribute. + queue_management.py: Defines the + [`QueueManagementMixin`][db_scripts.entities.mixins.queue_management.QueueManagementMixin], + which supports retrieval and filtering of task queues for entities with a `queues` field. + run_management.py: Defines the + [`RunManagementMixin`][db_scripts.entities.mixins.run_management.RunManagementMixin], + which provides functionality for managing run associations on entities that support saving, + reloading, and tracking linked run IDs. +""" diff --git a/merlin/db_scripts/entities/mixins/name.py b/merlin/db_scripts/entities/mixins/name.py new file mode 100644 index 000000000..d0c73bcff --- /dev/null +++ b/merlin/db_scripts/entities/mixins/name.py @@ -0,0 +1,42 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Module for managing entities with names. + +This module provides a mixin class, `NameMixin`, designed to simplify the management +of names associated with an entity. The mixin can be used by any class that has the +necessary attributes to support name retrieval, specifically an `entity_info` object +containing a `name` attribute. +""" + +# pylint: disable=too-few-public-methods + + +class NameMixin: + """ + Mixin for entities that have a name. + + This mixin provides a method for retrieving the name associated with an entity. + It assumes that the class using this mixin has an `entity_info` object containing + a `name` attribute. + + Methods: + get_name: + Retrieve the name associated with the entity. + """ + + def get_name(self) -> str: + """ + Get the name of this entity. + + Assumptions: + - The class using this must have an `entity_info` object containing a `name` attribute + + Returns: + The name of this entity. + """ + return self.entity_info.name diff --git a/merlin/db_scripts/entities/mixins/queue_management.py b/merlin/db_scripts/entities/mixins/queue_management.py new file mode 100644 index 000000000..82aa88671 --- /dev/null +++ b/merlin/db_scripts/entities/mixins/queue_management.py @@ -0,0 +1,45 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Module for managing task queues associated with entities. + +This module provides a mixin class, `QueueManagementMixin`, designed to simplify the +management of task queues associated with an entity. The mixin can be used by any +class that has the necessary attributes to support queue retrieval, specifically an +`entity_info` object containing a `queues` list. +""" + +from typing import List + + +# pylint: disable=too-few-public-methods + + +class QueueManagementMixin: + """ + Mixin for managing queues associated with an entity. + + This mixin provides a method for retrieving the task queues assigned to an entity. + It assumes that the class using this mixin has an `entity_info` object containing + a `queues` list. + + Methods: + get_queues: + Retrieve the task queues assigned to the entity. + """ + + def get_queues(self) -> List[str]: + """ + Get the queues that this entity is assigned to. + + Assumptions: + - The class using this must have an `entity_info` object containing a `queues` list + + Returns: + A list of queue names. + """ + return self.entity_info.queues diff --git a/merlin/db_scripts/entities/mixins/run_management.py b/merlin/db_scripts/entities/mixins/run_management.py new file mode 100644 index 000000000..ad269441f --- /dev/null +++ b/merlin/db_scripts/entities/mixins/run_management.py @@ -0,0 +1,107 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Module for managing entities and their associated runs in the database. + +This module provides a mixin class, `RunManagementMixin`, designed to simplify the management +of runs associated with an entity. The mixin can be used by any class that has the necessary +attributes and methods to support run management, such as `reload_data`, `save`, and an +`entity_info` object containing a `runs` list. +""" + +from typing import List + + +class RunManagementMixin: + """ + Mixin for managing runs associated with an entity. + + This mixin provides utility methods for handling run IDs associated with an entity. + It assumes that the class using this mixin has the necessary attributes and methods + to support run management, including `reload_data`, `save`, and an `entity_info` object + containing a `runs` list. + + Methods: + get_runs: + Retrieve the IDs of the runs associated with the entity. + + add_run: + Add a run ID to the list of runs. + + remove_run: + Remove a run ID from the list of runs. + """ + + def get_runs(self) -> List[str]: + """ + Get every run listed in this entity. + + Assumptions: + - The class using this must have a `reload_data` method + - The class using this must have an `entity_info` object containing a `runs` list + + Returns: + A list of run IDs. + """ + self.reload_data() + return self.entity_info.runs + + def add_run(self, run_id: str): + """ + Add a new run ID to the list of runs. + + Assumptions: + - The class using this must have an `entity_info` object containing a `runs` list + - The class using this must have a `save` method + + Args: + run_id: The ID of the run to add. + """ + self.entity_info.runs.append(run_id) + self.save() + + def remove_run(self, run_id: str): + """ + Remove a run ID from the list of runs. + + Does *not* delete the run entity from the database. This will only remove the run's ID + from the list in this entity. + + Assumptions: + - The class using this must have a `reload_data` method + - The class using this must have an `entity_info` object containing a `runs` list + - The class using this must have a `save` method + + Args: + run_id: The ID of the run to remove. + """ + self.reload_data() + self.entity_info.runs.remove(run_id) + self.save() + + def construct_run_string(self) -> str: + """ + Constructs and returns a formatted string representation of all runs associated + with the current instance. + + Returns: + A formatted string containing details of all runs. + """ + from merlin.db_scripts.entities.run_entity import RunEntity # pylint: disable=import-outside-toplevel + + runs = self.get_runs() + run_str = "" + if runs: + for run_id in runs: + try: + run = RunEntity.load(run_id, self.backend) + run_str += f" - ID: {run.get_id()}\n Workspace: {run.get_workspace()}\n" + except Exception: # pylint: disable=broad-exception-caught + run_str += f" - ID: {run_id} (Error loading run)\n" + else: + run_str = " No runs found.\n" + return run_str diff --git a/merlin/db_scripts/entities/physical_worker_entity.py b/merlin/db_scripts/entities/physical_worker_entity.py new file mode 100644 index 000000000..b866ca62a --- /dev/null +++ b/merlin/db_scripts/entities/physical_worker_entity.py @@ -0,0 +1,304 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Module for managing database entities related to physical workers. + +This module provides functionality for interacting with physical workers stored in a database, +including creating, retrieving, updating, and deleting them. It defines the `WorkerEntity` +class, which extends the abstract base class [`DatabaseEntity`][db_scripts.entities.db_entity.DatabaseEntity], +to encapsulate worker-specific operations and behaviors. +""" + +import logging +from datetime import datetime +from typing import Dict, Optional + +from merlin.common.enums import WorkerStatus +from merlin.db_scripts.data_models import PhysicalWorkerModel +from merlin.db_scripts.entities.db_entity import DatabaseEntity +from merlin.db_scripts.entities.mixins.name import NameMixin + + +LOG = logging.getLogger("merlin") + + +class PhysicalWorkerEntity(DatabaseEntity[PhysicalWorkerModel], NameMixin): + """ + A class representing a physical worker in the database. + + This class provides methods to interact with and manage a worker's data, including + retrieving information about the worker, updating its state, and saving or deleting + it from the database. + + Attributes: + entity_info (db_scripts.data_models.PhysicalWorkerModel): An instance of the `PhysicalWorkerModel` + class containing the physical worker's metadata. + backend (backends.results_backend.ResultsBackend): An instance of the `ResultsBackend` + class used to interact with the database. + + Methods: + __repr__: + Provide a string representation of the `PhysicalWorkerEntity` instance. + __str__: + Provide a human-readable string representation of the `PhysicalWorkerEntity` instance. + reload_data: + Reload the latest data for this worker from the database. + get_id: + Retrieve the ID of the worker. _Implementation found in + [`DatabaseEntity.get_id`][db_scripts.entities.db_entity.DatabaseEntity.get_id]._ + get_additional_data: + Retrieve any additional data saved to this worker. _Implementation found in + [`DatabaseEntity.get_additional_data`][db_scripts.entities.db_entity.DatabaseEntity.get_additional_data]._ + get_name: + Retrieve the name of this worker. + get_logical_worker_id: + Retrieve the ID of the logical worker that this physical worker was created from. + get_launch_cmd: + Retrieve the command used to launch this worker. + set_launch_cmd: + Update the launch command used to start this worker. + get_args: + Retrieve the arguments for this worker. + set_args: + Update the arguments used by this worker. + get_pid: + Retrieve the process ID for this worker. + set_pid: + Update the process ID for this worker. + get_status: + Retrieve the status of this worker. + set_status: + Update the status of this worker. + get_heartbeat_timestamp: + Retrieve the last heartbeat timestamp of this worker. + set_heartbeat_timestamp: + Update the latest heartbeat timestamp of this worker. + get_latest_start_time: + Retrieve the time this worker was last started. + set_latest_start_time: + Update the latest start time of this worker. + get_host: + Retrieve the hostname where this worker is running. + get_restart_count: + Retrieve the number of times this worker has been restarted. + increment_restart_count: + Increment the restart count for this worker. + save: + Save the current state of this worker to the database. + load: + (classmethod) Load a `PhysicalWorkerEntity` instance from the database by its ID or name. + delete: + (classmethod) Delete a worker from the database by its ID or name. + """ + + @classmethod + def _get_entity_type(cls) -> str: + return "physical_worker" + + def __repr__(self) -> str: + """ + Provide a string representation of the `PhysicalWorkerEntity` instance. + + Returns: + A human-readable string representation of the `PhysicalWorkerEntity` instance. + """ + return ( + f"PhysicalWorkerEntity(" + f"id={self.get_id()}, " + f"name={self.get_name()}, " + f"logical_worker_id={self.get_logical_worker_id()}, " + f"launch_cmd={self.get_launch_cmd()}, " + f"args={self.get_args()}, " + f"pid={self.get_pid()}, " + f"status={self.get_status()}, " + f"heartbeat_timestamp={self.get_heartbeat_timestamp()}, " + f"latest_start_time={self.get_latest_start_time()}, " + f"host={self.get_host()}, " + f"restart_count={self.get_restart_count()}, " + f"additional_data={self.get_additional_data()}, " + f"backend={self.backend.get_name()})" + ) + + def __str__(self) -> str: + """ + Provide a string representation of the `PhysicalWorkerEntity` instance. + + Returns: + A human-readable string representation of the `PhysicalWorkerEntity` instance. + """ + worker_id = self.get_id() + return ( + f"Physical Worker with ID {worker_id}\n" + f"------------{'-' * len(worker_id)}\n" + f"Name: {self.get_name()}\n" + f"Logical Worker ID: {self.get_logical_worker_id()}\n" + f"Launch Command: {self.get_launch_cmd()}\n" + f"Args: {self.get_args()}\n" + f"Process ID: {self.get_pid()}\n" + f"Status: {self.get_status()}\n" + f"Last Heartbeat: {self.get_heartbeat_timestamp()}\n" + f"Last Spinup: {self.get_latest_start_time()}\n" + f"Host: {self.get_host()}\n" + f"Restart Count: {self.get_restart_count()}\n" + f"Additional Data: {self.get_additional_data()}\n\n" + ) + + def get_logical_worker_id(self) -> str: + """ + Get the ID of the logical worker that this physical worker was created from. + + Returns: + The ID of the logical worker that this physical worker was created from. + """ + return self.entity_info.logical_worker_id + + def get_launch_cmd(self) -> str: + """ + Get the command used to launch this worker. + + Returns: + The command used to launch this worker. + """ + return self.entity_info.launch_cmd + + def set_launch_cmd(self, launch_cmd: str): + """ + Set the launch command used to start this worker. + + Args: + launch_cmd: The launch command used to start this worker. + """ + self.entity_info.launch_cmd = launch_cmd + self.save() + + def get_args(self) -> Dict: + """ + Get the arguments for this worker. + + Returns: + A dictionary of arguments for this worker. + """ + return self.entity_info.args + + def set_args(self, args: str): + """ + Set the arguments used by this worker. + + Args: + args: The arguments used by this worker. + """ + self.entity_info.args = args + self.save() + + def get_pid(self) -> Optional[int]: + """ + Get the process ID for this worker. + + Returns: + The process ID for this worker or None if not set. + """ + self.reload_data() + return int(self.entity_info.pid) if self.entity_info.pid else None + + def set_pid(self, pid: str): + """ + Set the PID of this worker. + + Args: + pid: The new PID of this worker. + """ + self.entity_info.pid = pid + self.save() + + def get_status(self) -> WorkerStatus: + """ + Get the status of this worker. + + Returns: + A [`WorkerStatus`][common.enums.WorkerStatus] enum representing + the status of this worker. + """ + self.reload_data() + return self.entity_info.status + + def set_status(self, status: WorkerStatus): + """ + Set the status of this worker. + + Args: + status: A [`WorkerStatus`][common.enums.WorkerStatus] enum representing + the new status of the worker. + """ + self.entity_info.status = status + self.save() + + def get_heartbeat_timestamp(self) -> str: + """ + Get the last heartbeat timestamp of this worker. + + Returns: + The last heartbeat timestamp we received from this worker + """ + self.reload_data() + return self.entity_info.heartbeat_timestamp + + def set_heartbeat_timestamp(self, heartbeat_timestamp: datetime): + """ + Set the latest heartbeat timestamp of this worker. + + Args: + heartbeat_timestamp: The latest heartbeat timestamp of this worker. + """ + self.entity_info.heartbeat_timestamp = heartbeat_timestamp + self.save() + + def get_latest_start_time(self) -> datetime: + """ + Get the time that this worker was last started. + + Returns: + A datetime object representing the last time this worker was started. + """ + self.reload_data() + return self.entity_info.latest_start_time + + def set_latest_start_time(self, latest_start_time: datetime): + """ + Set the latest start time of this worker. This will be set on worker + startup followed by any time the worker is restarted. + + Args: + latest_start_time: The latest start time of this worker. + """ + self.entity_info.latest_start_time = latest_start_time + self.save() + + def get_host(self) -> str: + """ + Get the hostname where this worker is running. + + Returns: + The name of the host that this worker is running on. + """ + return self.entity_info.host + + def get_restart_count(self) -> int: + """ + Get the number of times that this worker has been restarted. + + Returns: + The number of times that this worker has been restarted. + """ + self.reload_data() + return self.entity_info.restart_count + + def increment_restart_count(self): + """ + Add another restart to the restart count. + """ + self.entity_info.restart_count = self.get_restart_count() + 1 + self.save() diff --git a/merlin/db_scripts/entities/run_entity.py b/merlin/db_scripts/entities/run_entity.py new file mode 100644 index 000000000..13da4e029 --- /dev/null +++ b/merlin/db_scripts/entities/run_entity.py @@ -0,0 +1,345 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Module for managing database entities related to runs. + +This module provides functionality for interacting with runs stored in a database, +including creating, retrieving, updating, and deleting runs. It defines the `RunEntity` +class, which extends the abstract base class [`DatabaseEntity`][db_scripts.entities.db_entity.DatabaseEntity], +to encapsulate run-specific operations and behaviors. +""" + +import logging +import os +from typing import List + +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.data_models import RunModel +from merlin.db_scripts.entities.db_entity import DatabaseEntity +from merlin.db_scripts.entities.mixins.queue_management import QueueManagementMixin +from merlin.exceptions import RunNotFoundError + + +LOG = logging.getLogger("merlin") + + +class RunEntity(DatabaseEntity[RunModel], QueueManagementMixin): + """ + A class representing a run in the database. + + This class provides methods to interact with and manage a run's data, including + retrieving information about the run, updating its state, and saving or deleting + it from the database. + + Attributes: + entity_info (db_scripts.data_models.RunModel): An instance of the `RunModel` class + containing the run's metadata. + backend (backends.results_backend.ResultsBackend): An instance of the `ResultsBackend` + class used to interact with the database. + run_complete (bool): A property to get or set the completion status of the run. + + Methods: + __repr__: + Provide a string representation of the `RunEntity` instance. + + __str__: + Provide a human-readable string representation of the `RunEntity` instance. + + reload_data: + Reload the latest data for this run from the database. + + get_id: + Retrieve the ID of the run. _Implementation found in + [`DatabaseEntity.get_id`][db_scripts.entities.db_entity.DatabaseEntity.get_id]._ + + get_additional_data: + Retrieve any additional data saved to this run. _Implementation found in + [`DatabaseEntity.get_additional_data`][db_scripts.entities.db_entity.DatabaseEntity.get_additional_data]._ + + get_metadata_file: + Retrieve the path to the metadata file for this run. + + get_metadata_filepath: + (classmethod) Retrieve the path to the metadata file for a given workspace. + + get_study_id: + Retrieve the ID of the study associated with this run. + + get_workspace: + Retrieve the path to the output workspace for this run. + + get_queues: + Retrieve the task queues used for this run. + + get_workers: + Retrieve the workers used for this run. + + add_worker: + Add a worker to the list of workers used for this run. + + remove_worker: + Remove a worker from the list of workers used for this run. + + get_parent: + Retrieve the ID of the parent run that launched this run (if any). + + get_child: + Retrieve the ID of the child run launched by this run (if any). + + save: + Save the current state of the run to the database and dump its metadata. + + dump_metadata: + Dump all metadata for this run to a JSON file. + + load: + (classmethod) Load a `RunEntity` instance from the database by its ID or workspace. + + delete: + (classmethod) Delete a run from the database by its ID or workspace. + """ + + def __init__(self, run_info: RunModel, backend: ResultsBackend): + """ + Initialize a `RunEntity` instance. + + Args: + run_info (db_scripts.data_models.RunModel): The data model containing + information about the run. + backend (backends.results_backend.ResultsBackend): The backend instance used to + interact with the database. + """ + super().__init__(run_info, backend) + self._metadata_file = self.get_metadata_filepath(self.get_workspace()) + + @classmethod + def _get_entity_type(cls) -> str: + return "run" + + def __repr__(self) -> str: + """ + Provide a string representation of the `RunEntity` instance. + + Returns: + A human-readable string representation of the `RunEntity` instance. + """ + return ( + f"RunEntity(" + f"id={self.get_id()}, " + f"study_id={self.get_study_id()}, " + f"workspace={self.get_workspace()}, " + f"queues={self.get_queues()}, " + f"workers={self.get_workers()}, " + f"parent={self.get_parent()}, " + f"child={self.get_child()}, " + f"run_complete={self.run_complete}, " + f"additional_data={self.get_additional_data()}, " + f"backend={self.backend.get_name()})" + ) + + def __str__(self) -> str: + """ + Provide a string representation of the `RunEntity` instance. + + Returns: + A human-readable string representation of the `RunEntity` instance. + """ + from merlin.db_scripts.entities.study_entity import StudyEntity # pylint: disable=import-outside-toplevel + + run_id = self.get_id() + study = StudyEntity.load(self.get_study_id(), self.backend) + study_str = f" - ID: {study.get_id()}\n Name: {study.get_name()}" + return ( + f"Run with ID {run_id}\n" + f"------------{'-' * len(run_id)}\n" + f"Workspace: {self.get_workspace()}\n" + f"Study:\n{study_str}\n" + f"Queues: {self.get_queues()}\n" + f"Workers: {self.get_workers()}\n" + f"Parent: {self.get_parent()}\n" + f"Child: {self.get_child()}\n" + f"Run Complete: {self.run_complete}\n" + f"Additional Data: {self.get_additional_data()}\n\n" + ) + + @property + def run_complete(self) -> bool: + """ + An attribute representing whether this run is complete. + + A "complete" study is a study that has executed all steps. + + Returns: + True if the study is complete. False, otherwise. + """ + self.reload_data() + return self.entity_info.run_complete + + @run_complete.setter + def run_complete(self, value: bool): + """ + Update the run's completion status. + + Args: + value: The completion status of the run. + """ + self.entity_info.run_complete = value + + def get_metadata_file(self) -> str: + """ + Get the path to the metadata file for this run. + + Returns: + The path to the metadata file for this run + """ + return self._metadata_file + + @classmethod + def get_metadata_filepath(cls, workspace: str) -> str: + """ + Get the path to the metadata file for a given workspace. + This is needed for the [`load`][db_scripts.entities.run_entity.RunEntity.load] + method when loading from workspace as it can't use the non-classmethod version + of this method. + + Args: + workspace: The workspace directory for the run. + + Returns: + The path to the metadata file. + """ + return os.path.join(workspace, "merlin_info", "run_metadata.json") + + def get_study_id(self) -> str: + """ + Get the ID for the study associated with this run. + + Returns: + The ID for the study associated with this run. + """ + return self.entity_info.study_id + + def get_workspace(self) -> str: + """ + Get the path to the output workspace for this run. + + Returns: + A string representing the output workspace for this run. + """ + return self.entity_info.workspace + + def get_workers(self) -> List[str]: + """ + Get the logical workers that this run is using. + + Returns: + A list of logical worker ids. + """ + return self.entity_info.workers + + def add_worker(self, worker_id: str): + """ + Add a new worker id to the list of workers. + + Args: + worker_id: The id of the worker to add. + """ + self.entity_info.workers.append(worker_id) + self.save() + + def remove_worker(self, worker_id: str): + """ + Remove a worker id from the list of workers associated with this run. + + Does *not* delete a [`LogicalWorkerEntity`][db_scripts.entities.logical_worker_entity.LogicalWorkerEntity] + from the database. This will only remove the worker's id from the list in this run. + + Args: + worker_id: The ID of the worker to remove. + """ + self.reload_data() + self.entity_info.workers.remove(worker_id) + self.save() + + def get_parent(self) -> str: + """ + Get the ID of the run that launched this run (if any). + + This will only be set for iterative workflows with greater than 1 iteration. + + Returns: + The ID of the run that launched this run. + """ + self.reload_data() + return self.entity_info.parent + + def get_child(self) -> str: + """ + Get the ID of the run that was launched by this run (if any). + + This will only be set for iterative workflows with greater than 1 iteration. + + Returns: + The ID of the run that was launched by this run. + """ + self.reload_data() + return self.entity_info.child + + def _post_save_hook(self): + """ + Hook called after saving the run entity. + For runs, we also need to dump metadata to a file. + """ + self.dump_metadata() + + def dump_metadata(self): + """ + Dump all of the metadata for this run to a json file. + """ + self.entity_info.dump_to_json_file(self.get_metadata_file()) + + @classmethod + def load(cls, entity_identifier: str, backend: ResultsBackend) -> "RunEntity": + """ + Load a run from the database by id or workspace. + + Args: + entity_identifier: The ID or workspace of the run to load. + backend: A [`ResultsBackend`][backends.results_backend.ResultsBackend] instance. + + Returns: + A `RunEntity` instance. + + Raises: + (exceptions.RunNotFoundError): If an entry for run was not found in the database. + """ + if os.path.isdir(entity_identifier) and os.path.exists(entity_identifier): # Load from workspace + LOG.debug("Retrieving run from workspace.") + metadata_file = cls.get_metadata_filepath(entity_identifier) + entity_info = RunModel.load_from_json_file(metadata_file) + else: # Load from ID + LOG.debug("Retrieving run from backend.") + entity_info = backend.retrieve(entity_identifier, "run") + + if not entity_info: + raise RunNotFoundError(f"Run with ID or workspace {entity_identifier} not found in the database.") + + return cls(entity_info, backend) + + @classmethod + def delete(cls, entity_identifier: str, backend: ResultsBackend): + """ + Delete a run from the database by id or workpsace. + + Args: + entity_identifier: The ID or workspace of the run to delete. + backend: A [`ResultsBackend`][backends.results_backend.ResultsBackend] instance. + """ + LOG.debug(f"Deleting run with id or workspace '{entity_identifier}' from the database...") + self = cls.load(entity_identifier, backend) + backend.delete(self.get_id(), "run") + LOG.info(f"Run with id or workspace '{entity_identifier}' has been successfully deleted.") diff --git a/merlin/db_scripts/entities/study_entity.py b/merlin/db_scripts/entities/study_entity.py new file mode 100644 index 000000000..289421a85 --- /dev/null +++ b/merlin/db_scripts/entities/study_entity.py @@ -0,0 +1,114 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Module for managing database entities related to studies. + +This module defines the `StudyEntity` class, which extends the abstract base class +[`DatabaseEntity`][db_scripts.entities.db_entity.DatabaseEntity], to encapsulate study-specific +operations and behaviors. +""" + +import logging + +from merlin.db_scripts.data_models import StudyModel +from merlin.db_scripts.entities.db_entity import DatabaseEntity +from merlin.db_scripts.entities.mixins.name import NameMixin +from merlin.db_scripts.entities.mixins.run_management import RunManagementMixin + + +LOG = logging.getLogger("merlin") + + +class StudyEntity(DatabaseEntity[StudyModel], RunManagementMixin, NameMixin): + """ + A class representing a study in the database. + + This class provides methods to interact with and manage a study's data, including + retrieving, adding, and removing run IDs from the list of runs associated with the + study, as well as saving or deleting the study itself from the database. + + Attributes: + entity_info (db_scripts.data_models.StudyModel): An instance of the `StudyModel` + class containing the study's metadata. + backend (backends.results_backend.ResultsBackend): An instance of the `ResultsBackend` + class used to interact with the database. + + Methods: + __repr__: + Provide a string representation of the `StudyEntity` instance. + + __str__: + Provide a human-readable string representation of the `StudyEntity` instance. + + reload_data: + Reload the latest data for this study from the database. + + get_id: + Retrieve the unique ID of the study. _Implementation found in + [`DatabaseEntity.get_id`][db_scripts.entities.db_entity.DatabaseEntity.get_id]._ + + get_additional_data: + Retrieve any additional metadata associated with the study. _Implementation found in + [`DatabaseEntity.get_additional_data`][db_scripts.entities.db_entity.DatabaseEntity.get_additional_data]._ + + get_name: + Retrieve the name of the study. + + get_runs: + Retrieve the IDs of the runs associated with this study. + + add_run: + Add a run ID to the list of runs. + + remove_run: + Remove a run ID from the list of runs. + + save: + Save the current state of the study to the database. + + load: + (classmethod) Load a `StudyEntity` instance from the database by its ID or name. + + delete: + (classmethod) Delete a study from the database by its ID or name. Optionally, remove all associated runs. + """ + + @classmethod + def _get_entity_type(cls) -> str: + return "study" + + def __repr__(self) -> str: + """ + Provide a string representation of the `StudyEntity` instance. + + Returns: + A human-readable string representation of the `StudyEntity` instance. + """ + return ( + f"StudyEntity(" + f"id={self.get_id()}, " + f"name={self.get_name()}, " + f"runs={self.get_runs()}, " + f"additional_data={self.get_additional_data()}, " + f"backend={self.backend.get_name()})" + ) + + def __str__(self) -> str: + """ + Provide a string representation of the `StudyEntity` instance. + + Returns: + A human-readable string representation of the `StudyEntity` instance. + """ + study_id = self.get_id() + return ( + f"Study with ID {study_id}\n" + f"------------{'-' * len(study_id)}\n" + f"Name: {self.get_name()}\n" + f"Runs:\n{self.construct_run_string()}" + f"Additional Data: {self.get_additional_data()}\n\n" + ) diff --git a/merlin/db_scripts/entity_managers/__init__.py b/merlin/db_scripts/entity_managers/__init__.py new file mode 100644 index 000000000..3ba089c56 --- /dev/null +++ b/merlin/db_scripts/entity_managers/__init__.py @@ -0,0 +1,31 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +The `entity_managers` package contains classes responsible for managing high-level database +operations across all entity types in the Merlin system. + +Each manager provides CRUD (Create, Read, Update, Delete) operations tailored to a specific +entity, such as studies, runs, or workers. These managers act as intermediaries between the +underlying data models and higher-level business logic, encapsulating shared behaviors and +coordinating entity relationships, lifecycle events, and cross-references. + +Modules: + entity_manager.py: Defines the abstract base class + [`EntityManager`][db_scripts.entity_managers.entity_manager.EntityManager], + which provides shared infrastructure and helper methods for all entity managers. + logical_worker_manager.py: Manages logical workers, including queue-based identity + resolution and cleanup of run associations on deletion. + physical_worker_manager.py: Manages physical workers and handles cleanup of references + from associated logical workers. + run_manager.py: Manages run entities and coordinates updates to related studies and + logical workers. + study_manager.py: Manages study entities and orchestrates the creation and deletion of + studies along with their associated runs. + +These managers rely on the Merlin results backend and may optionally reference each other +via the central `MerlinDatabase` class to perform coordinated, entity-spanning operations. +""" diff --git a/merlin/db_scripts/entity_managers/entity_manager.py b/merlin/db_scripts/entity_managers/entity_manager.py new file mode 100644 index 000000000..8c890fe46 --- /dev/null +++ b/merlin/db_scripts/entity_managers/entity_manager.py @@ -0,0 +1,237 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module defines the abstract base class `EntityManager`, which provides a generic framework +for managing database-backed entities in the Merlin system. + +`EntityManager` is designed to be subclassed for specific entity types (e.g., studies, runs, +workers), allowing consistent implementation of core CRUD operations (Create, Read, Update, Delete) +across different entity managers. It provides shared helper methods for common operations such as +creating entities if they do not exist, retrieving entities by ID, and deleting entities. +""" + +import logging +from abc import ABC, abstractmethod +from typing import Any, Callable, Generic, List, Optional, Type, TypeVar + +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.data_models import BaseDataModel +from merlin.db_scripts.entities.db_entity import DatabaseEntity +from merlin.exceptions import RunNotFoundError, StudyNotFoundError, WorkerNotFoundError + + +T = TypeVar("T", bound=DatabaseEntity) +M = TypeVar("M", bound=BaseDataModel) + +LOG = logging.getLogger("merlin") + + +class EntityManager(Generic[T, M], ABC): + """ + Abstract base class for managing database entity lifecycles. + + This class defines a common interface and helper methods for managing entities stored in the + database, such as studies, runs, and workers. Subclasses must implement methods to create, + retrieve, and delete these entities using the provided backend. + + Generic Parameters: + T (DatabaseEntity): The entity class managed by this manager. + M (BaseDataModel): The data model class corresponding to the entity. + + Attributes: + backend: The backend interface used to persist and retrieve entity data. + + Methods: + create: Abstract method to create a new entity. + get: Abstract method to retrieve a single entity by its identifier. + get_all: Abstract method to retrieve all entities of this type. + delete: Abstract method to delete a specific entity by its identifier. + delete_all: Abstract method to delete all entities of this type. + _create_entity_if_not_exists: Creates an entity only if it does not already exist in the backend. + _get_entity: Retrieves an entity instance using its identifier. + _get_all_entities: Retrieves all entities of a specific type from the backend. + _delete_entity: Deletes an individual entity, optionally calling a cleanup function before deletion. + _delete_all_by_type: Deletes all entities of a certain type using the provided getter and deleter functions. + """ + + def __init__(self, backend: ResultsBackend): + """ + Initialize the EntityManager with a backend. + + Args: + backend: The backend interface used to persist and retrieve entities. + """ + self.backend = backend + self.db = None # Subclasses can set this by creating a set_db_reference method + + @abstractmethod + def create(self, *args: Any, **kwargs: Any) -> T: + """ + Create a new entity. + + This method must be implemented by subclasses to define how a specific entity is created. + + Args: + *args (Any): Optional positional arguments for create context. + **kwargs (Any): Optional keyword arguments for create context. + + Returns: + The newly created entity instance. + + Raises: + NotImplementedError: If a subclass has not implemented this method. + """ + raise NotImplementedError("Subclasses of `EntityManager` must implement a `create` method.") + + @abstractmethod + def get(self, identifier: str) -> T: + """ + Retrieve a single entity by its identifier. + + Args: + identifier (str): The unique identifier of the entity. + + Returns: + The entity instance corresponding to the identifier. + + Raises: + NotImplementedError: If a subclass has not implemented this method. + """ + raise NotImplementedError("Subclasses of `EntityManager` must implement a `get` method.") + + @abstractmethod + def get_all(self) -> List[T]: + """ + Retrieve all entities managed by this entity manager. + + Returns: + A list of all entities of the specified type. + + Raises: + NotImplementedError: If a subclass has not implemented this method. + """ + raise NotImplementedError("Subclasses of `EntityManager` must implement a `get_all` method.") + + @abstractmethod + def delete(self, identifier: str, **kwargs: Any): + """ + Delete a specific entity by its identifier. + + Args: + identifier (str): The unique identifier of the entity to delete. + **kwargs (Any): Optional keyword arguments for deletion context (e.g., cleanup flags). + + Raises: + NotImplementedError: If a subclass has not implemented this method. + """ + raise NotImplementedError("Subclasses of `EntityManager` must implement a `delete` method.") + + @abstractmethod + def delete_all(self, **kwargs: Any): + """ + Delete all entities managed by this entity manager. + + Args: + **kwargs (Any): Optional keyword arguments for deletion context. + + Raises: + NotImplementedError: If a subclass has not implemented this method. + """ + raise NotImplementedError("Subclasses of `EntityManager` must implement a `delete_all` method.") + + def _create_entity_if_not_exists( # pylint: disable=too-many-arguments,too-many-positional-arguments + self, + entity_class: Type[T], + model_class: Type[M], + identifier: str, + log_message_exists: str, + log_message_create: str, + **model_kwargs: Any, + ) -> T: + """ + Helper method to create an entity if it does not already exist. + + Args: + entity_class (Type[T]): The class used to load and instantiate the entity. + model_class (Type[M]): The data model class used to create a new entity if needed. + identifier (str): The unique identifier of the entity. + log_message_exists (str): Log message to emit if the entity already exists. + log_message_create (str): Log message to emit when creating a new entity. + **model_kwargs (Any): Keyword arguments passed to the model class constructor. + + Returns: + The existing or newly created entity. + """ + try: + entity = entity_class.load(identifier, self.backend) + LOG.info(log_message_exists) + except (WorkerNotFoundError, StudyNotFoundError, RunNotFoundError): + LOG.info(log_message_create) + model = model_class(**model_kwargs) + entity = entity_class(model, self.backend) + entity.save() + return entity + + def _get_entity(self, entity_class: Type[T], identifier: str) -> T: + """ + Retrieve a single entity instance using its identifier. + + Args: + entity_class (Type[T]): The class used to load the entity. + identifier (str): The unique identifier of the entity. + + Returns: + The loaded entity instance. + """ + return entity_class.load(identifier, self.backend) + + def _get_all_entities(self, entity_class: Type[T], entity_type: str) -> List[T]: + """ + Retrieve all entities of a specific type from the backend. + + Args: + entity_class (Type[T]): The class used to instantiate each entity. + entity_type (str): The type identifier used by the backend to filter entities. + + Returns: + A list of all entities of the specified type. + """ + all_entities = self.backend.retrieve_all(entity_type) + if not all_entities: + return [] + return [entity_class(entity_data, self.backend) for entity_data in all_entities] + + def _delete_entity(self, entity_class: Type[T], identifier: str, cleanup_fn: Optional[Callable] = None): + """ + Delete a single entity, optionally performing cleanup beforehand. + + Args: + entity_class (Type[T]): The class used to load and delete the entity. + identifier (str): The unique identifier of the entity. + cleanup_fn (Optional[Callable]): Optional function to perform cleanup before deletion. + """ + entity = self._get_entity(entity_class, identifier) + if cleanup_fn: + cleanup_fn(entity) + entity_class.delete(identifier, self.backend) + + def _delete_all_by_type(self, get_all_fn: Callable, delete_fn: Callable, entity_name: str, **delete_kwargs: Any): + """ + Delete all entities of a specific type using provided getter and deleter functions. + + Args: + get_all_fn (Callable): Function to retrieve all entities. + delete_fn (Callable): Function to delete an individual entity by ID. + entity_name (str): Human-readable name of the entity type, used for logging. + **delete_kwargs (Any): Additional keyword arguments passed to the delete function. + """ + all_entities = get_all_fn() + if all_entities: + for entity in all_entities: + delete_fn(entity.get_id(), **delete_kwargs) + else: + LOG.warning(f"No {entity_name} found in the database.") diff --git a/merlin/db_scripts/entity_managers/logical_worker_manager.py b/merlin/db_scripts/entity_managers/logical_worker_manager.py new file mode 100644 index 000000000..9cebe5f2f --- /dev/null +++ b/merlin/db_scripts/entity_managers/logical_worker_manager.py @@ -0,0 +1,188 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Module for managing logical worker entities. + +This module defines the `LogicalWorkerManager` class, which provides high-level operations +for creating, retrieving, and deleting logical workers stored in the database. It extends +the generic [`EntityManager`][db_scripts.entity_managers.entity_manager.EntityManager] class +with logic specific to logical worker entities, such as ID resolution based on worker name +and queues. + +The manager also integrates cleanup routines to maintain consistency across related entities, +e.g., by removing logical workers from associated runs before deletion. +""" + +from __future__ import annotations + +import logging +from typing import List + +from merlin.db_scripts.data_models import LogicalWorkerModel +from merlin.db_scripts.entities.logical_worker_entity import LogicalWorkerEntity +from merlin.db_scripts.entity_managers.entity_manager import EntityManager +from merlin.exceptions import RunNotFoundError + + +LOG = logging.getLogger("merlin") + +# Purposefully ignoring this pylint message as each entity will have different parameter requirements +# pylint: disable=arguments-differ,arguments-renamed + + +class LogicalWorkerManager(EntityManager[LogicalWorkerEntity, LogicalWorkerModel]): + """ + Manager class for handling logical worker entities. + + This class provides methods to create, retrieve, and delete logical workers from the + backend. It also handles internal ID resolution and cleanup of references to logical + workers from related entities such as runs. + + Attributes: + backend: The backend interface used for storing and retrieving logical workers. + db: Reference to the main database interface, used for cross-entity operations + such as detaching workers from runs. + + Methods: + create: Create a logical worker with the given name and queue list. + get: Retrieve a logical worker either by ID or by name and queues. + get_all: Retrieve all logical workers in the system. + delete: Delete a logical worker and remove it from all associated runs. + delete_all: Delete all logical workers currently stored in the backend. + set_db_reference: Set a reference to the main database object for accessing related entities. + """ + + def _resolve_worker_id(self, worker_id: str = None, worker_name: str = None, queues: List[str] = None) -> str: + """ + Resolve the logical worker ID based on provided parameters. + + Either a `worker_id` must be provided, or both `worker_name` and `queues`. + + Args: + worker_id (str, optional): The ID of the logical worker. + worker_name (str, optional): The name of the logical worker. + queues (List[str], optional): The queues the worker handles. + + Returns: + The resolved logical worker ID. + + Raises: + ValueError: If input arguments are invalid or insufficient. + """ + # Same implementation as in the original class + if worker_id is not None: + if worker_name is not None or queues is not None: + raise ValueError("Provide either `worker_id` or (`worker_name` and `queues`), but not both.") + return worker_id + if worker_name is None or queues is None: + raise ValueError("You must provide either `worker_id` or both `worker_name` and `queues`.") + + return LogicalWorkerModel.generate_id(worker_name, queues) + + def create(self, name: str, queues: List[str]) -> LogicalWorkerEntity: + """ + Create a new logical worker entity, or return it if it already exists. + + Args: + name (str): The name of the logical worker. + queues (List[str]): A list of queue names the worker is assigned to. + + Returns: + The created or existing logical worker entity. + """ + logical_worker_id = self._resolve_worker_id(worker_name=name, queues=queues) + log_message_create = ( + f"Logical worker with name '{name}' and queues '{queues}' does not yet have " + "an entry in the database. Creating one." + ) + log_message_exists = f"Logical worker with name '{name}' and queues '{queues}' already has an entry in the database." + return self._create_entity_if_not_exists( + entity_class=LogicalWorkerEntity, + model_class=LogicalWorkerModel, + identifier=logical_worker_id, + log_message_exists=log_message_exists, + log_message_create=log_message_create, + name=name, + queues=queues, + ) + + def get(self, worker_id: str = None, worker_name: str = None, queues: List[str] = None) -> LogicalWorkerEntity: + """ + Retrieve a logical worker entity by ID, or by name and queues. + + Args: + worker_id (str, optional): The unique identifier of the logical worker. + worker_name (str, optional): The name of the logical worker. + queues (List[str], optional): The queues the worker handles. + + Returns: + The retrieved logical worker entity. + + Raises: + ValueError: If input arguments are invalid or insufficient. + WorkerNotFoundError: If the specified worker does not exist. + """ + worker_id = self._resolve_worker_id(worker_id=worker_id, worker_name=worker_name, queues=queues) + return self._get_entity(LogicalWorkerEntity, worker_id) + + def get_all(self) -> List[LogicalWorkerEntity]: + """ + Retrieve all logical worker entities from the backend. + + Returns: + A list of all logical worker entities. + """ + return self._get_all_entities(LogicalWorkerEntity, "logical_worker") + + def delete(self, worker_id: str = None, worker_name: str = None, queues: List[str] = None): + """ + Delete a logical worker entity and clean up any references from associated runs. + + The method ensures the logical worker is removed from all runs that reference it + before deleting it from the backend. + + Args: + worker_id (str, optional): The ID of the logical worker. + worker_name (str, optional): The name of the logical worker. + queues (List[str], optional): The queues the worker handles. + + Raises: + ValueError: If input arguments are invalid or insufficient. + """ + logical_worker = self.get(worker_id=worker_id, worker_name=worker_name, queues=queues) + + def cleanup_logical_worker(worker): + runs_using_worker = worker.get_runs() + for run_id in runs_using_worker: + try: + run = self.db.runs.get(run_id) + run.remove_worker(worker.get_id()) + except RunNotFoundError: + LOG.warning(f"Couldn't find run with id {run_id}. Continuing with logical worker delete.") + + self._delete_entity(LogicalWorkerEntity, logical_worker.get_id(), cleanup_fn=cleanup_logical_worker) + + def delete_all(self): + """ + Delete all logical worker entities currently stored in the backend. + + Runs cleanup on each logical worker before deletion to remove dependencies. + """ + self._delete_all_by_type(get_all_fn=self.get_all, delete_fn=self.delete, entity_name="logical workers") + + def set_db_reference(self, db: MerlinDatabase): # noqa: F821 pylint: disable=undefined-variable + """ + Set a reference to the main Merlin database object for cross-entity operations. + + This allows the manager to access other entity managers (e.g., runs) when + performing operations like cleanup during deletions. + + Args: + db (db_scripts.merlin_db.MerlinDatabase): The database object that provides + access to related entity managers. + """ + self.db = db diff --git a/merlin/db_scripts/entity_managers/physical_worker_manager.py b/merlin/db_scripts/entity_managers/physical_worker_manager.py new file mode 100644 index 000000000..6ec06d077 --- /dev/null +++ b/merlin/db_scripts/entity_managers/physical_worker_manager.py @@ -0,0 +1,153 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Module for managing database entities related to physical workers. + +This module defines the `PhysicalWorkerManager` class, which provides high-level operations for +creating, retrieving, and deleting physical worker entities stored in the database. It acts as a +controller that encapsulates logic around +[`PhysicalWorkerEntity`][db_scripts.entities.physical_worker_entity.PhysicalWorkerEntity] +objects and their corresponding [`PhysicalWorkerModel`][db_scripts.data_models.PhysicalWorkerModel] +representations. + +The manager interacts with the results backend and optionally references the main database +object to support operations that involve other entities, such as cleanup of related logical workers. +""" + +from __future__ import annotations + +import logging +from typing import Any, List + +from merlin.db_scripts.data_models import PhysicalWorkerModel +from merlin.db_scripts.entities.physical_worker_entity import PhysicalWorkerEntity +from merlin.db_scripts.entity_managers.entity_manager import EntityManager +from merlin.exceptions import WorkerNotFoundError + + +LOG = logging.getLogger("merlin") + +# Purposefully ignoring this pylint message as each entity will have different parameter requirements +# pylint: disable=arguments-differ,arguments-renamed + + +class PhysicalWorkerManager(EntityManager[PhysicalWorkerEntity, PhysicalWorkerModel]): + """ + Manager for physical worker entities. + + This manager handles the creation, retrieval, and deletion of physical worker entities. + It abstracts lower-level backend interactions and optionally performs cleanup logic + that involves related logical workers through a reference to the main + [`MerlinDatabase`][db_scripts.merlin_db.MerlinDatabase]. + + Attributes: + backend (backends.results_backend.ResultsBackend): The backend used for database operations. + db (db_scripts.merlin_db.MerlinDatabase): Optional reference to the main database for cross-entity logic. + + Methods: + create: Create a new physical worker if it does not already exist. + get: Retrieve a physical worker entity by its ID or name. + get_all: Retrieve all physical worker entities from the database. + delete: Delete a specific physical worker, performing cleanup on related logical workers. + delete_all: Delete all physical worker entities from the database. + set_db_reference: Set a reference to the main database object for cross-entity operations. + """ + + def create(self, name: str, **kwargs: Any) -> PhysicalWorkerEntity: + """ + Create a physical worker entity if it does not already exist. + + This method checks whether a physical worker with the specified name exists. + If not, it creates a new one using the provided attributes. + + Args: + name (str): The name of the physical worker. + **kwargs (Any): Additional attributes to pass to the + [`PhysicalWorkerModel`][db_scripts.data_models.PhysicalWorkerModel] constructor. + + Returns: + The created or pre-existing physical worker entity. + """ + log_message_create = f"Physical worker with name '{name}' does not yet have an " "entry in the database. Creating one." + return self._create_entity_if_not_exists( + entity_class=PhysicalWorkerEntity, + model_class=PhysicalWorkerModel, + identifier=name, + log_message_exists=f"Physical worker with name '{name}' already has an entry in the database.", + log_message_create=log_message_create, + name=name, + **kwargs, + ) + + def get(self, worker_id_or_name: str) -> PhysicalWorkerEntity: + """ + Retrieve a physical worker entity by its ID or name. + + Args: + worker_id_or_name (str): The ID or name of the physical worker to retrieve. + + Returns: + The physical worker entity corresponding to the provided identifier. + + Raises: + WorkerNotFoundError: If the specified worker does not exist. + """ + return self._get_entity(PhysicalWorkerEntity, worker_id_or_name) + + def get_all(self) -> List[PhysicalWorkerEntity]: + """ + Retrieve all physical worker entities from the database. + + Returns: + A list of all physical workers stored in the database. + """ + return self._get_all_entities(PhysicalWorkerEntity, "physical_worker") + + def delete(self, worker_id_or_name: str): + """ + Delete a physical worker entity by its ID or name. + + This method will also attempt to remove the deleted physical worker's ID + from its associated logical worker. If the logical worker cannot be found, + a warning is logged and deletion continues. + + Args: + worker_id_or_name (str): The ID or name of the physical worker to delete. + """ + + def cleanup_physical_worker(worker): + logical_worker_id = worker.get_logical_worker_id() + try: + logical_worker = self.db.logical_workers.get(worker_id=logical_worker_id) + logical_worker.remove_physical_worker(worker.get_id()) + except WorkerNotFoundError: + LOG.warning( + f"Couldn't find logical worker with id {logical_worker_id}. Continuing with physical worker delete." + ) + + self._delete_entity(PhysicalWorkerEntity, worker_id_or_name, cleanup_fn=cleanup_physical_worker) + + def delete_all(self): + """ + Delete all physical worker entities from the database. + + This operation also performs cleanup on associated logical workers as needed. + """ + self._delete_all_by_type(get_all_fn=self.get_all, delete_fn=self.delete, entity_name="physical workers") + + def set_db_reference(self, db: MerlinDatabase): # noqa: F821 pylint: disable=undefined-variable + """ + Set a reference to the main Merlin database object for cross-entity operations. + + This allows the manager to access other entity managers (e.g., runs) when + performing operations like cleanup during deletions. + + Args: + db (db_scripts.merlin_db.MerlinDatabase): The database object that provides + access to related entity managers. + """ + self.db = db diff --git a/merlin/db_scripts/entity_managers/run_manager.py b/merlin/db_scripts/entity_managers/run_manager.py new file mode 100644 index 000000000..d22821e96 --- /dev/null +++ b/merlin/db_scripts/entity_managers/run_manager.py @@ -0,0 +1,174 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Module for managing run entities within the Merlin database. + +This module defines the `RunManager` class, which extends the generic +[`EntityManager`][db_scripts.entity_managers.entity_manager.EntityManager] base +class to provide CRUD operations for runs associated with studies, workspaces, +and queues. The manager coordinates with the study and logical worker managers +for consistent data handling during create and delete operations. +""" + +from __future__ import annotations + +import logging +from typing import Any, List + +from merlin.db_scripts.data_models import RunModel +from merlin.db_scripts.entities.run_entity import RunEntity +from merlin.db_scripts.entity_managers.entity_manager import EntityManager +from merlin.exceptions import StudyNotFoundError, WorkerNotFoundError + + +LOG = logging.getLogger("merlin") + +# Purposefully ignoring this pylint message as each entity will have different parameter requirements +# pylint: disable=arguments-differ,arguments-renamed + + +class RunManager(EntityManager[RunEntity, RunModel]): + """ + Manager for run entities. + + This class handles creation, retrieval, updating, and deletion of runs in the database. + It maintains consistency by coordinating with study and logical worker entities during + operations such as run creation and deletion. + + Attributes: + backend: The database backend used for storing run entities. + db: Reference to the main Merlin database, allowing access to other entity managers + such as studies and logical workers. + + Methods: + create: Create a new run associated with a study and workspace. + get: Retrieve a run by its ID or workspace identifier. + get_all: Retrieve all runs from the database. + delete: Delete a run and perform cleanup of related entities. + delete_all: Delete all runs in the database. + set_db_reference: Set the reference to the main Merlin database for cross-entity operations. + """ + + def create(self, study_name: str, workspace: str, queues: List[str], **kwargs: Any) -> RunEntity: + """ + Create a new run associated with a study, workspace, and queues. + + This method ensures the study exists (creating it if necessary), then creates + and saves a new run entity. Additional keyword arguments that correspond to valid + [`RunModel`][db_scripts.data_models.RunModel] fields are included; other kwargs are + stored as additional data. + + Args: + study_name (str): Name of the study this run belongs to. + workspace (str): Workspace identifier for the run. + queues (List[str]): List of queues associated with the run. + **kwargs (Any): Additional optional fields for the run entity. + + Returns: + The created run entity. + """ + # Create the study if it doesn't exist + study_entity = self.db.studies.create(study_name) + + # Filter valid fields for the RunModel + valid_fields = {f.name for f in RunModel.get_class_fields()} + valid_kwargs = {key: val for key, val in kwargs.items() if key in valid_fields} + additional_data = {key: val for key, val in kwargs.items() if key not in valid_fields} + + # Create the RunModel and save it + new_run = RunModel( + study_id=study_entity.get_id(), + workspace=workspace, + queues=queues, + **valid_kwargs, + additional_data=additional_data, + ) + run_entity = RunEntity(new_run, self.backend) + run_entity.save() + + # Add the run ID to the study + study_entity.add_run(run_entity.get_id()) + + return run_entity + + def get(self, run_id_or_workspace: str) -> RunEntity: + """ + Retrieve a run entity by its unique ID or workspace identifier. + + Args: + run_id_or_workspace (str): The unique identifier or workspace string of the run. + + Returns: + The run entity corresponding to the given identifier. + + Raises: + RunNotFoundError: If no run is found matching the identifier. + """ + return self._get_entity(RunEntity, run_id_or_workspace) + + def get_all(self) -> List[RunEntity]: + """ + Retrieve all run entities stored in the database. + + Returns: + A list of all run entities. + """ + return self._get_all_entities(RunEntity, "run") + + def delete(self, run_id_or_workspace: str): + """ + Delete a run entity by its ID or workspace identifier. + + Performs cleanup operations to maintain consistency:\n + - Removes the run from its associated study. + - Removes the run from all logical workers referencing it. + + Args: + run_id_or_workspace (str): The unique identifier or workspace string of the run to delete. + + Raises: + RunNotFoundError: If no run is found matching the identifier. + """ + + def cleanup_run(run): + # Remove from study + try: + study = self.db.studies.get(run.get_study_id()) + study.remove_run(run.get_id()) + except StudyNotFoundError: + LOG.warning(f"Couldn't find study with id {run.get_study_id()}. Continuing with run delete.") + + # Remove from logical workers + for worker_id in run.get_workers(): + try: + logical_worker = self.db.logical_workers.get(worker_id=worker_id) + logical_worker.remove_run(run.get_id()) + except WorkerNotFoundError: + LOG.warning(f"Couldn't find logical worker with id {worker_id}. Continuing with run delete.") + + self._delete_entity(RunEntity, run_id_or_workspace, cleanup_fn=cleanup_run) + + def delete_all(self): + """ + Delete all run entities from the database. + + This method calls `delete` on each run entity to ensure proper cleanup. + """ + self._delete_all_by_type(get_all_fn=self.get_all, delete_fn=self.delete, entity_name="runs") + + def set_db_reference(self, db: MerlinDatabase): # noqa: F821 pylint: disable=undefined-variable + """ + Set a reference to the main Merlin database object for cross-entity operations. + + This allows the manager to access other entity managers (e.g., logical workers) when + performing operations like cleanup during deletions. + + Args: + db (db_scripts.merlin_db.MerlinDatabase): The database object that provides + access to related entity managers. + """ + self.db = db diff --git a/merlin/db_scripts/entity_managers/study_manager.py b/merlin/db_scripts/entity_managers/study_manager.py new file mode 100644 index 000000000..45765baf6 --- /dev/null +++ b/merlin/db_scripts/entity_managers/study_manager.py @@ -0,0 +1,143 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +`StudyManager` module for managing study entities in the Merlin database. + +This module defines a manager class responsible for the creation, retrieval, +and deletion of studies. It also ensures appropriate cleanup of associated +run entities when a study is deleted. +""" + +from __future__ import annotations + +from typing import List + +from merlin.db_scripts.data_models import StudyModel +from merlin.db_scripts.entities.study_entity import StudyEntity +from merlin.db_scripts.entity_managers.entity_manager import EntityManager + + +# Purposefully ignoring this pylint message as each entity will have different parameter requirements +# pylint: disable=arguments-differ,arguments-renamed + + +class StudyManager(EntityManager[StudyEntity, StudyModel]): + """ + Manager class for handling study entities. + + The `StudyManager` interacts with the underlying storage backend to create, + retrieve, and delete study records. It also supports cleanup of related + run entities during deletion to ensure data integrity. + + Attributes: + backend (backends.results_backend.ResultsBackend): Backend interface used to persist + and query study data. + db (db_scripts.merlin_db.MerlinDatabase): Reference to the full `MerlinDatabase`, + used for cross-entity operations (e.g., deleting associated runs). + + Methods: + create: Create a new study if it doesn't already exist. + get: Retrieve a study by ID or name. + get_all: Retrieve all study entities. + delete: Delete a study, with optional cleanup of related runs. + delete_all: Delete all studies and optionally their runs. + set_db_reference: Set reference to the MerlinDatabase for cross-entity access. + """ + + def create(self, study_name: str) -> StudyEntity: + """ + Create a study if it does not already exist. + + If a study with the given name is not found in the database, + a new study entity is created and persisted. + + Args: + study_name (str): The name of the study to create. + + Returns: + The newly created or existing study entity. + """ + return self._create_entity_if_not_exists( + entity_class=StudyEntity, + model_class=StudyModel, + identifier=study_name, + log_message_exists=f"Study with name '{study_name}' already has an entry in the database.", + log_message_create=f"Study with name '{study_name}' does not yet have an entry in the database. Creating one.", + name=study_name, + ) + + def get(self, study_id_or_name: str) -> StudyEntity: + """ + Retrieve a study by its ID or name. + + Args: + study_id_or_name (str): The unique study ID or name to look up. + + Returns: + The corresponding study entity. + + Raises: + StudyNotFoundError: If no study matches the given ID or name. + """ + return self._get_entity(StudyEntity, study_id_or_name) + + def get_all(self) -> List[StudyEntity]: + """ + Retrieve all study entities stored in the database. + + Returns: + A list of all available study entities. + """ + return self._get_all_entities(StudyEntity, "study") + + def delete(self, study_id_or_name: str, remove_associated_runs: bool = True): + """ + Delete a study and optionally its associated runs. + + If `remove_associated_runs` is True, all runs linked to the study + will be deleted before the study itself is removed. + + Args: + study_id_or_name (str): The ID or name of the study to delete. + remove_associated_runs (bool, optional): Whether to delete runs + associated with the study. Defaults to True. + """ + + def cleanup_study(study): + if remove_associated_runs: + for run_id in study.get_runs(): + self.db.runs.delete(run_id) + + self._delete_entity(StudyEntity, study_id_or_name, cleanup_fn=cleanup_study) + + def delete_all(self, remove_associated_runs: bool = True): + """ + Delete all studies in the database, and optionally their associated runs. + + Args: + remove_associated_runs (bool, optional): Whether to delete runs + linked to each study. Defaults to True. + """ + self._delete_all_by_type( + get_all_fn=self.get_all, + delete_fn=self.delete, + entity_name="studies", + remove_associated_runs=remove_associated_runs, + ) + + def set_db_reference(self, db: MerlinDatabase): # noqa: F821 pylint: disable=undefined-variable + """ + Set a reference to the main Merlin database object for cross-entity operations. + + This allows the manager to access other entity managers (e.g., runs) when + performing operations like cleanup during deletions. + + Args: + db (db_scripts.merlin_db.MerlinDatabase): The database object that provides + access to related entity managers. + """ + self.db = db diff --git a/merlin/db_scripts/merlin_db.py b/merlin/db_scripts/merlin_db.py new file mode 100644 index 000000000..eedbd05d2 --- /dev/null +++ b/merlin/db_scripts/merlin_db.py @@ -0,0 +1,267 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module contains the functionality necessary to interact with everything +stored in Merlin's database. +""" + +import logging +from typing import Any, Dict, List + +from merlin.backends.backend_factory import backend_factory +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.entity_managers.entity_manager import EntityManager +from merlin.db_scripts.entity_managers.logical_worker_manager import LogicalWorkerManager +from merlin.db_scripts.entity_managers.physical_worker_manager import PhysicalWorkerManager +from merlin.db_scripts.entity_managers.run_manager import RunManager +from merlin.db_scripts.entity_managers.study_manager import StudyManager +from merlin.exceptions import EntityManagerNotSupportedError + + +LOG = logging.getLogger("merlin") + + +class MerlinDatabase: + """ + High-level interface for accessing Merlin database entities. + + This class provides a unified interface to all entity managers in Merlin. + + Attributes: + backend (backends.results_backend.ResultsBackend): A `ResultsBackend` instance. + logical_workers (db_scripts.entity_managers.logical_worker_manager.LogicalWorkerManager): + A `LogicalWorkerManager` instance. + physical_workers (db_scripts.entity_managers.physical_worker_manager.PhysicalWorkerManager): + A `PhysicalWorkerManager` instance. + runs (db_scripts.entity_managers.run_manager.RunManager): A `RunManager` instance. + studies (db_scripts.entity_managers.study_manager.StudyManager): A `StudyManager` instance. + + Methods: + get_db_type: Retrieve the type of the backend being used (e.g., Redis, SQL). + get_db_version: Retrieve the version of the backend. + get_connection_string: Retrieve the backend connection string. + create: Create a new entity of the specified type. + get: Get an entity by type and identifier. + get_all: Get all entities of a specific type. + delete: Delete an entity by type and identifier. + delete_all: Delete all entities of a specific type. + get_everything: Get all entities from all entity managers. + delete_everything: Delete all entities from all entity managers. + """ + + def __init__(self): + """ + Initialize a new MerlinDatabase instance. + """ + from merlin.config.configfile import CONFIG # pylint: disable=import-outside-toplevel + + self.backend: ResultsBackend = backend_factory.get_backend(CONFIG.results_backend.name.lower()) + self._entity_managers: Dict[str, EntityManager] = { + "study": StudyManager(self.backend), + "run": RunManager(self.backend), + "logical_worker": LogicalWorkerManager(self.backend), + "physical_worker": PhysicalWorkerManager(self.backend), + } + + # Set up cross-references for managers that need them + for manager in self._entity_managers.values(): + if hasattr(manager, "set_db_reference"): + manager.set_db_reference(self) + + # Provide direct access to entity managers for convenience + @property + def studies(self) -> StudyManager: + """ + Get the study manager. + + Returns: + A [`StudyManager`][db_scripts.entity_managers.study_manager.StudyManager] + instance. + """ + return self._entity_managers["study"] + + @property + def runs(self) -> RunManager: + """ + Get the run manager. + + Returns: + A [`RunManager`][db_scripts.entity_managers.run_manager.RunManager] + instance. + """ + return self._entity_managers["run"] + + @property + def logical_workers(self) -> LogicalWorkerManager: + """ + Get the logical worker manager. + + Returns: + A [`LogicalWorkerManager`][db_scripts.entity_managers.logical_worker_manager.LogicalWorkerManager] + instance. + """ + return self._entity_managers["logical_worker"] + + @property + def physical_workers(self) -> PhysicalWorkerManager: + """ + Get the physical worker manager. + + Returns: + A [`PhysicalWorkerManager`][db_scripts.entity_managers.physical_worker_manager.PhysicalWorkerManager] + instance. + """ + return self._entity_managers["physical_worker"] + + def get_db_type(self) -> str: + """ + Retrieve the type of backend. + + Returns: + The type of backend (e.g. redis, sql, etc.). + """ + return self.backend.get_name() + + def get_db_version(self) -> str: + """ + Get the version of the backend. + + Returns: + The version number of the backend. + """ + return self.backend.get_version() + + def get_connection_string(self) -> str: + """ + Get the connection string to the backend. + + Returns: + The connection string to the backend. + """ + return self.backend.get_connection_string() + + def _validate_entity_type(self, entity_type: str): + """ + Check to make sure the entity type passed in is supported. + + Args: + entity_type: The type of entity to validate (study, run, logical_worker, physical_worker). + """ + if entity_type not in self._entity_managers: + raise EntityManagerNotSupportedError(f"Entity type not supported: {entity_type}") + + def create(self, entity_type: str, *args, **kwargs) -> Any: + """ + Create a new entity of the specified type. + + Args: + entity_type: The type of entity to create (study, run, logical_worker, physical_worker). + + Returns: + The created entity. + + Raises: + EntityManagerNotSupportedError: If the entity type is not supported. + """ + self._validate_entity_type(entity_type) + return self._entity_managers[entity_type].create(*args, **kwargs) + + def get(self, entity_type: str, *args, **kwargs) -> Any: + """ + Get an entity by type and identifier. + + Args: + entity_type: The type of entity to get (study, run, logical_worker, physical_worker). + + Returns: + The requested entity. + + Raises: + EntityManagerNotSupportedError: If the entity type is not supported. + """ + self._validate_entity_type(entity_type) + return self._entity_managers[entity_type].get(*args, **kwargs) + + def get_all(self, entity_type: str) -> List[Any]: + """ + Get all entities of a specific type. + + Args: + entity_type: The type of entities to get (study, run, logical_worker, physical_worker). + + Returns: + A list of all entities of the specified type. + + Raises: + EntityManagerNotSupportedError: If the entity type is not supported. + """ + self._validate_entity_type(entity_type) + return self._entity_managers[entity_type].get_all() + + def delete(self, entity_type: str, *args, **kwargs) -> None: + """ + Delete an entity by type and identifier. + + Args: + entity_type: The type of entity to delete (study, run, logical_worker, physical_worker). + + Raises: + EntityManagerNotSupportedError: If the entity type is not supported. + """ + self._validate_entity_type(entity_type) + self._entity_managers[entity_type].delete(*args, **kwargs) + + def delete_all(self, entity_type: str, **kwargs) -> None: + """ + Delete all entities of a specific type. + + Args: + entity_type: The type of entities to delete (study, run, logical_worker, physical_worker). + + Raises: + EntityManagerNotSupportedError: If the entity type is not supported. + """ + self._validate_entity_type(entity_type) + self._entity_managers[entity_type].delete_all(**kwargs) + + def get_everything(self) -> List[Any]: + """ + Get all entities from all entity managers. + + Returns: + A dictionary mapping entity types to lists of entities. + """ + result = [] + for manager in self._entity_managers.values(): + result.extend(manager.get_all()) + return result + + def delete_everything(self, force: bool = False) -> None: + """ + Delete all entities from all entity managers. + + This method deletes studies last to ensure proper cleanup of dependencies. + """ + flush_database = False + if force: + flush_database = True + else: + # Ask the user for confirmation + valid_inputs = ["y", "n"] + user_input = input("Are you sure you want to flush the entire database? (y/n): ").strip().lower() + while user_input not in valid_inputs: + user_input = input("Invalid input. Use 'y' for 'yes' or 'n' for 'no': ").strip().lower() + + if user_input == "y": + flush_database = True + + if flush_database: + LOG.info("Flushing the database...") + self.backend.flush_database() + LOG.info("Database successfully flushed.") + else: + LOG.info("Database flush cancelled.") diff --git a/merlin/display.py b/merlin/display.py index 65818ce05..f96227d9a 100644 --- a/merlin/display.py +++ b/merlin/display.py @@ -1,32 +1,8 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ Manages formatting for displaying information to the console. @@ -37,11 +13,13 @@ import shutil import time import traceback +from argparse import Namespace from datetime import datetime from multiprocessing import Pipe, Process -from typing import Dict +from multiprocessing.connection import Connection +from typing import Any, Dict, List, Union -from kombu import Connection +from kombu import Connection as KombuConnection from tabulate import tabulate from merlin.ascii_art import banner_small @@ -72,16 +50,39 @@ class ConnProcess(Process): """ - An extension of Multiprocessing's Process class in order - to overwrite the run and exception defintions. + An extension of the multiprocessing's Process class that allows for + custom handling of exceptions and inter-process communication. + + This class overrides the `run` method to capture exceptions that occur + during the execution of the process and sends them back to the parent + process via a pipe. It also provides a property to retrieve any + exceptions that were raised during execution. + + Attributes: + _pconn: The parent connection for inter-process communication. + _cconn: The child connection for inter-process communication. + exception: Stores the exception raised during the process run. + + Methods: + run: Executes the process's main logic. """ def __init__(self, *args, **kwargs): Process.__init__(self, *args, **kwargs) + self._pconn: Connection + self._cconn: Connection self._pconn, self._cconn = Pipe() self._exception = None def run(self): + """ + Executes the process's main logic. + + This method overrides the default run method of the Process class. + It attempts to run the process and captures any exceptions that occur. + If an exception is raised, it sends the exception and its traceback + back to the parent process via the child connection. + """ try: Process.run(self) self._cconn.send(None) @@ -91,17 +92,35 @@ def run(self): # raise e # You can still rise this exception if you need to @property - def exception(self): - """Create custom exception""" + def exception(self) -> Union[Exception, None]: + """ + Retrieves the exception raised during the process execution. + + This property checks if there is an exception available from the + parent connection. If an exception was raised, it is received and + stored for later access. + + Returns: + The exception raised during the process run, or None if no exception occurred. + """ if self._pconn.poll(): self._exception = self._pconn.recv() return self._exception -def check_server_access(sconf): +def check_server_access(sconf: Dict[str, Any]): """ Check if there are any issues connecting to the servers. If there are, output the errors. + + This function iterates through a predefined list of servers and checks + their connectivity based on the provided server configuration. If any + connection issues are detected, the exceptions are collected and printed. + + Args: + sconf: A dictionary containing server configurations, where keys + represent server names and values contain connection details. + The function expects keys corresponding to the servers being checked. """ servers = ["broker server", "results server"] @@ -120,7 +139,24 @@ def check_server_access(sconf): print(f"{key}: {val}") -def _examine_connection(server, sconf, excpts): +def _examine_connection(server: str, sconf: Dict[str, Any], excpts: Dict[str, Exception]): + """ + Examine the connection to a specified server and handle any exceptions. + + This function attempts to establish a connection to the given server using + the configuration provided in `sconf`. It utilizes a separate process to + manage the connection attempt and checks for timeouts. If the connection + fails or times out, the error is recorded in the `excpts` dictionary. + + Args: + server: A string representing the name of the server to connect to. + This should correspond to a key in the `sconf` dictionary. + sconf: A dictionary containing server configurations, where keys + represent server names and values contain connection details. + excpts: A dictionary to store exceptions encountered during the + connection attempt, with server names as keys and exceptions + as values. + """ from merlin.config import broker, results_backend # pylint: disable=C0415 connect_timeout = 60 @@ -130,7 +166,7 @@ def _examine_connection(server, sconf, excpts): ssl_conf = broker.get_ssl_config() if "results" in server: ssl_conf = results_backend.get_ssl_config() - conn = Connection(sconf[server], ssl=ssl_conf) + conn = KombuConnection(sconf[server], ssl=ssl_conf) conn_check = ConnProcess(target=conn.connect) conn_check.start() counter = 0 @@ -153,7 +189,11 @@ def _examine_connection(server, sconf, excpts): def display_config_info(): """ - Prints useful configuration information to the console. + Prints useful configuration information for the Merlin application to the console. + + This function retrieves and displays the connection strings and SSL configurations + for the broker and results servers. It handles any exceptions that may occur during + the retrieval process, providing error messages for any issues encountered. """ from merlin.config import broker, results_backend # pylint: disable=C0415 from merlin.config.configfile import default_config_info # pylint: disable=C0415 @@ -191,12 +231,13 @@ def display_config_info(): check_server_access(sconf) -def display_multiple_configs(files, configs): +def display_multiple_configs(files: List[str], configs: List[Dict]): """ Logic for displaying multiple Merlin config files. - :param `files`: List of merlin config files - :param `configs`: List of merlin configurations + Args: + files: List of merlin config files + configs: List of merlin configurations """ print("=" * 50) print(" MERLIN CONFIG ") @@ -211,13 +252,19 @@ def display_multiple_configs(files, configs): # Might use args here in the future so we'll disable the pylint warning for now -def print_info(args): # pylint: disable=W0613 +def print_info(args: Namespace): # pylint: disable=W0613 """ Provide version and location information about python and packages to facilitate user troubleshooting. Also provides info about server connections and configurations. - :param `args`: parsed CLI arguments + Note: + The `args` parameter is currently unused but is included for + compatibility with the command-line interface (CLI) in case we decide to use + args here in the future. + + Args: + args: parsed CLI arguments (currently unused). """ print(banner_small) display_config_info() @@ -235,16 +282,30 @@ def print_info(args): # pylint: disable=W0613 def display_status_task_by_task(status_obj: "DetailedStatus", test_mode: bool = False): # noqa: F821 """ - Displays a low level overview of the status of a study. This is a task-by-task - status display where each task will show: - step name, worker name, task queue, cmd & restart parameters, - step workspace, step status, return code, elapsed time, run time, and num restarts. - If too many tasks are found and the pager is disabled, prompts will appear for the user to decide - what to do that way we don't overload the terminal (unless the no-prompts flag is provided). - - :param `status_obj`: A DetailedStatus object - :param `test_mode`: If true, run this in testing mode and don't print any output. This will also - decrease the limit on the number of tasks allowed before a prompt is displayed. + Displays a low-level overview of the status of a study in a task-by-task format. + + Each task will display the following details: + - Step name + - Worker name + - Task queue + - Command and restart parameters + - Step workspace + - Step status + - Return code + - Elapsed time + - Run time + - Number of restarts + + If the number of tasks exceeds a certain limit and the pager is disabled, the user + will be prompted to apply additional filters to avoid overwhelming the terminal output, + unless the prompts are disabled through the no-prompts flag. + + Args: + status_obj (study.status.DetailedStatus): An instance of + [`DetailedStatus`][study.status.DetailedStatus] containing information about + the current state of tasks. + test_mode: If True, runs the function in testing mode, suppressing output and + reducing the task limit for prompts. Defaults to False. """ args = status_obj.args try: @@ -299,10 +360,18 @@ def display_status_task_by_task(status_obj: "DetailedStatus", test_mode: bool = def _display_summary(state_info: Dict[str, str], cb_help: bool): """ - Given a dict of state info for a step, print a summary of the task states. - - :param `state_info`: A dictionary of information related to task states for a step - :param `cb_help`: True if colorblind assistance (using symbols) is needed. False otherwise. + Prints a summary of task states based on the provided state information. + + This function takes a dictionary of state information for a step and + prints a formatted summary, including optional colorblind assistance using + symbols if specified. + + Args: + state_info: A dictionary containing information related to task states + for a step. Each entry should correspond to a specific task state + with its associated properties (e.g., count, total, name). + cb_help: If True, provides colorblind assistance by using symbols in the + display. Defaults to False for standard output. """ # Build a summary list of task info print("\nSUMMARY:") @@ -338,18 +407,27 @@ def _display_summary(state_info: Dict[str, str], cb_help: bool): def display_status_summary( # pylint: disable=R0912 - status_obj: "Status", non_workspace_keys: set, test_mode=False # noqa: F821 + status_obj: "Status", non_workspace_keys: set, test_mode: bool = False # noqa: F821 ) -> Dict: """ - Displays a high level overview of the status of a study. This includes - progress bars for each step and a summary of the number of initialized, - running, finished, cancelled, dry ran, failed, and unknown tasks. - - :param `status_obj`: A Status object - :param `non_workspace_keys`: A set of keys in requested_statuses that are not workspace keys. - This will be set("parameters", "task_queue", "workers") - :param `test_mode`: If True, don't print anything and just return a dict of all the state info for each step - :returns: A dict that's empty usually. If ran in test_mode it will be a dict of state_info for every step. + Displays a high-level overview of the status of a study, including progress bars for each step + and a summary of the number of initialized, running, finished, cancelled, dry ran, failed, and + unknown tasks. + + The function prints a summary for each step and collects state information. In test mode, + it suppresses output and returns a dictionary of state information instead. + + Args: + status_obj (study.status.Status): An instance of [`Status`][study.status.Status] containing + information about task states and associated data for the study. + non_workspace_keys: A set of keys in requested_statuses that are not workspace keys. + Typically includes keys like "parameters", "task_queue", and "workers". + test_mode: If True, runs in test mode; suppresses printing and returns a dictionary + of state information for each step. Defaults to False. + + Returns: + An empty dictionary in regular mode. In test mode, returns a dictionary containing + the state information for each step. """ all_state_info = {} if not test_mode: @@ -429,32 +507,40 @@ def display_status_summary( # pylint: disable=R0912 # Credit to this stack overflow post: https://stackoverflow.com/a/34325723 def display_progress_bar( # pylint: disable=R0913,R0914 - current, - total, - state_info=None, - prefix="", - suffix="", - decimals=1, - length=80, - fill="█", - print_end="\n", - color=None, - cb_help=False, + current: int, + total: int, + state_info: Dict[str, Any] = None, + prefix: str = "", + suffix: str = "", + decimals: int = 1, + length: int = 80, + fill: str = "█", + print_end: str = "\n", + color: str = None, + cb_help: bool = False, ): """ - Prints a progress bar based on current and total. - - :param `current`: current number (Int) - :param `total`: total number (Int) - :param `state_info`: information about the state of tasks (Dict) (overrides color) - :param `prefix`: prefix string (Str) - :param `suffix`: suffix string (Str) - :param `decimals`: positive number of decimals in percent complete (Int) - :param `length`: character length of bar (Int) - :param `fill`: bar fill character (Str) - :param `print_end`: end character (e.g. "\r", "\r\n") (Str) - :param `color`: color of the progress bar (ANSI Str) (overridden by state_info) - :param `cb_help`: true if color blind help is needed; false otherwise (Bool) + Prints a customizable progress bar that visually represents the completion percentage + relative to a given total. + + The function can display additional state information for detailed tracking, including + support for color customization and adaptation for color-blind users. It updates the + display based on current progress and optionally accepts state information to adjust the + appearance of the progress bar. + + Args: + current: Current progress value. + total: Total value representing 100% completion. + state_info: Dictionary containing state information about tasks. This can override + color settings and modifies how the progress bar is displayed. + prefix: Optional prefix string to display before the progress bar. + suffix: Optional suffix string to display after the progress bar. + decimals: Number of decimal places to display in the percentage (default is 1). + length: Character length of the progress bar (default is 80). + fill: Character used to fill the progress bar (default is "█"). + print_end: Character(s) to print at the end of the line (e.g., '\\r', '\\n'). + color: ANSI color string for the progress bar. Overrides state_info colors. + cb_help: If True, provides color-blind assistance by adapting the fill characters. """ # Set the color of the bar if color and color in ANSI_COLORS: diff --git a/merlin/examples/__init__.py b/merlin/examples/__init__.py index 37cabcad1..3d75e38f3 100644 --- a/merlin/examples/__init__.py +++ b/merlin/examples/__init__.py @@ -1,29 +1,14 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +The `examples` package provides resources for learning and setting up Merlin workflows. + +Modules: + examples.py: Contains example specification files for Merlin workflows, along with + detailed explanations of each block in the specification. + generator.py: Provides utilities for managing and generating example workflows. +""" diff --git a/merlin/examples/dev_workflows/multiple_workers.yaml b/merlin/examples/dev_workflows/multiple_workers.yaml index 8785d9e9a..967582a53 100644 --- a/merlin/examples/dev_workflows/multiple_workers.yaml +++ b/merlin/examples/dev_workflows/multiple_workers.yaml @@ -46,11 +46,11 @@ merlin: resources: workers: step_1_merlin_test_worker: - args: -l INFO + args: -l INFO --concurrency 1 steps: [step_1] step_2_merlin_test_worker: - args: -l INFO + args: -l INFO --concurrency 1 steps: [step_2] other_merlin_test_worker: - args: -l INFO + args: -l INFO --concurrency 1 steps: [step_3, step_4] diff --git a/merlin/examples/examples.py b/merlin/examples/examples.py index 1b88783ff..308685d03 100644 --- a/merlin/examples/examples.py +++ b/merlin/examples/examples.py @@ -1,33 +1,16 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### -"""This module contains example spec files with explanations of each block""" +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module contains example specification files for Merlin workflows, +along with detailed explanations of each block in the specification. + +The examples and templates are useful for understanding how to structure +Merlin workflows, define tasks, manage parameters, and configure resources. +""" # Taken from https://lc.llnl.gov/mlsi/docs/merlin/merlin_config.html TEMPLATE_FILE_CONTENTS = """ diff --git a/merlin/examples/generator.py b/merlin/examples/generator.py index bdb764b30..ffdaf20ca 100644 --- a/merlin/examples/generator.py +++ b/merlin/examples/generator.py @@ -1,44 +1,21 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ This module contains a list of examples that can be used when learning to use Merlin, or for setting up new workflows. Examples are packaged in directories, with the directory name denoting -the example name. This must match the name of the merlin specification inside. +the example name. This must match the name of the Merlin specification inside. """ import glob import logging import os import shutil +from typing import Dict, List, Union import tabulate import yaml @@ -57,25 +34,37 @@ # - all openfoam examples should just be under one openfoam label -def gather_example_dirs(): - """Get all the example directories""" +def gather_example_dirs() -> Dict[str, str]: + """ + Get all the example directories. + + Returns: + A dictionary where the keys and values are the names of example directories. + """ result = {} for directory in sorted(os.listdir(EXAMPLES_DIR)): result[directory] = directory return result -def gather_all_examples(): - """Get all the example yaml files""" +def gather_all_examples() -> List[str]: + """ + Get all the example YAML files. + + Returns: + A list of file paths to all YAML files in the example directories. + """ path = os.path.join(os.path.join(EXAMPLES_DIR, ""), os.path.join("*", "*.yaml")) return glob.glob(path) -def write_example(src_path, dst_path): +def write_example(src_path: str, dst_path: str): """ - Write out the example workflow to a file. - :param src_path: The path to copy from. - :param content: The formatted content to write the file to. + Write out the example workflow to a file or directory. + + Args: + src_path: The path to copy the example from. + dst_path: The destination path to copy the example to. """ if os.path.isdir(src_path): shutil.copytree(src_path, dst_path) @@ -83,8 +72,13 @@ def write_example(src_path, dst_path): shutil.copy(src_path, dst_path) -def list_examples(): - """List all available examples.""" +def list_examples() -> str: + """ + List all available examples with their descriptions. + + Returns: + A formatted string table of example names and their descriptions. + """ headers = ["name", "description"] rows = [] for example_dir in gather_example_dirs(): @@ -111,8 +105,17 @@ def list_examples(): return "\n" + tabulate.tabulate(rows, headers) + "\n" -def setup_example(name, outdir): - """Setup the given example.""" +def setup_example(name: str, outdir: str) -> Union[str, None]: + """ + Set up the given example by copying it to the specified output directory. + + Args: + name: The name of the example to set up. + outdir: The output directory where the example will be copied. + + Returns: + The name of the example if successful, or None if the example was not found or an error occurred. + """ example = None spec_paths = gather_all_examples() spec_path = None diff --git a/merlin/examples/workflows/feature_demo/requirements.txt b/merlin/examples/workflows/feature_demo/requirements.txt index e308e1895..3eee4d90c 100644 --- a/merlin/examples/workflows/feature_demo/requirements.txt +++ b/merlin/examples/workflows/feature_demo/requirements.txt @@ -1,2 +1,3 @@ scikit-learn -merlin-spellbook +merlin-spellbook; python_version < "3.12" +merlin-spellbook>=0.9.0; python_version >= "3.12" diff --git a/merlin/examples/workflows/optimization/requirements.txt b/merlin/examples/workflows/optimization/requirements.txt index a487c9f39..378678b1e 100644 --- a/merlin/examples/workflows/optimization/requirements.txt +++ b/merlin/examples/workflows/optimization/requirements.txt @@ -1,4 +1,5 @@ -merlin-spellbook +merlin-spellbook; python_version < "3.12" +merlin-spellbook>=0.9.0; python_version >= "3.12" numpy scikit-learn matplotlib diff --git a/merlin/examples/workflows/remote_feature_demo/requirements.txt b/merlin/examples/workflows/remote_feature_demo/requirements.txt index e308e1895..3eee4d90c 100644 --- a/merlin/examples/workflows/remote_feature_demo/requirements.txt +++ b/merlin/examples/workflows/remote_feature_demo/requirements.txt @@ -1,2 +1,3 @@ scikit-learn -merlin-spellbook +merlin-spellbook; python_version < "3.12" +merlin-spellbook>=0.9.0; python_version >= "3.12" diff --git a/merlin/exceptions/__init__.py b/merlin/exceptions/__init__.py index 158c6f073..8262d0f94 100644 --- a/merlin/exceptions/__init__.py +++ b/merlin/exceptions/__init__.py @@ -1,32 +1,8 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ Module of all Merlin-specific exception types. @@ -78,7 +54,7 @@ def __init__(self): class InvalidChainException(Exception): """ - Exception for invalid Merlin step DAGs. + Exception for invalid Merlin step Directed Acyclic Graphs (DAGs). """ def __init__(self): @@ -91,8 +67,8 @@ class RestartException(Exception): the restart command if present , else retry. """ - def __init__(self): - super().__init__() + def __init__(self, message): + super().__init__(message) class NoWorkersException(Exception): @@ -103,3 +79,65 @@ class NoWorkersException(Exception): def __init__(self, message): super().__init__(message) + + +class MerlinInvalidTaskServerError(Exception): + """ + Exception to signal that an invalid task server was provided. + """ + + def __init__(self, message): + super().__init__(message) + + +class BackendNotSupportedError(Exception): + """ + Exception to signal that the provided backend is not supported by Merlin. + """ + + def __init__(self, message): + super().__init__(message) + + +############################### +# Database-Related Exceptions # +############################### + + +class EntityNotFoundError(Exception): + """ + Fallback error for entities that can't be found. + """ + + +class StudyNotFoundError(EntityNotFoundError): + """ + Exception to signal that the study you were looking for cannot be found in + Merlin's database. + """ + + +class RunNotFoundError(EntityNotFoundError): + """ + Exception to signal that the run you were looking for cannot be found in + Merlin's database. + """ + + +class WorkerNotFoundError(EntityNotFoundError): + """ + Exception to signal that the worker you were looking for cannot be found in + Merlin's database. + """ + + +class UnsupportedDataModelError(Exception): + """ + Exception to signal that the data model you're trying to use is not supported. + """ + + +class EntityManagerNotSupportedError(Exception): + """ + Exception to signal that the provided entity manager is not supported by Merlin. + """ diff --git a/merlin/log_formatter.py b/merlin/log_formatter.py index 55f5f37f8..e88b418e1 100644 --- a/merlin/log_formatter.py +++ b/merlin/log_formatter.py @@ -1,34 +1,10 @@ -"""This module handles setting up the extensive logging system in Merlin.""" +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +"""This module handles setting up the extensive logging system in Merlin.""" import logging import sys @@ -43,12 +19,14 @@ } -def setup_logging(logger, log_level="INFO", colors=True): +def setup_logging(logger: logging.Logger, log_level: str = "INFO", colors: bool = True): """ Setup and configure Python logging. - :param `logger`: a logging.Logger object - :param `log_level`: logger level + Args: + logger: A logging.Logger object. + log_level: Logger level. + colors: If True use colored logs. """ formatter = logging.Formatter() handler = logging.StreamHandler(sys.stdout) diff --git a/merlin/main.py b/merlin/main.py index 6ca79e86d..6cde71339 100644 --- a/merlin/main.py +++ b/merlin/main.py @@ -1,34 +1,10 @@ -"""The top level main function for invoking Merlin.""" +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +"""The top level main function for invoking Merlin.""" from __future__ import print_function @@ -39,21 +15,29 @@ import time import traceback from argparse import ( + SUPPRESS, ArgumentDefaultsHelpFormatter, ArgumentParser, + ArgumentTypeError, Namespace, RawDescriptionHelpFormatter, RawTextHelpFormatter, ) from contextlib import suppress -from typing import Dict, List, Optional, Union +from typing import Dict, List, Optional, Tuple, Union +import yaml from tabulate import tabulate from merlin import VERSION, router from merlin.ascii_art import banner_small +from merlin.config.configfile import initialize_config +from merlin.config.merlin_config_manager import MerlinConfigManager +from merlin.db_scripts.db_commands import database_delete, database_get, database_info +from merlin.db_scripts.merlin_db import MerlinDatabase from merlin.examples.generator import list_examples, setup_example from merlin.log_formatter import setup_logging +from merlin.monitor.monitor import Monitor from merlin.server.server_commands import config_server, init_server, restart_server, start_server, status_server, stop_server from merlin.spec.expansion import RESERVED, get_spec_with_expansion from merlin.spec.specification import MerlinSpec @@ -69,10 +53,21 @@ class HelpParser(ArgumentParser): - """This class overrides the error message of the argument parser to - print the help message when an error happens.""" + """ + This class overrides the error message of the argument parser to + print the help message when an error happens. + + Methods: + error: Override the error message of the `ArgumentParser` class. + """ + + def error(self, message: str): + """ + Override the error message of the `ArgumentParser` class. - def error(self, message): + Args: + message: The error message to log. + """ sys.stderr.write(f"error: {message}\n") self.print_help() sys.exit(2) @@ -82,13 +77,27 @@ def parse_override_vars( variables_list: Optional[List[str]], ) -> Optional[Dict[str, Union[str, int]]]: """ - Parse a list of variables from command line syntax - into a valid dictionary of variable keys and values. - - :param [List[str]] `variables_list`: an optional list of strings, e.g. ["KEY=val",...] - - :return: returns either None or a Dict keyed with strs, linked to strs and ints. - :rtype: Dict + Parse a list of command-line variables into a dictionary of key-value pairs. + + This function takes an optional list of strings following the syntax + "KEY=val" and converts them into a dictionary. It validates the format + of the variables and ensures that keys are valid according to specified rules. + + Args: + variables_list: An optional list of strings, where each string should be in the + format "KEY=val", e.g., ["KEY1=value1", "KEY2=42"]. + + Returns: + A dictionary where the keys are variable names (str) and the + values are either strings or integers. If `variables_list` is + None or empty, returns None. + + Raises: + ValueError: If the input format is incorrect, including:\n + - Missing '=' operator. + - Excess '=' operators in a variable assignment. + - Invalid variable names (must be alphanumeric and underscores). + - Attempting to override reserved variable names. """ if variables_list is None: return None @@ -121,11 +130,26 @@ def parse_override_vars( return result -def get_merlin_spec_with_override(args): +def get_merlin_spec_with_override(args: Namespace) -> Tuple[MerlinSpec, str]: """ - Shared command to return the spec object. - - :param 'args': parsed CLI arguments + Shared command to retrieve a [`MerlinSpec`][spec.specification.MerlinSpec] object + and an expanded filepath. + + This function processes parsed command-line interface (CLI) arguments to validate + and expand the specified filepath and any associated variables. It then constructs + and returns a [`MerlinSpec`][spec.specification.MerlinSpec] object based on the + provided specification. + + Args: + args: Parsed CLI arguments containing:\n + - `specification`: the path to the specification file + - `variables`: optional variable overrides to customize the spec. + + Returns: + spec (spec.specification.MerlinSpec): An instance of the + [`MerlinSpec`][spec.specification.MerlinSpec] class with the expanded + configuration based on the provided filepath and variables. + filepath: The expanded filepath derived from the specification. """ filepath = verify_filepath(args.specification) variables_dict = parse_override_vars(args.variables) @@ -133,11 +157,27 @@ def get_merlin_spec_with_override(args): return spec, filepath -def process_run(args: Namespace) -> None: +def process_run(args: Namespace): """ CLI command for running a study. - :param [Namespace] `args`: parsed CLI arguments + This function initializes and runs a study using the specified parameters. + It handles file verification, variable parsing, and checks for required + arguments related to the study configuration and execution. + + Args: + args: Parsed CLI arguments containing:\n + - `specification`: Path to the specification file for the study. + - `variables`: Optional variable overrides for the study. + - `samples_file`: Optional path to a samples file. + - `dry`: If True, runs the study in dry-run mode (without actual execution). + - `no_errors`: If True, suppresses error reporting. + - `pgen_file`: Optional path to the pgen file, required if `pargs` is specified. + - `pargs`: Additional arguments for parallel processing. + + Raises: + ValueError: + If the `pargs` parameter is used without specifying a `pgen_file`. """ print(banner_small) filepath: str = verify_filepath(args.specification) @@ -161,14 +201,50 @@ def process_run(args: Namespace) -> None: pgen_file=args.pgen_file, pargs=args.pargs, ) + + if args.run_mode == "local": + initialize_config(local_mode=True) + + # Initialize the database + merlin_db = MerlinDatabase() + + # Create a run entry + run_entity = merlin_db.create( + "run", + study_name=study.expanded_spec.name, + workspace=study.workspace, + queues=study.expanded_spec.get_queue_list(["all"]), + ) + + # Create logical worker entries + step_queue_map = study.expanded_spec.get_task_queues() + for worker, steps in study.expanded_spec.get_worker_step_map().items(): + worker_queues = set([step_queue_map[step] for step in steps]) + logical_worker_entity = merlin_db.create("logical_worker", worker, worker_queues) + + # Add the run id to the worker entry and the worker id to the run entry + logical_worker_entity.add_run(run_entity.get_id()) + run_entity.add_worker(logical_worker_entity.get_id()) + router.run_task_server(study, args.run_mode) -def process_restart(args: Namespace) -> None: +def process_restart(args: Namespace): """ CLI command for restarting a study. - :param [Namespace] `args`: parsed CLI arguments + This function handles the restart process by verifying the specified restart + directory, locating a valid provenance specification file, and initiating + the study from that point. + + Args: + args: Parsed CLI arguments containing:\n + - `restart_dir`: Path to the directory where the restart specifications are located. + - `run_mode`: The mode for running the study (e.g., normal, dry-run). + + Raises: + ValueError: If the `restart_dir` does not contain a valid provenance spec file or + if multiple files match the specified pattern. """ print(banner_small) restart_dir: str = verify_dirpath(args.restart_dir) @@ -181,34 +257,68 @@ def process_restart(args: Namespace) -> None: filepath: str = verify_filepath(possible_specs[0]) LOG.info(f"Restarting workflow at '{restart_dir}'") study: MerlinStudy = MerlinStudy(filepath, restart_dir=restart_dir) + + if args.run_mode == "local": + initialize_config(local_mode=True) + router.run_task_server(study, args.run_mode) -def launch_workers(args): +def launch_workers(args: Namespace): """ CLI command for launching workers. - :param `args`: parsed CLI arguments + This function initializes worker processes for executing tasks as defined + in the Merlin specification. + + Args: + args: Parsed CLI arguments containing:\n + - `worker_echo_only`: If True, don't start the workers and just echo the launch command + - Additional worker-related parameters such as: + - `worker_steps`: Only start workers for these steps. + - `worker_args`: Arguments to pass to the worker processes. + - `disable_logs`: If True, disables logging for the worker processes. """ if not args.worker_echo_only: print(banner_small) + else: + initialize_config(local_mode=True) + spec, filepath = get_merlin_spec_with_override(args) if not args.worker_echo_only: LOG.info(f"Launching workers from '{filepath}'") + + # Initialize the database + merlin_db = MerlinDatabase() + + # Create logical worker entries + step_queue_map = spec.get_task_queues() + for worker, steps in spec.get_worker_step_map().items(): + worker_queues = set([step_queue_map[step] for step in steps]) + merlin_db.create("logical_worker", worker, worker_queues) + + # Launch the workers launch_worker_status = router.launch_workers( spec, args.worker_steps, args.worker_args, args.disable_logs, args.worker_echo_only ) + if args.worker_echo_only: print(launch_worker_status) else: LOG.debug(f"celery command: {launch_worker_status}") -def purge_tasks(args): +def purge_tasks(args: Namespace): """ - CLI command for purging tasks. + CLI command for purging tasks from the task server. + + This function removes specified tasks from the task server based on the provided + Merlin specification. It allows for targeted purging or forced removal of tasks. - :param `args`: parsed CLI arguments + Args: + args: Parsed CLI arguments containing:\n + - `purge_force`: If True, forces the purge operation without confirmation. + - `purge_steps`: Steps or criteria based on which tasks will be purged. """ print(banner_small) spec, _ = get_merlin_spec_with_override(args) @@ -222,15 +332,28 @@ def purge_tasks(args): LOG.info(f"Purge return = {ret} .") -def query_status(args): +def query_status(args: Namespace): """ - CLI command for querying status of studies. - Based on the parsed CLI args, construct either a Status object or a DetailedStatus object - and display the appropriate output. - Object mapping is as follows: - merlin status -> Status object ; merlin detailed-status -> DetailedStatus object - - :param `args`: parsed CLI arguments + CLI command for querying the status of studies. + + This function processes the given command-line arguments to determine the + status of a study. It constructs either a [`Status`][study.status.Status] object + or a [`DetailedStatus`][study.status.DetailedStatus] object based on the specified + command and the arguments provided. The function handles validations for the task + server input and the output format specified for status dumping. + + Object mapping: + - `merlin status` -> [`Status`][study.status.Status] object + - `merlin detailed-status` -> [`DetailedStatus`][study.status.DetailedStatus] + object + + Args: + args: Parsed CLI arguments containing user inputs for the status query. + + Raises: + ValueError: + - If the task server specified is not supported (only "celery" is valid). + - If the --dump filename provided does not end with ".csv" or ".json". """ print(banner_small) @@ -274,11 +397,24 @@ def query_status(args): return None -def query_queues(args): +def query_queues(args: Namespace): """ - CLI command for finding all workers. - - :param args: parsed CLI arguments + CLI command for finding all workers and their associated queues. + + This function processes the command-line arguments to retrieve and display + information about the available workers and their queues within the task server. + It validates the necessary parameters, handles potential file dumping, and + formats the output for easy readability. + + Args: + args: Parsed CLI arguments containing user inputs related to the query. + + Raises: + ValueError: + - If a specification is not provided when steps are specified and the + steps do not include "all". + - If variables are included without a corresponding specification. + - If the specified dump filename does not end with '.json' or '.csv'. """ print(banner_small) @@ -318,11 +454,19 @@ def query_queues(args): router.dump_queue_info(args.task_server, queue_information, args.dump) -def query_workers(args): +def query_workers(args: Namespace): """ CLI command for finding all workers. - :param `args`: parsed CLI arguments + This function retrieves and queries the names of any active workers. + If the `--spec` argument is included, only query the workers defined in the spec file. + + Args: + args: Parsed command-line arguments, which may include:\n + - `spec`: Path to the specification file. + - `task_server`: Address of the task server to query. + - `queues`: List of queue names to filter workers. + - `workers`: List of specific worker names to query. """ print(banner_small) @@ -340,11 +484,20 @@ def query_workers(args): router.query_workers(args.task_server, worker_names, args.queues, args.workers) -def stop_workers(args): +def stop_workers(args: Namespace): """ CLI command for stopping all workers. - :param `args`: parsed CLI arguments + This function stops any active workers connected to a user's task server. + If the `--spec` argument is provided, this function retrieves the names of + workers from a the spec file and then issues a command to stop them. + + Args: + args: Parsed command-line arguments, which may include:\n + - `spec`: Path to the specification file to load worker names. + - `task_server`: Address of the task server to send the stop command to. + - `queues`: List of queue names to filter the workers. + - `workers`: List of specific worker names to stop. """ print(banner_small) worker_names = [] @@ -362,11 +515,12 @@ def stop_workers(args): router.stop_workers(args.task_server, worker_names, args.queues, args.workers) -def print_info(args): +def print_info(args: Namespace): """ - CLI command to print merlin config info. + CLI command to print merlin configuration info. - :param `args`: parsed CLI arguments + Args: + args: Parsed CLI arguments. """ # if this is moved to the toplevel per standard style, merlin is unable to generate the (needed) default config file from merlin import display # pylint: disable=import-outside-toplevel @@ -374,24 +528,57 @@ def print_info(args): display.print_info(args) -def config_merlin(args: Namespace) -> None: +def config_merlin(args: Namespace): """ - CLI command to setup default merlin config. + CLI command to manage Merlin configuration files. - :param [Namespace] `args`: parsed CLI arguments - """ - output_dir: Optional[str] = args.output_dir - if output_dir is None: - user_home: str = os.path.expanduser("~") - output_dir: str = os.path.join(user_home, ".merlin") + This function handles various configuration-related operations based on + the provided subcommand. It ensures that the specified configuration + file has a valid YAML extension (i.e., `.yaml` or `.yml`). + + If no output file is explicitly provided, a default path is used. - router.create_config(args.task_server, output_dir, args.broker, args.test) + Args: + args: Parsed command-line arguments. + """ + if args.commands != "create": # Check that this is a valid yaml file + try: + with open(args.config_file, "r") as conf_file: + yaml.safe_load(conf_file) + except FileNotFoundError: + raise ArgumentTypeError(f"The file '{args.config_file}' does not exist.") + except yaml.YAMLError as e: + raise ArgumentTypeError(f"The file '{args.config_file}' is not a valid YAML file: {e}") + + config_manager = MerlinConfigManager(args) + + if args.commands == "create": + config_manager.create_template_config() + config_manager.save_config_path() + elif args.commands == "update-broker": + config_manager.update_broker() + elif args.commands == "update-backend": + config_manager.update_backend() + elif args.commands == "use": # Config file path is updated in constructor of MerlinConfigManager + config_manager.config_file = args.config_file + config_manager.save_config_path() def process_example(args: Namespace) -> None: - """Either lists all example workflows, or sets up an example as a workflow to be run at root dir. - - :param [Namespace] `args`: parsed CLI arguments + """ + CLI command to set up or list Merlin example workflows. + + This function either lists all available example workflows or sets + up a specified example workflow to be run in the root directory. The + behavior is determined by the `workflow` argument. + + Args: + args: Parsed command-line arguments, which may include:\n + - `workflow`: The action to perform; should be "list" + to display all examples or the name of a specific example + workflow to set up. + - `path`: The directory where the example workflow + should be set up. Only applicable when `workflow` is not "list". """ if args.workflow == "list": print(list_examples()) @@ -400,30 +587,62 @@ def process_example(args: Namespace) -> None: setup_example(args.workflow, args.path) -def process_monitor(args): +def process_monitor(args: Namespace): """ - CLI command to monitor merlin workers and queues to keep - the allocation alive - - :param `args`: parsed CLI arguments + CLI command to monitor Merlin workers and queues to maintain + allocation status. + + This function periodically checks the status of Merlin workers and + the associated queues to ensure that the allocation remains active. + It includes a sleep interval to wait before each check, including + the initial one. + + Args: + args: Parsed command-line arguments, which may include:\n + - `sleep`: The duration (in seconds) to wait before + checking the queue status again. """ - LOG.info("Monitor: checking queues ...") spec, _ = get_merlin_spec_with_override(args) # Give the user time to queue up jobs in case they haven't already time.sleep(args.sleep) - # Check if we still need our allocation - while router.check_merlin_status(args, spec): - LOG.info("Monitor: found tasks in queues and/or tasks being processed") - time.sleep(args.sleep) + if args.steps != ["all"]: + LOG.warning( + "The `--steps` argument of the `merlin monitor` command is set to be deprecated in Merlin v1.14 " + "For now, using this argument will tell merlin to use the version of the monitor command from Merlin v1.12." + ) + # Check if we still need our allocation + while router.check_merlin_status(args, spec): + LOG.info("Monitor: found tasks in queues and/or tasks being processed") + time.sleep(args.sleep) + else: + monitor = Monitor(spec, args.sleep, args.task_server) + monitor.monitor_all_runs() + LOG.info("Monitor: ... stop condition met") def process_server(args: Namespace): """ - Route to the correct function based on the command - given via the CLI + Route to the appropriate server function based on the command + specified via the CLI. + + This function processes commands related to server management, + directing the flow to the corresponding function for actions such + as initializing, starting, stopping, checking status, restarting, + or configuring the server. + + Args: + args: Parsed command-line arguments, which includes:\n + - `commands`: The server management command to execute. + Possible values are: + - `init`: Initialize the server. + - `start`: Start the server. + - `stop`: Stop the server. + - `status`: Check the server status. + - `restart`: Restart the server. + - `config`: Configure the server. """ try: lc_all_val = os.environ["LC_ALL"] @@ -447,11 +666,34 @@ def process_server(args: Namespace): config_server(args) +def process_database(args: Namespace): + """ + Process database commands by routing to the correct function. + + Args: + args: An argparse Namespace containing user arguments. + """ + if args.local: + initialize_config(local_mode=True) + + if args.commands == "info": + database_info() + elif args.commands == "get": + database_get(args) + elif args.commands == "delete": + database_delete(args) + + # Pylint complains that there's too many statements here and wants us # to split the function up but that wouldn't make much sense so we ignore it def setup_argparse() -> None: # pylint: disable=R0915 """ - Setup argparse and any CLI options we want available via the package. + Set up the command-line argument parser for the Merlin package. + + This function configures the ArgumentParser for the Merlin CLI, allowing users + to interact with various commands related to workflow management and task handling. + It includes options for running a workflow, restarting tasks, purging task queues, + generating configuration files, and managing/configuring the server. """ parser: HelpParser = HelpParser( prog="merlin", @@ -602,35 +844,90 @@ def setup_argparse() -> None: # pylint: disable=R0915 formatter_class=ArgumentDefaultsHelpFormatter, ) mconfig.set_defaults(func=config_merlin) + # The below option makes it so the `config_path.txt` file is written to the test directory mconfig.add_argument( - "--task_server", + "-t", + "--test", + action="store_true", + help=SUPPRESS, # Hides from `--help` + ) + mconfig_subparsers = mconfig.add_subparsers(dest="commands", help="Subcommands for 'config'") + default_config_file = os.path.join(os.path.expanduser("~"), ".merlin", "app.yaml") + + # Subcommand: melrin config create + config_create_parser = mconfig_subparsers.add_parser("create", help="Create a new configuration file.") + config_create_parser.add_argument( + "--task-server", type=str, default="celery", - help="Task server type for which to create the config.\ - Default: %(default)s", + help="Task server type for which to create the config. Default: %(default)s", ) - mconfig.add_argument( + config_create_parser.add_argument( "-o", - "--output_dir", + "--output-file", + dest="config_file", type=str, - default=None, - help="Optional directory to place the default config file.\ - Default: ~/.merlin", + default=default_config_file, + help=f"Optional file name for your configuration. Default: {default_config_file}", ) - mconfig.add_argument( + config_create_parser.add_argument( "--broker", type=str, default=None, - help="Optional broker type, backend will be redis\ - Default: rabbitmq", - ) - mconfig.add_argument( - "--test", - type=str, - default=None, - help="A config used in the testing suite (or for exemplative purposes).\ - Default: rabbitmq", - ) + help="Optional broker type, backend will be redis. Default: rabbitmq", + ) + + # Subcommand: merlin config update-broker + config_broker_parser = mconfig_subparsers.add_parser("update-broker", help="Update broker settings in app.yaml") + config_broker_parser.add_argument( + "-t", + "--type", + required=True, + choices=["redis", "rabbitmq"], + help="Type of broker to configure (redis or rabbitmq).", + ) + config_broker_parser.add_argument( + "--cf", + "--config-file", + dest="config_file", + default=default_config_file, + help=f"The path to the config file that will be updated. Default: {default_config_file}", + ) + config_broker_parser.add_argument("-u", "--username", help="Broker username (only for rabbitmq)") + config_broker_parser.add_argument("--pf", "--password-file", dest="password_file", help="Path to password file") + config_broker_parser.add_argument("-s", "--server", help="The URL of the server") + config_broker_parser.add_argument("-p", "--port", type=int, help="Broker port") + config_broker_parser.add_argument("-v", "--vhost", help="Broker vhost (only for rabbitmq)") + config_broker_parser.add_argument("-c", "--cert-reqs", help="Broker cert requirements") + config_broker_parser.add_argument("-d", "--db-num", type=int, help="Redis database number (only for redis).") + + # Subcommand: merlin config update-backend + config_backend_parser = mconfig_subparsers.add_parser("update-backend", help="Update results backend settings in app.yaml") + config_backend_parser.add_argument( + "-t", + "--type", + required=True, + choices=["redis"], + help="Type of results backend to configure.", + ) + config_backend_parser.add_argument( + "--cf", + "--config-file", + dest="config_file", + default=default_config_file, + help=f"The path to the config file that will be updated. Default: {default_config_file}", + ) + config_backend_parser.add_argument("-u", "--username", help="Backend username") + config_backend_parser.add_argument("--pf", "--password-file", dest="password_file", help="Path to password file") + config_backend_parser.add_argument("-s", "--server", help="The URL of the server") + config_backend_parser.add_argument("-p", "--port", help="Backend port") + config_backend_parser.add_argument("-d", "--db-num", help="Backend database number") + config_backend_parser.add_argument("-c", "--cert-reqs", help="Backend cert requirements") + config_backend_parser.add_argument("-e", "--encryption-key", help="Path to encryption key file") + + # Subcommand: merlin config use + config_use_parser = mconfig_subparsers.add_parser("use", help="Use a different configuration file.") + config_use_parser.add_argument("config_file", type=str, help="The path to the new configuration file to use.") # merlin example example: ArgumentParser = subparsers.add_parser( @@ -795,14 +1092,265 @@ def setup_argparse() -> None: # pylint: disable=R0915 help="Set append only filename for merlin server container.", ) + # merlin database + database: ArgumentParser = subparsers.add_parser( + "database", + help="Interact with Merlin's database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + database.set_defaults(func=process_database) + + database.add_argument( + "-l", + "--local", + action="store_true", + help="Use the local SQLite database for this command.", + ) + + database_commands: ArgumentParser = database.add_subparsers(dest="commands") + + # Subcommand: database info + database_commands.add_parser( + "info", + help="Print information about the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: database delete + db_delete: ArgumentParser = database_commands.add_parser( + "delete", + help="Delete information stored in the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Add subcommands for delete + delete_subcommands = db_delete.add_subparsers(dest="delete_type", required=True) + + # TODO enable support for deletion of study by passing in spec file + # Subcommand: delete study + delete_study = delete_subcommands.add_parser( + "study", + help="Delete one or more studies by ID or name.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + delete_study.add_argument( + "study", + type=str, + nargs="+", + help="A space-delimited list of IDs or names of studies to delete.", + ) + delete_study.add_argument( + "-k", + "--keep-associated-runs", + action="store_true", + help="Keep runs associated with the studies.", + ) + + # Subcommand: delete run + delete_run = delete_subcommands.add_parser( + "run", + help="Delete one or more runs by ID or workspace.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + delete_run.add_argument( + "run", + type=str, + nargs="+", + help="A space-delimited list of IDs or workspaces of runs to delete.", + ) + # TODO implement the below option; this removes the output workspace from file system + # delete_run.add_argument( + # "--delete-workspace", + # action="store_true", + # help="Delete the output workspace for the run.", + # ) + + # Subcommand: delete logical-worker + delete_logical_worker = delete_subcommands.add_parser( + "logical-worker", + help="Delete one or more logical workers by ID.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + delete_logical_worker.add_argument( + "worker", + type=str, + nargs="+", + help="A space-delimited list of IDs of logical workers to delete.", + ) + + # Subcommand: delete physical-worker + delete_physical_worker = delete_subcommands.add_parser( + "physical-worker", + help="Delete one or more physical workers by ID or name.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + delete_physical_worker.add_argument( + "worker", + type=str, + nargs="+", + help="A space-delimited list of IDs of physical workers to delete.", + ) + + # Subcommand: delete all-studies + delete_all_studies = delete_subcommands.add_parser( + "all-studies", + help="Delete all studies from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + delete_all_studies.add_argument( + "-k", + "--keep-associated-runs", + action="store_true", + help="Keep runs associated with the studies.", + ) + + # Subcommand: delete all-runs + delete_subcommands.add_parser( + "all-runs", + help="Delete all runs from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: delete all-logical-workers + delete_subcommands.add_parser( + "all-logical-workers", + help="Delete all logical workers from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: delete all-physical-workers + delete_subcommands.add_parser( + "all-physical-workers", + help="Delete all physical workers from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: delete everything + delete_everything = delete_subcommands.add_parser( + "everything", + help="Delete everything from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + delete_everything.add_argument( + "-f", + "--force", + action="store_true", + help="Delete everything in the database without confirmation.", + ) + + # Subcommand: database get + db_get: ArgumentParser = database_commands.add_parser( + "get", + help="Get information stored in the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Add subcommands for get + get_subcommands = db_get.add_subparsers(dest="get_type", required=True) + + # TODO enable support for retrieval of study by passing in spec file + # Subcommand: get study + get_study = get_subcommands.add_parser( + "study", + help="Get one or more studies by ID or name.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + get_study.add_argument( + "study", + type=str, + nargs="+", + help="A space-delimited list of IDs or names of the studies to get.", + ) + + # Subcommand: get run + get_run = get_subcommands.add_parser( + "run", + help="Get one or more runs by ID or workspace.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + get_run.add_argument( + "run", + type=str, + nargs="+", + help="A space-delimited list of IDs or workspaces of the runs to get.", + ) + + # Subcommand get logical-worker + get_logical_worker = get_subcommands.add_parser( + "logical-worker", + help="Get one or more logical workers by ID.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + get_logical_worker.add_argument( + "worker", + type=str, + nargs="+", + help="A space-delimited list of IDs of the logical workers to get.", + ) + + # Subcommand get physical-worker + get_physical_worker = get_subcommands.add_parser( + "physical-worker", + help="Get one or more physical workers by ID or name.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + get_physical_worker.add_argument( + "worker", + type=str, + nargs="+", + help="A space-delimited list of IDs or names of the physical workers to get.", + ) + + # Subcommand: get all-studies + get_subcommands.add_parser( + "all-studies", + help="Get all studies from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: get all-runs + get_subcommands.add_parser( + "all-runs", + help="Get all runs from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: get all-logical-workers + get_subcommands.add_parser( + "all-logical-workers", + help="Get all logical workers from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: get all-physical-workers + get_subcommands.add_parser( + "all-physical-workers", + help="Get all physical workers from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: get everything + get_subcommands.add_parser( + "everything", + help="Get everything from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + return parser def generate_worker_touching_parsers(subparsers: ArgumentParser) -> None: - """All CLI arg parsers directly controlling or invoking workers are generated here. + """ + Generate command-line argument parsers for managing worker operations. + + This function sets up subparsers for CLI commands that directly control or invoke + workers in the context of the Merlin framework. It provides options for running, + querying, stopping, and monitoring workers associated with a Merlin YAML study + specification. - :param [ArgumentParser] `subparsers`: the subparsers needed for every CLI command that directly controls or invokes - workers. + Args: + subparsers: An instance of ArgumentParser for adding command-line subcommands + related to worker management. """ # merlin run-workers run_workers: ArgumentParser = subparsers.add_parser( @@ -947,11 +1495,18 @@ def generate_worker_touching_parsers(subparsers: ArgumentParser) -> None: monitor.set_defaults(func=process_monitor) -def generate_diagnostic_parsers(subparsers: ArgumentParser) -> None: - """All CLI arg parsers generally used diagnostically are generated here. +def generate_diagnostic_parsers(subparsers: ArgumentParser): + """ + Generate command-line argument parsers for diagnostic operations in the Merlin framework. - :param [ArgumentParser] `subparsers`: the subparsers needed for every CLI command that handles diagnostics for a - Merlin job. + This function sets up subparsers for CLI commands that handle diagnostics related + to Merlin jobs. It provides options to check the status of studies, gather queue + statistics, and retrieve configuration information, making it easier for users to + diagnose issues with their workflows. + + Args: + subparsers: An instance of ArgumentParser that will be used to add command-line + subcommands for various diagnostic activities. """ # merlin status status_cmd: ArgumentParser = subparsers.add_parser( @@ -1141,7 +1696,13 @@ def generate_diagnostic_parsers(subparsers: ArgumentParser) -> None: def main(): """ - High-level CLI operations. + Entry point for the Merlin command-line interface (CLI) operations. + + This function sets up the argument parser, handles command-line arguments, + initializes logging, and executes the appropriate function based on the + provided command. It ensures that the user receives help information if + no arguments are provided and performs error handling for any exceptions + that may occur during command execution. """ parser = setup_argparse() if len(sys.argv) == 1: diff --git a/merlin/merlin_templates.py b/merlin/merlin_templates.py deleted file mode 100644 index f82d814f6..000000000 --- a/merlin/merlin_templates.py +++ /dev/null @@ -1,75 +0,0 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### - -""" -This module handles the CLI for the deprecated `merlin-templates` command. -""" -import argparse -import logging -import sys - -from merlin.ascii_art import banner_small -from merlin.log_formatter import setup_logging - - -LOG = logging.getLogger("merlin-templates") -DEFAULT_LOG_LEVEL = "ERROR" - -# We disable all pylint errors in this file since this is deprecated anyways - - -def process_templates(args): # pylint: disable=W0613,C0116 - LOG.error("The command `merlin-templates` has been deprecated in favor of `merlin example`.") - - -def setup_argparse(): # pylint: disable=C0116 - parser = argparse.ArgumentParser( - prog="Merlin Examples", - description=banner_small, - formatter_class=argparse.RawDescriptionHelpFormatter, - ) - parser.set_defaults(func=process_templates) - return parser - - -def main(): # pylint: disable=C0116 - try: - parser = setup_argparse() - args = parser.parse_args() - setup_logging(logger=LOG, log_level=DEFAULT_LOG_LEVEL, colors=True) - args.func(args) - sys.exit() - except Exception as ex: # pylint: disable=W0718 - print(ex) - sys.exit(1) - - -if __name__ == "__main__": - main() diff --git a/merlin/monitor/__init__.py b/merlin/monitor/__init__.py new file mode 100644 index 000000000..671d1f18b --- /dev/null +++ b/merlin/monitor/__init__.py @@ -0,0 +1,24 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +The `monitor` package provides classes and utilities for monitoring the health and progress +of task servers and workflows in Merlin. It includes abstract interfaces, concrete implementations, +and factory classes to manage task server monitors for supported systems like Celery. + +Modules: + monitor_factory.py: Provides a factory class to manage and retrieve task server monitors for + supported task servers. + monitor.py: Defines the [`Monitor`][monitor.monitor.Monitor] class for monitoring the progress + of Merlin workflows, ensuring workers and tasks are functioning correctly and preventing + workflow hangs. + task_server_monitor.py: Defines the [`TaskServerMonitor`][monitor.task_server_monitor.TaskServerMonitor] + abstract base class, which serves as a common interface for monitoring task servers, including + methods for worker and task monitoring. + celery_monitor.py: Implements the [`CeleryMonitor`][monitor.celery_monitor.CeleryMonitor] class, + a concrete subclass of [`TaskServerMonitor`][monitor.task_server_monitor.TaskServerMonitor] + for monitoring Celery task servers, including worker health checks and task queue monitoring. +""" diff --git a/merlin/monitor/celery_monitor.py b/merlin/monitor/celery_monitor.py new file mode 100644 index 000000000..0e11fee9b --- /dev/null +++ b/merlin/monitor/celery_monitor.py @@ -0,0 +1,188 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module provides the `CeleryMonitor` class, a concrete implementation of the +[`TaskServerMonitor`][monitor.task_server_monitor.TaskServerMonitor] interface for monitoring +Celery task servers. Celery is a distributed task queue system commonly used for executing +asynchronous tasks and managing worker nodes. + +The `CeleryMonitor` class combines task and worker monitoring functionality specific to Celery. +It provides methods to: + +- Wait for workers to start. +- Check for tasks in the queues. +- Monitor worker activity. +- Run health checks to ensure workers are alive and functioning. +""" + +import logging +import time +from typing import List, Set + +from merlin.db_scripts.entities.run_entity import RunEntity +from merlin.exceptions import NoWorkersException +from merlin.monitor.task_server_monitor import TaskServerMonitor +from merlin.study.celeryadapter import get_workers_from_app, query_celery_queues + + +LOG = logging.getLogger(__name__) + + +class CeleryMonitor(TaskServerMonitor): + """ + Implementation of [`TaskServerMonitor`][monitor.task_server_monitor.TaskServerMonitor] + for Celery task servers. This class provides methods to monitor Celery workers, tasks, + and workflows. + + Methods: + wait_for_workers: Wait for Celery workers to start up. + check_workers_processing: Check if any Celery workers are still processing tasks. + _restart_workers: Restart a list of (dead) Celery workers. + _get_dead_workers: Get a list of dead Celery workers. + run_worker_health_check: Check the health of Celery workers and restart any that are dead. + check_tasks: Checks the status of tasks in the Celery queues for a given workflow run. + """ + + def wait_for_workers(self, workers: List[str], sleep: int): + """ + Wait for Celery workers to start up. + + Args: + workers: A list of worker names or IDs to wait for. + sleep: The interval (in seconds) between checks for worker availability. + + Raises: + NoWorkersException: When workers don't start in (`self.sleep` * 10) seconds. + """ + count = 0 + max_count = 10 + while count < max_count: + worker_status = get_workers_from_app() + LOG.debug(f"CeleryMonitor: checking for workers, running workers = {worker_status} ...") + + # Check if any of the desired workers have started + check = any(any(iwn in iws for iws in worker_status) for iwn in workers) + if check: + break + + count += 1 + time.sleep(sleep) + + if count == max_count: + raise NoWorkersException("Monitor: no workers available to process the non-empty queue") + + def check_workers_processing(self, queues: List[str]) -> bool: + """ + Check if any Celery workers are still processing tasks. + + Args: + queues: A list of queue names to check for active tasks. + + Returns: + True if workers are processing tasks in the specified queues, False otherwise. + """ + from merlin.celery import app # pylint: disable=import-outside-toplevel + + # Query celery for active tasks + active_tasks = app.control.inspect().active() + + # Search for the queues we provided + if active_tasks is not None: + for tasks in active_tasks.values(): + for task in tasks: + if task["delivery_info"]["routing_key"] in queues: + return True + + return False + + def _restart_workers(self, workers: List[str]): + """ + Restart a dead Celery worker. + + Args: + workers: A list of worker names or IDs to restart. + """ + for worker in workers: + try: + LOG.warning(f"CeleryMonitor: Worker '{worker}' has died. Attempting to restart...") + # TODO figure out the restart logic; will likely need stuff from manager branch + LOG.info(f"CeleryMonitor: Worker '{worker}' has been successfully restarted.") + except Exception as e: # pylint: disable=broad-exception-caught + LOG.error(f"CeleryMonitor: Failed to restart worker '{worker}'. Error: {e}") + + def _get_dead_workers(self, workers: List[str]) -> Set[str]: + """ + Identify unresponsive Celery workers from a given list. + + This function sends a ping to all specified workers and identifies + which workers did not respond within the given timeout. + + Args: + workers: A list of Celery worker names to check. + + Returns: + Set[str]: A set of unresponsive worker names. + """ + from merlin.celery import app # pylint: disable=import-outside-toplevel + + # Send ping to all workers + responses = app.control.ping(destination=workers, timeout=5.0) # TODO May want to customize timeout like manager does + + # Extract unresponsive workers + unresponsive_workers = set() + for response in responses: + for worker, reply in response.items(): + if not reply.get("ok") == "pong": + unresponsive_workers.add(worker) + LOG.debug(f"CeleryMonitor: Unresponsive worker '{worker}' gave this reply when pinged: {reply}") + + if unresponsive_workers: + LOG.warning(f"CeleryMonitor: Found unresponsive workers: {unresponsive_workers}") + else: + LOG.info("CeleryMonitor: All workers are alive and responsive.") + + return unresponsive_workers + + def run_worker_health_check(self, workers: List[str]): + """ + Check the health of Celery workers and restart any that are dead. + + Args: + workers: A list of worker names or IDs to check for health. + + Raises: + WorkerRestartException: If a worker fails to restart. + """ + dead_workers = self._get_dead_workers(workers) + if dead_workers: + self._restart_workers(dead_workers) + + def check_tasks(self, run: RunEntity) -> bool: + """ + Check the status of tasks in Celery queues for the given workflow run. + + Args: + run: A [`RunEntity`][db_scripts.entities.run_entity.RunEntity] instance representing + the workflow run whose tasks are being monitored. + + Returns: + True if tasks are active in the workflow (i.e., jobs are present in the queues), + False otherwise. + """ + queues_in_run = run.get_queues() + LOG.debug(f"CeleryMonitor: queues_in_run={queues_in_run}") + queue_status = query_celery_queues(queues_in_run) + LOG.debug(f"CeleryMonitor: Result of querying celery queues: {queue_status}") + + total_jobs = 0 + for queue_info in queue_status.values(): + total_jobs += queue_info["jobs"] + LOG.debug(f"CeleryMonitor: total_jobs={total_jobs}") + + if total_jobs > 0: + return True + return False diff --git a/merlin/monitor/monitor.py b/merlin/monitor/monitor.py new file mode 100644 index 000000000..d35095e2f --- /dev/null +++ b/merlin/monitor/monitor.py @@ -0,0 +1,198 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module provides the `Monitor` class, which is responsible for monitoring the progress of +Merlin workflows. It ensures that workers are running, tasks are being processed, and workflows +are restarted if needed to prevent hanging. The `Monitor` class uses worker and task monitors +to manage the health and progress of workflows. + +The module interacts with the Merlin database to retrieve study and run information and +uses the `monitor_factory` to create monitors for task and worker systems (e.g., Celery). + +Exceptions such as Redis timeouts, Kombu operational errors, and other runtime issues are +handled gracefully to ensure that monitoring continues without interruption. +""" + +import logging +import subprocess +import time +import traceback + +from kombu.exceptions import OperationalError +from redis.exceptions import TimeoutError as RedisTimeoutError + +from merlin.db_scripts.entities.run_entity import RunEntity +from merlin.db_scripts.merlin_db import MerlinDatabase +from merlin.exceptions import RestartException +from merlin.monitor.monitor_factory import monitor_factory +from merlin.monitor.task_server_monitor import TaskServerMonitor +from merlin.spec.specification import MerlinSpec +from merlin.utils import verify_dirpath + + +LOG = logging.getLogger(__name__) + + +class Monitor: + """ + The `Monitor` class is responsible for monitoring the progress of Merlin workflows. It ensures + that workers are running, tasks are being processed, and workflows are restarted if necessary + to prevent hanging. As a side-effect of the monitor, the users allocation will remain alive for + however long the monitor lives. The class interacts with the Merlin database to retrieve study + and run information and uses a task server monitor to help manage workflow health. + + Attributes: + spec (MerlinSpec): The Merlin specification that defines the workflow. + sleep (int): The interval (in seconds) between monitoring checks. + task_server_monitor (TaskServerMonitor): A monitor for interacting with whichever task server + that the user is utilizing. + + Methods: + monitor_all_runs: Monitors all runs of the current study until they are complete. + monitor_single_run: Monitors a single run of a study until it completes. + restart_workflow: Restart a run of a workflow. + """ + + def __init__(self, spec: MerlinSpec, sleep: int, task_server: str): + """ + Initializes the `Monitor` instance with the given Merlin specification, sleep interval, + and task server type. The task server monitor is created using the + [`monitor_factory`][monitor.monitor_factory.MonitorFactory]. + + Args: + spec (MerlinSpec): The Merlin specification that defines the workflow. + sleep (int): The interval (in seconds) between monitoring checks. + task_server (str): The type of task server being used (e.g., "celery"). + """ + self.spec: MerlinSpec = spec + self.sleep: int = sleep + self.task_server_monitor: TaskServerMonitor = monitor_factory.get_monitor(task_server) + self.merlin_db = MerlinDatabase() + + def monitor_all_runs(self): + """ + Monitors all runs of the current study until they are complete. For each run, it checks + if the run is already complete. If not, it monitors the run until it finishes. This + method ensures that all runs in the study are processed. This is necessary to be able to + monitor iterative workflows. + + The method retrieves all runs from the database and iterates through them sequentially. + If a run is incomplete, it calls [`monitor_single_run`][monitor.monitor.Monitor.monitor_single_run] + to monitor it until completion. + """ + study_entity = self.merlin_db.get("study", self.spec.name) + + index = 0 + while True: + # Always refresh the list at the start of the loop; there could be new runs (think iterative studies) + all_runs = [self.merlin_db.get("run", run_id) for run_id in study_entity.get_runs()] + if index >= len(all_runs): # Break if there are no more runs to process + break + + run = all_runs[index] + run_workspace = run.get_workspace() + LOG.info(f"Monitor: Checking if run with workspace '{run_workspace}' has completed...") + + if run.run_complete: + LOG.info( + f"Monitor: Determined that run with workspace '{run_workspace}' has already completed. " + "Moving on to the next run." + ) + index += 1 + continue + + LOG.info(f"Monitor: Run with workspace '{run_workspace}' has not yet completed.") + + # Monitor the run until it completes + self.monitor_single_run(run) + + index += 1 + + def monitor_single_run(self, run: RunEntity): + """ + Monitors a single run of a study until it completes to ensure that the allocation stays alive + and workflows are restarted if necessary. + + Args: + run: A [`RunEntity`][db_scripts.entities.run_entity.RunEntity] instance representing + the run that's going to be monitored. + """ + run_workspace = run.get_workspace() + run_complete = run.run_complete # Saving this to a variable as it queries the db each time it's called + + LOG.info(f"Monitor: Monitoring run with workspace '{run_workspace}'...") + + # Wait for workers to spin up before checking on tasks + worker_names = [ + self.merlin_db.get("logical_worker", worker_id=worker_id).get_name() for worker_id in run.get_workers() + ] + LOG.info(f"Monitor: Waiting for the following workers to start: {worker_names}...") + self.task_server_monitor.wait_for_workers(worker_names, self.sleep) + LOG.info("Monitor: Workers have started.") + + while not run_complete: + try: + # Run worker health check (checks for dead workers and restarts them if necessary) + self.task_server_monitor.run_worker_health_check(run.get_workers()) + + # Check if any tasks are currently in the queues + active_tasks = self.task_server_monitor.check_tasks(run) + if active_tasks: + LOG.info("Monitor: Found tasks in queues, keeping allocation alive.") + else: + # If no tasks are in the queues, check if workers are processing tasks + active_tasks = self.task_server_monitor.check_workers_processing(run.get_queues()) + if active_tasks: + LOG.info("Monitor: Found workers processing tasks, keeping allocation alive.") + + # If no tasks are in the queues or being processed by workers and the run is not complete, we have a hanging + # workflow so restart it + run_complete = run.run_complete # Re-query db for this value + if not active_tasks and not run_complete: + self.restart_workflow(run) + + if not run_complete: + time.sleep(self.sleep) + # The below exceptions do not modify the `run_complete` value so the loop should retry + except RedisTimeoutError as exc: + LOG.warning(f"Redis timed out:\n{exc}") + LOG.warning(f"Full traceback:\n{traceback.format_exc()}") + time.sleep(self.sleep) + except OperationalError as exc: + LOG.warning(f"Kombu raised an error:\n{exc}") + LOG.warning(f"Full traceback:\n{traceback.format_exc()}") + time.sleep(self.sleep) + except TimeoutError as exc: + LOG.warning(f"A standard TimeoutError has occurred:\n{exc}") + LOG.warning(f"Full traceback:\n{traceback.format_exc()}") + time.sleep(self.sleep) + + LOG.info(f"Monitor: Run with workspace '{run_workspace}' has completed.") + + def restart_workflow(self, run: RunEntity): + """ + Restart a run of a workflow. + + Args: + run: A [`RunEntity`][db_scripts.entities.run_entity.RunEntity] instance representing + the run that's going to be restarted. + + Raises: + RestartException: If the workflow restart process fails. + """ + try: + run_workspace = verify_dirpath(run.get_workspace()) + LOG.info(f"Monitor: Restarting workflow for run with workspace '{run_workspace}'...") + restart_proc = subprocess.run(f"merlin restart {run_workspace}", shell=True, capture_output=True, text=True) + if restart_proc.returncode != 0: + LOG.error(f"Monitor: Failed to restart workflow: {restart_proc.stderr}") + raise RestartException(f"Restart process failed with error: {restart_proc.stderr}") + LOG.info(f"Monitor: Workflow restarted successfully: {restart_proc.stdout}") + except ValueError: + LOG.warning( + f"Monitor: Run with workspace '{run.get_workspace()}' was not found. Ignoring the restart of this workspace." + ) diff --git a/merlin/monitor/monitor_factory.py b/merlin/monitor/monitor_factory.py new file mode 100644 index 000000000..b3c5fc138 --- /dev/null +++ b/merlin/monitor/monitor_factory.py @@ -0,0 +1,75 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module provides a factory class to manage and retrieve task server monitors +for supported task servers in Merlin. +""" + +from typing import Dict, List + +from merlin.exceptions import MerlinInvalidTaskServerError +from merlin.monitor.celery_monitor import CeleryMonitor +from merlin.monitor.task_server_monitor import TaskServerMonitor + + +class MonitorFactory: + """ + A factory class for managing and retrieving task server monitors + for supported task servers in Merlin. + + Attributes: + _monitors (Dict[str, TaskServerMonitor]): A dictionary mapping task server names + to their corresponding monitor classes. + + Methods: + get_supported_task_servers: Get a list of the supported task servers in Merlin. + get_monitor: Get the monitor instance for the specified task server. + """ + + def __init__(self): + """ + Initialize the `MonitorFactory` with the supported task server monitors. + """ + self._monitors: Dict[str, TaskServerMonitor] = { + "celery": CeleryMonitor, + } + + def get_supported_task_servers(self) -> List[str]: + """ + Get a list of the supported task servers in Merlin. + + Returns: + A list of names representing the supported task servers in Merlin. + """ + return list(self._monitors.keys()) + + def get_monitor(self, task_server: str) -> TaskServerMonitor: + """ + Get the task server monitor for whichever task server the user is utilizing. + + Args: + task_server: The name of the task server to use when loading a task server monitor. + + Returns: + An instantiated [`TaskServerMonitor`][monitor.task_server_monitor.TaskServerMonitor] + object for the specified task server. + + Raises: + MerlinInvalidTaskServerError: If the requested task server is not supported. + """ + monitor_object = self._monitors.get(task_server, None) + + if monitor_object is None: + raise MerlinInvalidTaskServerError( + f"Task server unsupported by Merlin: {task_server}. " + "Supported task servers are: {self.get_supported_task_servers()}" + ) + + return monitor_object() + + +monitor_factory = MonitorFactory() diff --git a/merlin/monitor/task_server_monitor.py b/merlin/monitor/task_server_monitor.py new file mode 100644 index 000000000..d81a453e1 --- /dev/null +++ b/merlin/monitor/task_server_monitor.py @@ -0,0 +1,86 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module defines the `TaskServerMonitor` abstract base class, which serves as a common interface +for monitoring task servers. Task servers are responsible for managing the execution of tasks and +workers in distributed systems, and this class provides an abstraction for monitoring their health +and progress. + +The `TaskServerMonitor` class is intended to be subclassed for specific task server implementations +(e.g., Celery, TaskVine). Subclasses must implement all abstract methods to provide task server-specific +functionality, such as waiting for workers and checking task queues. +""" + +from abc import ABC, abstractmethod +from typing import List + +from merlin.db_scripts.entities.run_entity import RunEntity + + +class TaskServerMonitor(ABC): + """ + Abstract base class for monitoring task servers. This class defines the interface + for monitoring tasks and workers for a specific task server (e.g., Celery, TaskVine). + + Subclasses must implement all abstract methods to provide specific functionality + for their respective task server. + + Methods: + wait_for_workers: Wait for workers to start up. + check_workers_processing: Check if any workers are still processing tasks. + restart_worker: Restart a dead worker. + run_worker_health_check: Check the health of workers and restart any that are dead. + check_tasks: Abstract method to check the status of tasks in a workflow run. + Must be implemented by subclasses. + """ + + @abstractmethod + def wait_for_workers(self, workers: List[str], sleep: int): # TODO should workers list be worker names or IDs? + """ + Wait for workers to start up. + + Args: + workers: A list of worker names or IDs to wait for. + sleep: The interval (in seconds) between checks for worker availability. + + Raises: + NoWorkersException: When workers don't start in (`self.sleep` * 10) seconds. + """ + + @abstractmethod + def check_workers_processing(self, queues: List[str]) -> bool: + """ + Check if any workers are still processing tasks. + + Args: + queues: A list of queue names to check for active tasks. + + Returns: + True if workers are processing tasks in the specified queues, False otherwise. + """ + + @abstractmethod + def run_worker_health_check(self, workers: List[str]): + """ + Checks the health of the workers provided and restarts any that are dead. + + Args: + workers: A list of workers to check for worker health. + """ + + @abstractmethod + def check_tasks(self, run: RunEntity) -> bool: + """ + Check the status of tasks in the given workflow run. + + Args: + run: A [`RunEntity`][db_scripts.entities.run_entity.RunEntity] instance representing + the workflow run whose tasks are being monitored. + + Returns: + True if tasks are active in the workflow, False otherwise. + """ diff --git a/merlin/router.py b/merlin/router.py index 552323260..51ad03a2a 100644 --- a/merlin/router.py +++ b/merlin/router.py @@ -1,32 +1,8 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ This module routes actions from the Merlin CLI to the appropriate tasking @@ -36,15 +12,15 @@ decoupled from the logic the tasks are running. """ import logging -import os import time +from argparse import Namespace from typing import Dict, List, Tuple from merlin.exceptions import NoWorkersException +from merlin.spec.specification import MerlinSpec from merlin.study.celeryadapter import ( build_set_of_queues, check_celery_workers_processing, - create_celery_config, dump_celery_queue_info, get_active_celery_queues, get_workers_from_app, @@ -55,12 +31,7 @@ start_celery_workers, stop_celery_workers, ) - - -try: - from importlib import resources -except ImportError: - import importlib_resources as resources +from merlin.study.study import MerlinStudy LOG = logging.getLogger(__name__) @@ -70,12 +41,20 @@ # and try to resolve them -def run_task_server(study, run_mode=None): +def run_task_server(study: MerlinStudy, run_mode: str = None): """ - Creates the task server interface for communicating the tasks. - - :param `study`: The MerlinStudy object - :param `run_mode`: The type of run mode, e.g. local, batch + Creates the task server interface for managing task communications. + + This function determines which server to send tasks to. It checks if + Celery is set as the task server; if not, it logs an error message. + The run mode can be specified to determine how tasks should be executed. + + Args: + study (study.study.MerlinStudy): The study object representing the + current experiment setup, containing configuration details for + the task server. + run_mode: The type of run mode to use for task execution. This can + include options such as 'local' or 'batch'. """ if study.expanded_spec.merlin["resources"]["task_server"] == "celery": run_celery(study, run_mode) @@ -83,14 +62,36 @@ def run_task_server(study, run_mode=None): LOG.error("Celery is not specified as the task server!") -def launch_workers(spec, steps, worker_args="", disable_logs=False, just_return_command=False): +def launch_workers( + spec: MerlinSpec, + steps: List[str], + worker_args: str = "", + disable_logs: bool = False, + just_return_command: bool = False, +) -> str: """ - Launches workers for the specified study. - - :param `specs`: Tuple of (YAMLSpecification, MerlinSpec) - :param `steps`: The steps in the spec to tie the workers to - :param `worker_args`: Optional arguments for the workers - :param `just_return_command`: Don't execute, just return the command + Launches workers for the specified study based on the provided + specification and steps. + + This function checks if Celery is configured as the task server + and initiates the specified workers accordingly. It provides options + for additional worker arguments, logging control, and command-only + execution without launching the workers. + + Args: + spec (spec.specification.MerlinSpec): Specification details + necessary for launching the workers. + steps: The specific steps in the specification that the workers + will be associated with. + worker_args: Additional arguments to be passed to the workers. + Defaults to an empty string. + disable_logs: Flag to disable logging during worker execution. + Defaults to False. + just_return_command: If True, the function will not execute the + command but will return it instead. Defaults to False. + + Returns: + A string of the worker launch command(s). """ if spec.merlin["resources"]["task_server"] == "celery": # pylint: disable=R1705 # Start workers @@ -101,15 +102,30 @@ def launch_workers(spec, steps, worker_args="", disable_logs=False, just_return_ return "No workers started" -def purge_tasks(task_server, spec, force, steps): +def purge_tasks(task_server: str, spec: MerlinSpec, force: bool, steps: List[str]) -> int: """ - Purges all tasks. - - :param `task_server`: The task server from which to purge tasks. - :param `spec`: A MerlinSpec object - :param `force`: Purge without asking for confirmation - :param `steps`: Space-separated list of stepnames defining queues to purge, - default is all steps + Purges all tasks from the specified task server. + + This function removes tasks from the designated queues associated + with the specified steps. It operates without confirmation if + the `force` parameter is set to True. The function logs the + steps being purged and checks if Celery is the configured task + server before proceeding. + + Args: + task_server: The task server from which to purge tasks. + spec (spec.specification.MerlinSpec): A + [`MerlinSpec`][spec.specification.MerlinSpec] object + containing the configuration needed to generate queue + specifications. + force: If True, purge the tasks without any confirmation prompt. + steps: A space-separated list of step names that define + which queues to purge. If not specified, defaults to purging + all steps. + + Returns: + The result of the purge operation; -1 if the task server is not + supported (i.e., not Celery). """ LOG.info(f"Purging queues for steps = {steps}") @@ -124,12 +140,18 @@ def purge_tasks(task_server, spec, force, steps): def dump_queue_info(task_server: str, query_return: List[Tuple[str, int, int]], dump_file: str): """ - Format the information we're going to dump in a way that the Dumper class can - understand and add a timestamp to the info. - - :param task_server: The task server from which to query queues - :param query_return: The output of `query_queues` - :param dump_file: The filepath of the file we'll dump queue info to + Formats and dumps queue information for the specified task server. + + This function prepares the queue data returned from the queue + query and formats it in a way that the [`Dumper`][common.dumper.Dumper] + class can process. It also adds a timestamp to the information before + dumping it to the specified file. + + Args: + task_server: The task server from which to query queues. + query_return: The output from the [`query_queues`][router.query_queues] + function, containing tuples of queue information. + dump_file: The filepath where the queue information will be dumped. """ if task_server == "celery": dump_celery_queue_info(query_return, dump_file) @@ -139,19 +161,35 @@ def dump_queue_info(task_server: str, query_return: List[Tuple[str, int, int]], def query_queues( task_server: str, - spec: "MerlinSpec", # noqa: F821 + spec: MerlinSpec, steps: List[str], specific_queues: List[str], verbose: bool = True, -): +) -> Dict[str, Dict[str, int]]: """ - Queries status of queues. - - :param task_server: The task server from which to query queues - :param spec: A MerlinSpec object or None - :param steps: Spaced-separated list of stepnames to query. Default is all - :param specific_queues: A list of queue names to query or None - :param verbose: A bool to determine whether to output log statements or not + Queries the status of queues from the specified task server. + + This function checks the status of queues tied to a given task server, + building a list of queues based on the provided steps and specific queue + names. It supports querying Celery task servers and returns the results + in a structured format. Logging behavior can be controlled with the verbose + parameter. + + Args: + task_server: The task server from which to query queues. + spec (spec.specification.MerlinSpec): A + [`MerlinSpec`][spec.specification.MerlinSpec] object used to define + the configuration of queues. Can also be None. + steps: A space-separated list of step names to query. Default is to query + all available steps if this is empty. + specific_queues: A list of specific queue names to query. Can be empty or + None to query all relevant queues. + verbose: If True, enables logging of query operations. Defaults to True. + + Returns: + A dictionary where the keys are queue names and the values are dictionaries + containing the number of workers (consumers) and tasks (jobs) attached + to each queue. """ if task_server == "celery": # pylint: disable=R1705 # Build a set of queues to query and query them @@ -159,14 +197,18 @@ def query_queues( return query_celery_queues(queues) else: LOG.error("Celery is not specified as the task server!") - return [] + return {} -def query_workers(task_server, spec_worker_names, queues, workers_regex): +def query_workers(task_server: str, spec_worker_names: List[str], queues: List[str], workers_regex: str): """ - Gets info from workers. + Retrieves information from workers associated with the specified task server. - :param `task_server`: The task server to query. + Args: + task_server: The task server to query. + spec_worker_names: A list of specific worker names to query. + queues: A list of queues to search for associated workers. + workers_regex: A regex pattern used to filter worker names during the query. """ LOG.info("Searching for workers...") @@ -176,12 +218,17 @@ def query_workers(task_server, spec_worker_names, queues, workers_regex): LOG.error("Celery is not specified as the task server!") -def get_workers(task_server): - """Get all workers. +def get_workers(task_server: str) -> List[str]: + """ + This function queries the designated task server to obtain a list of all + workers that are currently connected. + + Args: + task_server: The task server to query. - :param `task_server`: The task server to query. - :return: A list of all connected workers - :rtype: list + Returns: + A list of all connected workers. If the task server is not supported, + an empty list is returned. """ if task_server == "celery": # pylint: disable=R1705 return get_workers_from_app() @@ -190,14 +237,17 @@ def get_workers(task_server): return [] -def stop_workers(task_server, spec_worker_names, queues, workers_regex): +def stop_workers(task_server: str, spec_worker_names: List[str], queues: List[str], workers_regex: str): """ - Stops workers. - - :param `task_server`: The task server from which to stop workers. - :param `spec_worker_names`: Worker names to stop, drawn from a spec. - :param `queues` : The queues to stop - :param `workers_regex` : Regex for workers to stop + This function sends a command to stop workers that match the specified + criteria from the designated task server. + + Args: + task_server: The task server from which to stop workers. + spec_worker_names: A list of worker names to stop, as defined + in a specification. + queues: A list of queues from which to stop associated workers. + workers_regex: A regex pattern used to filter the workers to stop. """ LOG.info("Stopping workers...") @@ -208,42 +258,26 @@ def stop_workers(task_server, spec_worker_names, queues, workers_regex): LOG.error("Celery is not specified as the task server!") -def create_config(task_server: str, config_dir: str, broker: str, test: str) -> None: - """ - Create a config for the given task server. - - :param [str] `task_server`: The task server from which to stop workers. - :param [str] `config_dir`: Optional directory to install the config. - :param [str] `broker`: string indicated the broker, used to check for redis. - :param [str] `test`: string indicating if the app.yaml is used for testing. - """ - if test: - LOG.info("Creating test config ...") - else: - LOG.info("Creating config ...") - - if not os.path.isdir(config_dir): - os.makedirs(config_dir) - - if task_server == "celery": - config_file = "app.yaml" - data_config_file = "app.yaml" - if broker == "redis": - data_config_file = "app_redis.yaml" - elif test: - data_config_file = "app_test.yaml" - with resources.path("merlin.data.celery", data_config_file) as data_file: - create_celery_config(config_dir, config_file, data_file) - else: - LOG.error("Only celery can be configured currently.") +# TODO in Merlin 1.14 delete all of the below functions since we're deprecating the old version of the monitor +# and a lot of this stuff is in the new monitor classes def get_active_queues(task_server: str) -> Dict[str, List[str]]: """ - Get a dictionary of active queues and the workers attached to these queues. + Retrieve a dictionary of active queues and their associated workers for the specified task server. - :param `task_server`: The task server to query for active queues - :returns: A dict where keys are queue names and values are a list of workers watching them + This function queries the given task server for its active queues and gathers + information about which workers are currently monitoring these queues. It supports + the 'celery' task server and returns a structured dictionary containing the queue + names as keys and lists of worker names as values. + + Args: + task_server: The task server to query for active queues. + + Returns: + A dictionary where:\n + - The keys are the names of the active queues. + - The values are lists of worker names that are currently attached to those queues. """ active_queues = {} @@ -257,15 +291,25 @@ def get_active_queues(task_server: str) -> Dict[str, List[str]]: return active_queues -def wait_for_workers(sleep: int, task_server: str, spec: "MerlinSpec"): # noqa +def wait_for_workers(sleep: int, task_server: str, spec: MerlinSpec): # noqa """ - Wait on workers to start up. Check on worker start 10 times with `sleep` seconds between - each check. If no workers are started in time, raise an error to kill the monitor (there - was likely an issue with the task server that caused worker launch to fail). - - :param `sleep`: An integer representing the amount of seconds to sleep between each check - :param `task_server`: The task server from which to look for workers - :param `spec`: A MerlinSpec object representing the spec we're monitoring + Wait for workers to start up by checking their status at regular intervals. + + This function monitors the specified task server for the startup of worker processes. + It checks for the existence of the expected workers up to 10 times, sleeping for a + specified number of seconds between each check. If no workers are detected after + the maximum number of attempts, it raises an error to terminate the monitoring + process, indicating a potential issue with the task server. + + Args: + sleep: The number of seconds to pause between each check for worker status. + task_server: The task server from which to query for worker status. + spec (spec.specification.MerlinSpec): An instance of the + [`MerlinSpec`][spec.specification.MerlinSpec] class that contains the + specification for the workers being monitored. + + Raises: + NoWorkersException: If no workers are detected after the maximum number of checks. """ # Get the names of the workers that we're looking for worker_names = spec.get_worker_names() @@ -297,14 +341,17 @@ def check_workers_processing(queues_in_spec: List[str], task_server: str) -> boo """ Check if any workers are still processing tasks by querying the task server. - :param `queues_in_spec`: A list of queues to check if tasks are still active in - :param `task_server`: The task server from which to query - :returns: True if workers are still processing tasks, False otherwise + Args: + queues_in_spec: A list of queue names to check for active tasks. + task_server: The task server from which to query the processing status. + + Returns: + True if workers are still processing tasks, False otherwise. """ result = False if task_server == "celery": - from merlin.celery import app + from merlin.celery import app # pylint: disable=import-outside-toplevel result = check_celery_workers_processing(queues_in_spec, app) else: @@ -313,13 +360,23 @@ def check_workers_processing(queues_in_spec: List[str], task_server: str) -> boo return result -def check_merlin_status(args: "Namespace", spec: "MerlinSpec") -> bool: # noqa +def check_merlin_status(args: Namespace, spec: MerlinSpec) -> bool: """ - Function to check merlin workers and queues to keep the allocation alive + Function to check Merlin workers and queues to keep the allocation alive. + + This function monitors the status of workers and jobs within the specified task server + and the provided Merlin specification. It checks for active tasks and workers, ensuring + that the allocation remains valid. + + Args: + args: Parsed command-line interface arguments, including task server + specifications and sleep duration. + spec (spec.specification.MerlinSpec): The parsed spec.yaml as a + [`MerlinSpec`][spec.specification.MerlinSpec] object, containing queue + and worker definitions. - :param `args`: parsed CLI arguments - :param `spec`: the parsed spec.yaml as a MerlinSpec object - :returns: True if there are still tasks being processed, False otherwise + Returns: + True if there are still tasks being processed, False otherwise. """ # Initialize the variable to track if there are still active tasks active_tasks = False @@ -354,7 +411,7 @@ def check_merlin_status(args: "Namespace", spec: "MerlinSpec") -> bool: # noqa # If there are no workers, wait for the workers to start if total_consumers == 0: - wait_for_workers(args.sleep, args.task_server, spec) + wait_for_workers(args.sleep, args.task_server, spec=spec) # If we're here, workers have started and jobs should be queued if total_jobs > 0: diff --git a/merlin/server/__init__.py b/merlin/server/__init__.py index df2fc6e6a..d4e6aa54a 100644 --- a/merlin/server/__init__.py +++ b/merlin/server/__init__.py @@ -1,30 +1,15 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +""" +The `server` package defines the functionality for managing a containerized server +through Merlin. + +Modules: + server_commands.py: Main functions to interact with the server. + server_config.py: Server configuration functions. + server_util.py: Defines the structure of our server configurations. +""" diff --git a/merlin/server/server_commands.py b/merlin/server/server_commands.py index bb31d6b09..7ae2c1c04 100644 --- a/merlin/server/server_commands.py +++ b/merlin/server/server_commands.py @@ -1,34 +1,10 @@ -"""Main functions for instantiating and running Merlin server containers.""" +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +"""Main functions for instantiating and running Merlin server containers.""" import logging import os @@ -54,9 +30,9 @@ LOG = logging.getLogger("merlin") -def init_server() -> None: +def init_server(): """ - Initialize merlin server by checking and initializing main configuration directory + Initializes the Merlin server by setting up the main configuration directory and local server configuration. """ @@ -72,10 +48,18 @@ def init_server() -> None: def apply_config_changes(server_config: ServerConfig, args: Namespace): """ - Apply any configuration changes that the user is requesting. - - :param server_config: An instance of ServerConfig containing all the necessary configuration values - :param args: An argumentparser namespace object with args from the user + Applies configuration changes to the Merlin server based on user-provided arguments. + + This function modifies the Redis configuration and user settings as specified by + the user through the provided arguments. It updates various Redis settings, such as + IP address, port, password, directory, snapshot settings, append mode, and file paths. + If any changes are made, the updated configuration is written to the appropriate files. + + Args: + server_config (server.server_util.ServerConfig): An instance of `ServerConfig` containing + all the necessary configuration values for the server. + args (Namespace): An argparse `Namespace` object containing user-provided arguments + from the argument parser. """ redis_config = RedisConfig(server_config.container.get_config_path()) @@ -113,12 +97,14 @@ def apply_config_changes(server_config: ServerConfig, args: Namespace): # Pylint complains that there's too many branches in this function but # it looks clean to me so we'll ignore it -def config_server(args: Namespace) -> None: # pylint: disable=R0912 +def config_server(args: Namespace): # pylint: disable=R0912 """ - Process the merlin server config flags to make changes and edits to appropriate configurations - based on the input passed in by the user. + Processes the Merlin server configuration flags to make changes and edits + to appropriate configurations based on user input. - :param args: An argumentparser namespace object with args from the user + Args: + args (Namespace): An argparse `Namespace` object containing user-provided arguments + from the argument parser. """ server_config_before_changes = pull_server_config() if not server_config_before_changes: @@ -163,9 +149,12 @@ def config_server(args: Namespace) -> None: # pylint: disable=R0912 return None -def status_server() -> None: +def status_server(): """ - Get the server status of the any current running containers for merlin server + Retrieves and displays the current status of the Merlin server. + + This function checks the status of any running containers for the Merlin server + and logs appropriate messages based on the server's state. """ current_status = get_server_status() if current_status == ServerStatus.NOT_INITIALIZED: @@ -182,10 +171,13 @@ def status_server() -> None: def check_for_not_running_server() -> bool: """ - When starting a server the status must be NOT_RUNNING. If it's any other - status we need to log an error for the user to see. + Checks if the Merlin server status is `NOT_RUNNING` before starting a new server. - :returns: True if the status is NOT_RUNNING. False otherwise. + If the server status is anything other than `NOT_RUNNING`, logs an appropriate + error message to inform the user. + + Returns: + True if the server status is `NOT_RUNNING`, False otherwise. """ current_status = get_server_status() uninitialized_err = "Merlin server has not been intitialized. Please run 'merlin server init' first." @@ -205,10 +197,20 @@ def check_for_not_running_server() -> bool: def start_container(server_config: ServerConfig) -> subprocess.Popen: """ - Given a server configuration, use it to start up a container. + Starts a container based on the provided server configuration. + + This function uses the server configuration to locate the necessary image and + configuration files, validates their existence, and starts the container using + a subprocess. - :param server_config: The ServerConfig instance that holds information about the server to start. - :returns: A subprocess started with subprocess.Popen that's executing the command to start the container. + Args: + server_config (server.server_util.ServerConfig): An instance of `ServerConfig` + containing information about the server to start, including paths to the image + and configuration files. + + Returns: + A subprocess object representing the running container process, or `None` if + required files are missing. """ image_path = server_config.container.get_image_path() config_path = server_config.container.get_config_path() @@ -245,11 +247,21 @@ def start_container(server_config: ServerConfig) -> subprocess.Popen: def server_started(process: subprocess.Popen, server_config: ServerConfig) -> bool: """ - Check that the server spun up by `start_container` was started properly. + Verifies that the server started by [`start_container`][server.server_commands.start_container] + is running properly. + + This function checks the Redis output to ensure the server started successfully, + creates a process file for the container, and validates that the server status + is `RUNNING`. - :param process: The subprocess that was started by `start_container` - :param server_config: The ServerConfig instance that holds information about the redis server to start - :returns: True if the server started properly. False otherwise. + Args: + process (subprocess.Popen): The subprocess object representing the container + process started by `start_container`. + server_config (server.server_util.ServerConfig): An instance of `ServerConfig` + containing information about the Redis server configuration. + + Returns: + True if the server started successfully, False otherwise. """ redis_start, redis_out = parse_redis_output(process.stdout) @@ -277,8 +289,16 @@ def server_started(process: subprocess.Popen, server_config: ServerConfig) -> bo def start_server() -> bool: # pylint: disable=R0911 """ - Start a merlin server container using singularity. - :return:: True if server was successful started and False if failed. + Starts a Merlin server container. + + This function performs several steps to start the server, including checking + for an existing non-running server, pulling the server configuration, starting + the container, verifying the server startup, and applying Redis user and + configuration settings. It also generates a new `app.yaml` file for the server + configuration. + + Returns: + True if the server was successfully started, False otherwise. """ if not check_for_not_running_server(): return False @@ -310,10 +330,17 @@ def start_server() -> bool: # pylint: disable=R0911 return True -def stop_server(): +def stop_server() -> bool: """ - Stop running merlin server containers. - :return:: True if server was stopped successfully and False if failed. + Stops a running Merlin server container. + + This function checks the current server status, retrieves the server configuration, + and attempts to terminate the running server process. If successful, the server + process is stopped, and the function returns True. Otherwise, it logs errors + and returns False. + + Returns: + True if the server was successfully stopped, False otherwise. """ if get_server_status() != ServerStatus.RUNNING: LOG.info("There is no instance of merlin server running.") @@ -358,8 +385,13 @@ def stop_server(): def restart_server() -> bool: """ - Restart a running merlin server instance. - :return:: True if server was restarted successfully and False if failed. + Restarts a running Merlin server instance. + + This function stops the currently running Merlin server and then starts it again. + If the server is not running, it logs a message and returns False. + + Returns: + True if the server was successfully restarted, False otherwise. """ if get_server_status() != ServerStatus.RUNNING: LOG.info("Merlin server is not currently running.") diff --git a/merlin/server/server_config.py b/merlin/server/server_config.py index 36c2165bb..6e087ae3f 100644 --- a/merlin/server/server_config.py +++ b/merlin/server/server_config.py @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """This module represents everything that goes into server configuration""" import enum @@ -35,14 +12,15 @@ import random import string import subprocess +from importlib import resources from io import BufferedReader -from typing import Tuple +from typing import Dict, Tuple import yaml +from merlin.config.config_filepaths import MERLIN_HOME from merlin.server.server_util import ( CONTAINER_TYPES, - MERLIN_CONFIG_DIR, MERLIN_SERVER_CONFIG, MERLIN_SERVER_SUBDIR, AppYaml, @@ -52,16 +30,9 @@ ) -try: - from importlib import resources -except ImportError: - import importlib_resources as resources - - LOG = logging.getLogger("merlin") # Default values for configuration -CONFIG_DIR = os.path.abspath("./merlin_server/") IMAGE_NAME = "redis_latest.sif" PROCESS_FILE = "merlin_server.pf" CONFIG_FILE = "redis.conf" @@ -73,7 +44,14 @@ class ServerStatus(enum.Enum): """ - Different states in which the server can be in. + Represents different states that a server can be in. + + Attributes: + RUNNING (int): Indicates the server is running and operational. Numeric value: 0. + NOT_INITIALIZED (int): Indicates the server has not been initialized yet. Numeric value: 1. + MISSING_CONTAINER (int): Indicates the server is missing a required container. Numeric value: 2. + NOT_RUNNING (int): Indicates the server is not currently running. Numeric value: 3. + ERROR (int): Indicates the server encountered an error. Numeric value: 4. """ RUNNING = 0 @@ -85,11 +63,19 @@ class ServerStatus(enum.Enum): def generate_password(length, pass_command: str = None) -> str: """ - Function for generating passwords for redis container. If a specified command is given - then a password would be generated with the given command. If not a password will be - created by combining a string a characters based on the given length. + Generates a password for a Redis container. + + If a specific command is provided, the password will be generated using the output + of the given command. Otherwise, a random password will be created by combining + characters (letters, digits, and special symbols) based on the specified length. + + Args: + length (int): The desired length of the password. + pass_command (str, optional): A shell command to generate the password. + If provided, the command's output will be used as the password. - :return:: string value with given length + Returns: + The generated password. """ if pass_command: process = subprocess.run(pass_command, shell=True, capture_output=True, text=True) @@ -109,10 +95,19 @@ def generate_password(length, pass_command: str = None) -> str: def parse_redis_output(redis_stdout: BufferedReader) -> Tuple[bool, str]: """ - Parse the redis output for a the redis container. It will get all the necessary information - from the output and returns a dictionary of those values. + Parses the Redis output from a Redis container. - :return:: two values is_successful, dictionary of values from redis output + This function processes the Redis container's output to extract necessary information, + such as configuration details and server state. It determines whether the server was + successfully initialized and ready to accept connections, or if an error occurred. + + Args: + redis_stdout (BufferedReader): A buffered reader object containing the Redis container's output. + + Returns: + A tuple containing:\n + - A boolean indicating whether the server was successfully initialized and ready. + - A dictionary containing parsed configuration values if successful, or an error message otherwise. """ if redis_stdout is None: return False, "None passed as redis output" @@ -138,11 +133,13 @@ def parse_redis_output(redis_stdout: BufferedReader) -> Tuple[bool, str]: def copy_container_command_files(config_dir: str) -> bool: """ - Copy the yaml files that contain instructions on how to run certain commands - for each container type to the config directory. + Copies YAML files containing command instructions for container types to the specified configuration directory. + + Args: + config_dir (str): The path to the configuration directory where the YAML files will be copied. - :param config_dir: The path to the configuration dir where we'll copy files. - :returns: True if successful. False otherwise. + Returns: + True if all files are successfully copied or already exist. False otherwise. """ files = [i + ".yaml" for i in CONTAINER_TYPES] for file in files: @@ -163,17 +160,23 @@ def copy_container_command_files(config_dir: str) -> bool: def create_server_config() -> bool: """ - Create main configuration file for merlin server in the - merlin configuration directory. If a configuration already - exists it will not replace the current configuration and exit. + Creates the main configuration file for the Merlin server in the Merlin configuration directory. - :return:: True if success and False if fail + This function checks for the existence of the Merlin configuration directory and creates a default + server configuration if none exists. It also copies necessary container command files, applies the + server configuration to `app.yaml`, and initializes the server configuration directory. If the + configuration already exists, it will not overwrite it. + + Returns: + True if the configuration is successfully created and applied. False otherwise. """ - if not os.path.exists(MERLIN_CONFIG_DIR): - LOG.error(f"Unable to find main merlin configuration directory at {MERLIN_CONFIG_DIR}") + # Check for ~/.merlin/ directory + if not os.path.exists(MERLIN_HOME): + LOG.error(f"Unable to find main merlin configuration directory at {MERLIN_HOME}") return False - config_dir = os.path.join(MERLIN_CONFIG_DIR, MERLIN_SERVER_SUBDIR) + # Create ~/.merlin/server/ directory if it doesn't already exist + config_dir = os.path.join(MERLIN_HOME, MERLIN_SERVER_SUBDIR) if not os.path.exists(config_dir): LOG.info("Unable to find exisiting server configuration.") LOG.info(f"Creating default configuration in {config_dir}") @@ -183,34 +186,39 @@ def create_server_config() -> bool: LOG.error(err) return False + # Copy container-specific yaml files to ~/.merlin/server/ if not copy_container_command_files(config_dir): return False - # Load Merlin Server Configuration and apply it to app.yaml + # Load Merlin Server Configuration and apply it to merlin_server/app.yaml with resources.path("merlin.server", MERLIN_SERVER_CONFIG) as merlin_server_config: with open(merlin_server_config) as f: # pylint: disable=C0103 main_server_config = yaml.load(f, yaml.Loader) - filename = LOCAL_APP_YAML if os.path.exists(LOCAL_APP_YAML) else AppYaml.default_filename - merlin_app_yaml = AppYaml(filename) - merlin_app_yaml.update_data(main_server_config) - merlin_app_yaml.write(filename) - LOG.info("Applying merlin server configuration to app.yaml") - server_config = pull_server_config() - if not server_config: - LOG.error('Try to run "merlin server init" again to reinitialize values.') - return False + server_config = load_server_config(main_server_config) + if not server_config: + LOG.error('Try to run "merlin server init" again to reinitialize values.') + return False - if not os.path.exists(server_config.container.get_config_dir()): - LOG.info("Creating merlin server directory.") - os.mkdir(server_config.container.get_config_dir()) + if not os.path.exists(server_config.container.get_config_dir()): + LOG.info("Creating merlin server directory.") + os.mkdir(server_config.container.get_config_dir()) + + LOG.info("Applying merlin server configuration to ./merlin_server/app.yaml") + server_app_yaml = AppYaml() # By default this will point to ./merlin_server/app.yaml + server_app_yaml.update_data(main_server_config) + server_app_yaml.write() return True def config_merlin_server(): """ - Configurate the merlin server with configurations such as username password and etc. + Configures the Merlin server with necessary settings, including username and password. + + This function sets up the Merlin server by generating and storing a password file, creating a user file, + and configuring Redis settings. If the password or user files already exist, it skips the respective + setup steps. The function ensures that default and environment-specific users are added to the user file. """ server_config = pull_server_config() @@ -246,22 +254,24 @@ def config_merlin_server(): return None -def pull_server_config() -> ServerConfig: +def load_server_config(server_config: Dict) -> ServerConfig: """ - Pull the main configuration file and corresponding format configuration file - as well. Returns the values as a dictionary. + Given a dictionary containing server configuration values, load them into a + [`ServerConfig`][server.server_util.ServerConfig] instance. + + Args: + server_config: A dictionary containing server configuration values. Should have + a 'container' entry and a 'process' entry. - :return: A instance of ServerConfig containing all the necessary configuration values. + Returns: + A `ServerConfig` object containing the loaded server configuration. """ return_data = {} + return_data.update(server_config) format_needed_keys = ["command", "run_command", "stop_command", "pull_command"] process_needed_keys = ["status", "kill"] - merlin_app_yaml = AppYaml(LOCAL_APP_YAML) - server_config = merlin_app_yaml.get_data() - return_data.update(server_config) - - config_dir = os.path.join(MERLIN_CONFIG_DIR, MERLIN_SERVER_SUBDIR) + config_dir = os.path.join(MERLIN_HOME, MERLIN_SERVER_SUBDIR) if "container" in server_config: if "format" in server_config["container"]: @@ -274,30 +284,52 @@ def pull_server_config() -> ServerConfig: return None return_data.update(format_data) else: - LOG.error(f'Unable to find "format" in {merlin_app_yaml.default_filename}') + LOG.error('Unable to find "format" in server_config object.') return None else: - LOG.error(f'Unable to find "container" object in {merlin_app_yaml.default_filename}') + LOG.error('Unable to find "container" in server_config object.') return None # Checking for process values that are needed for main functions and defaults if "process" not in server_config: - LOG.error(f"Process config not found in {merlin_app_yaml.default_filename}") + LOG.error('Unable to find "process" in server_config object.') return None for key in process_needed_keys: if key not in server_config["process"]: - LOG.error(f'Process necessary "{key}" command configuration not found in {merlin_app_yaml.default_filename}') + LOG.error(f'Process necessary "{key}" command configuration not found in server_config object.') return None return ServerConfig(return_data) +def pull_server_config(app_yaml_path: str = None) -> ServerConfig: + """ + Retrieves the main configuration file and its corresponding format configuration file for the Merlin server. + + This function reads the `app.yaml` configuration file and additional format-specific configuration files + to construct a complete configuration dictionary. It validates the presence of required keys in the format + and process configurations. If any required configuration is missing, an error is logged and `None` is returned. + + Returns: + An instance of [`ServerConfig`][server.server_util.ServerConfig] containing all necessary configuration values. + """ + app_yaml_file = app_yaml_path if app_yaml_path is not None else LOCAL_APP_YAML + merlin_app_yaml = AppYaml(app_yaml_file) + server_config = merlin_app_yaml.get_data() + return load_server_config(server_config) + + def pull_server_image() -> bool: """ - Fetch the server image using singularity. + Fetches the server image and ensures the necessary configuration files are in place. - :return: True if success and False if fail + This function retrieves the server image from a specified URL and saves it locally if it does not already exist. + Additionally, it copies the default Redis configuration file to the appropriate location if it is missing. + The function relies on the server configuration to determine the image URL, image path, and configuration file details. + + Returns: + True if the server image and configuration file are successfully set up, False if an error occurs. """ server_config = pull_server_config() if not server_config: @@ -337,15 +369,20 @@ def pull_server_image() -> bool: return True -def get_server_status(): +def get_server_status() -> ServerStatus: """ - Determine the status of the current server. - This function can be used to check if the servers - have been initalized, started, or stopped. - - :param `server_dir`: location of all server related files. - :param `image_name`: name of the image when fetched. - :return:: A enum value of ServerStatus describing its current state. + Determines the current status of the server. + + This function checks the server's state by verifying the existence of necessary files, + including configuration files, the container image, and the process file. It also checks + if the server process is actively running. + + Returns: + An enum value representing the server's current state:\n + - `ServerStatus.NOT_INITIALIZED`: The server has not been initialized. + - `ServerStatus.MISSING_CONTAINER`: The server container image is missing. + - `ServerStatus.NOT_RUNNING`: The server process is not running. + - `ServerStatus.RUNNING`: The server is actively running. """ server_config = pull_server_config() if not server_config: @@ -375,10 +412,18 @@ def get_server_status(): return ServerStatus.RUNNING -def check_process_file_format(data: dict) -> bool: +def check_process_file_format(data: Dict) -> bool: """ - Check to see if the process file has the correct format and contains the expected key values. - :return:: True if success and False if fail + Validates the format of a process file. + + This function checks if the given process file data (in dictionary format) contains all the + required keys: "parent_pid", "image_pid", "port", and "hostname". + + Args: + data (Dict): The process file data to validate. + + Returns: + True if the process file contains all required keys, False otherwise. """ required_keys = ["parent_pid", "image_pid", "port", "hostname"] for key in required_keys: @@ -387,11 +432,19 @@ def check_process_file_format(data: dict) -> bool: return True -def pull_process_file(file_path: str) -> dict: +def pull_process_file(file_path: str) -> Dict: """ - Pull the data from the process file. If one is found returns the data in a dictionary - if not returns None - :return:: Data containing in process file. + Reads and parses data from a process file. + + This function attempts to load the contents of a process file located at the specified + file path. If the file exists and its format is valid, the data is returned as a dictionary. + If the format is invalid or the file cannot be processed, `None` is returned. + + Args: + file_path (str): The path to the process file. + + Returns: + A dictionary containing the data from the process file if the format is valid. """ with open(file_path, "r") as f: # pylint: disable=C0103 data = yaml.load(f, yaml.Loader) @@ -400,10 +453,22 @@ def pull_process_file(file_path: str) -> dict: return None -def dump_process_file(data: dict, file_path: str): +def dump_process_file(data: Dict, file_path: str) -> bool: """ - Dump the process data from the dictionary to the specified file path. - :return:: True if success and False if fail + Writes process data to a specified file. + + This function takes a dictionary containing process data and writes it to the specified + file path in YAML format. Before writing, the function validates the format of the data. + If the data format is invalid, the function returns `False`. If the operation is successful, + it returns `True`. + + Args: + data (Dict): The process data to be written to the file. + file_path (str): The path to the file where the data will be written. + + Returns: + True if the data is successfully written to the file, False if the data + format is invalid or the operation fails. """ if not check_process_file_format(data): return False diff --git a/merlin/server/server_util.py b/merlin/server/server_util.py index 15f591264..ddf9b5e69 100644 --- a/merlin/server/server_util.py +++ b/merlin/server/server_util.py @@ -1,37 +1,15 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """Utils relating to merlin server""" import hashlib import logging import os +from typing import Dict, List import redis import yaml @@ -43,6 +21,7 @@ # Constants for main merlin server configuration values. CONTAINER_TYPES = ["singularity", "docker", "podman"] +CONFIG_DIR = os.path.abspath("./merlin_server/") MERLIN_CONFIG_DIR = os.path.join(os.path.expanduser("~"), ".merlin") MERLIN_SERVER_SUBDIR = "server/" MERLIN_SERVER_CONFIG = "merlin_server.yaml" @@ -50,7 +29,17 @@ def valid_ipv4(ip: str) -> bool: # pylint: disable=C0103 """ - Checks valid ip address + Validates whether a given string is a valid IPv4 address. + + An IPv4 address consists of four octets separated by dots, where each octet + is a number between 0 and 255 (inclusive). This function checks if the input + string meets these criteria. + + Args: + ip: The string to validate as an IPv4 address. + + Returns: + True if the input string is a valid IPv4 address, False otherwise. """ if not ip: return False @@ -68,7 +57,16 @@ def valid_ipv4(ip: str) -> bool: # pylint: disable=C0103 def valid_port(port: int) -> bool: """ - Checks valid network port + Validates whether a given integer is a valid network port number. + + A valid network port number is an integer in the range 1 to 65535 (inclusive). + This function checks if the provided port falls within this range. + + Args: + port: The port number to validate. + + Returns: + True if the port is valid, False otherwise. """ if 0 < port < 65536: return True @@ -78,116 +76,650 @@ def valid_port(port: int) -> bool: # Pylint complains about too many instance variables but it's necessary here so ignore class ContainerConfig: # pylint: disable=R0902 """ - ContainerConfig provides interface for parsing and interacting with the container value specified within - the merlin_server.yaml configuration file. Dictionary of the config values should be passed when initialized - to parse values. This can be done after parsing yaml to data dictionary. - If there are missing values within the configuration it will be populated with default values for - singularity container. - - Configuration contains values for setting up containers and storing values specific to each container. - Values that are stored consist of things within the local configuration directory as different runs - can have differnt configuration values. + A class for parsing and interacting with container configuration values. + + The `ContainerConfig` class provides an interface for handling container-related + configuration values specified in the `merlin_server.yaml` file. It initializes + with a dictionary of configuration values, allowing for default values to be + populated for a Singularity container if any values are missing. + + The configuration contains values for setting up containers and storing alues specific + to each container. These values are used to manage container setup and store configuration + details specific to each container run. + + Attributes: + FORMAT (str): Default container format (e.g., "singularity"). + IMAGE_TYPE (str): Default image type (e.g., "redis"). + IMAGE_NAME (str): Default image name (e.g., "redis_latest.sif"). + REDIS_URL (str): Default URL for the container image (e.g., "docker://redis"). + CONFIG_FILE (str): Default name of the configuration file (e.g., "redis.conf"). + CONFIG_DIR (str): Default path to the configuration directory. + PROCESS_FILE (str): Default name of the process file (e.g., "merlin_server.pf"). + PASSWORD_FILE (str): Default name of the password file (e.g., "redis.pass"). + USERS_FILE (str): Default name of the users file (e.g., "redis.users"). + + format (str): Container format, initialized with the provided data or default. + image_type (str): Image type, initialized with the provided data or default. + image (str): Image name, initialized with the provided data or default. + url (str): Image URL, initialized with the provided data or default. + config (str): Configuration file name, initialized with the provided data or default. + config_dir (str): Configuration directory path, initialized with the provided data or default. + pfile (str): Process file name, initialized with the provided data or default. + pass_file (str): Password file name, initialized with the provided data or default. + user_file (str): Users file name, initialized with the provided data or default. + + Methods: + __eq__: Determines equality between two `ContainerConfig` instances. + get_format: Returns the container format. + get_image_type: Returns the image type. + get_image_name: Returns the image name. + get_image_url: Returns the image URL. + get_image_path: Returns the full path to the image file. + get_config_name: Returns the name of the configuration file. + get_config_path: Returns the full path to the configuration file. + get_config_dir: Returns the configuration directory path. + get_pfile_name: Returns the name of the process file. + get_pfile_path: Returns the full path to the process file. + get_pass_file_name: Returns the name of the password file. + get_pass_file_path: Returns the full path to the password file. + get_user_file_name: Returns the name of the users file. + get_user_file_path: Returns the full path to the users file. + get_container_password: Reads and returns the container password from the password file. """ # Default values for configuration - FORMAT = "singularity" - IMAGE_TYPE = "redis" - IMAGE_NAME = "redis_latest.sif" - REDIS_URL = "docker://redis" - CONFIG_FILE = "redis.conf" - CONFIG_DIR = os.path.abspath("./merlin_server/") - PROCESS_FILE = "merlin_server.pf" - PASSWORD_FILE = "redis.pass" - USERS_FILE = "redis.users" - - format = FORMAT - image_type = IMAGE_TYPE - image = IMAGE_NAME - url = REDIS_URL - config = CONFIG_FILE - config_dir = CONFIG_DIR - pfile = PROCESS_FILE - pass_file = PASSWORD_FILE - user_file = USERS_FILE - - def __init__(self, data: dict) -> None: - self.format = data["format"] if "format" in data else self.FORMAT - self.image_type = data["image_type"] if "image_type" in data else self.IMAGE_TYPE - self.image = data["image"] if "image" in data else self.IMAGE_NAME - self.url = data["url"] if "url" in data else self.REDIS_URL - self.config = data["config"] if "config" in data else self.CONFIG_FILE - self.config_dir = os.path.abspath(data["config_dir"]) if "config_dir" in data else self.CONFIG_DIR - self.pfile = data["pfile"] if "pfile" in data else self.PROCESS_FILE - self.pass_file = data["pass_file"] if "pass_file" in data else self.PASSWORD_FILE - self.user_file = data["user_file"] if "user_file" in data else self.USERS_FILE - - def __eq__(self, other: "ContainerFormatConfig"): - """ - Equality magic method used for testing this class - - :param other: Another ContainerFormatConfig object to check if they're the same + FORMAT: str = "singularity" + IMAGE_TYPE: str = "redis" + IMAGE_NAME: str = "redis_latest.sif" + REDIS_URL: str = "docker://redis" + CONFIG_FILE: str = "redis.conf" + PROCESS_FILE: str = "merlin_server.pf" + PASSWORD_FILE: str = "redis.pass" + USERS_FILE: str = "redis.users" + + format: str = FORMAT + image_type: str = IMAGE_TYPE + image: str = IMAGE_NAME + url: str = REDIS_URL + config: str = CONFIG_FILE + config_dir: str = CONFIG_DIR + pfile: str = PROCESS_FILE + pass_file: str = PASSWORD_FILE + user_file: str = USERS_FILE + + def __init__(self, data: Dict): + """ + Initializes a `ContainerConfig` instance with configuration values. + + Take in a dictionary of configuration values and set up the attributes of the + `ContainerConfig` instance. If any values are missing from the provided dictionary, + default values for a Singularity container are used. + + Args: + data: A dictionary containing configuration values. Keys can include:\n + - `format` (str): Container format (e.g., "singularity"). + - `image_type` (str): Image type (e.g., "redis"). + - `image` (str): Image name (e.g., "redis_latest.sif"). + - `url` (str): URL for the container image (e.g., "docker://redis"). + - `config` (str): Name of the configuration file (e.g., "redis.conf"). + - `config_dir` (str): Path to the configuration directory. + - `pfile` (str): Name of the process file (e.g., "merlin_server.pf"). + - `pass_file` (str): Name of the password file (e.g., "redis.pass"). + - `user_file` (str): Name of the users file (e.g., "redis.users"). + """ + self.format: str = data["format"] if "format" in data else self.FORMAT + self.image_type: str = data["image_type"] if "image_type" in data else self.IMAGE_TYPE + self.image: str = data["image"] if "image" in data else self.IMAGE_NAME + self.url: str = data["url"] if "url" in data else self.REDIS_URL + self.config: str = data["config"] if "config" in data else self.CONFIG_FILE + self.config_dir: str = os.path.abspath(data["config_dir"]) if "config_dir" in data else CONFIG_DIR + self.pfile: str = data["pfile"] if "pfile" in data else self.PROCESS_FILE + self.pass_file: str = data["pass_file"] if "pass_file" in data else self.PASSWORD_FILE + self.user_file: str = data["user_file"] if "user_file" in data else self.USERS_FILE + + def __eq__(self, other: "ContainerConfig") -> bool: + """ + Checks equality between two `ContainerConfig` instances. + + This magic method overrides the equality operator (`==`) to compare two + `ContainerConfig` objects. It checks if all relevant attributes + of the two objects are equal. + + Args: + other: Another instance of `ContainerConfig` to compare against. + + Returns: + True if all attributes are equal between the two objects. False otherwise. + + Example: + ```python + >>> config1_data = { + ... "format": "singularity", + ... "image": "redis_latest.sif", + ... "config_dir": "/configs", + ... } + >>> config2_data = { + ... "format": "singularity", + ... "image": "redis_latest.sif", + ... "config_dir": "/configs", + ... } + >>> config3_data = { + ... "format": "singularity", + ... "image": "redis_latest.sif", + ... "config_dir": "/other_configs", + ... } + >>> config1 = ContainerConfig(config1_data) + >>> config2 = ContainerConfig(config2_data) + >>> config3 = ContainerConfig(config3_data) + >>> config1 == config2 + True + >>> config1 == config3 + False + ``` """ variables = ("format", "image_type", "image", "url", "config", "config_dir", "pfile", "pass_file", "user_file") return all(getattr(self, attr) == getattr(other, attr) for attr in variables) + def __str__(self) -> str: + """ + Returns a human-readable string representation of the ContainerConfig. + + This method provides a formatted, user-friendly view of the container + configuration, showing the key attributes in a readable format. + + Returns: + A human-readable string representation of the configuration. + """ + return ( + f"ContainerConfig:\n" + f" Format: {self.format}\n" + f" Image Type: {self.image_type}\n" + f" Image: {self.image}\n" + f" URL: {self.url}\n" + f" Config Dir: {self.config_dir}\n" + f" Config File: {self.config}\n" + f" Process File: {self.pfile}\n" + f" Password File: {self.pass_file}\n" + f" Users File: {self.user_file}" + ) + + def __repr__(self) -> str: + """ + Returns an unambiguous string representation of the ContainerConfig. + + This method provides a string that could be used to recreate the object, + showing the constructor call with the current configuration values. + + Returns: + An unambiguous string representation that shows how to recreate the object. + """ + config_dict = { + "format": self.format, + "image_type": self.image_type, + "image": self.image, + "url": self.url, + "config": self.config, + "config_dir": self.config_dir, + "pfile": self.pfile, + "pass_file": self.pass_file, + "user_file": self.user_file, + } + return f"ContainerConfig({config_dict!r})" + def get_format(self) -> str: - """Getter method to get the container format""" + """ + Retrieves the container format. + + This method returns the format of the container, which specifies the type + of container being used (e.g., "singularity", "docker"). + + Returns: + The container format. + + Example: + ```python + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_format() + 'singularity' + ``` + """ return self.format def get_image_type(self) -> str: - """Getter method to get the image type""" + """ + Retrieves the image type. + + This method returns the type of the container image, which typically + describes the application or service associated with the image + (e.g., "redis", "mysql"). + + Returns: + The image type. + + Example: + ```python + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_image_type() + 'redis' + ``` + """ return self.image_type def get_image_name(self) -> str: - """Getter method to get the image name""" + """ + Retrieves the image name. + + This method returns the name of the container image, which may include + the version or tag (e.g., "redis_latest.sif", "mysql:8.0"). + + Returns: + The image name. + + Example: + ```python + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_image_name() + 'redis_latest.sif' + ``` + """ return self.image def get_image_url(self) -> str: - """Getter method to get the image url""" + """ + Retrieves the URL of the image. + + This method returns the URL where the container image is hosted or can + be downloaded from (e.g., a public or private registry URL). + + Returns: + The URL of the container image. + + Example: + ```python + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_image_url() + 'docker://redis' + ``` + """ return self.url def get_image_path(self) -> str: - """Getter method to get the path to the image""" + """ + Retrieves the full path to the image file. + + This method constructs and returns the absolute path to the container + image by combining the configuration directory and the image name. + + Returns: + The full path to the container image file. + + Example: + ```python + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_image_path() + '/configs/redis_latest.sif' + ``` + """ return os.path.join(self.config_dir, self.image) def get_config_name(self) -> str: - """Getter method to get the configuration file name""" + """ + Retrieves the name of the configuration file. + + This method returns the name of the configuration file associated with + the container or application (e.g., "redis.conf", "my.cnf"). + + Returns: + The name of the configuration file. + + Example: + ```python + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_config_name() + 'redis.conf' + ``` + """ return self.config def get_config_path(self) -> str: - """Getter method to get the configuration file path""" + """ + Retrieves the full path to the configuration file. + + This method constructs and returns the absolute path to the configuration + file by combining the configuration directory and the configuration file name. + + Returns: + The full path to the configuration file. + + Example: + ```python + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_config_path() + '/configs/redis.conf' + ``` + """ return os.path.join(self.config_dir, self.config) def get_config_dir(self) -> str: - """Getter method to get the configuration directory""" + """ + Retrieves the path to the configuration directory. + + This method returns the directory where configuration files are stored, + which can be used as a base path for accessing specific configuration files. + + Returns: + The path to the configuration directory. + + Example: + ```python + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_config_dir() + '/configs' + ``` + """ return self.config_dir def get_pfile_name(self) -> str: - """Getter method to get the process file name""" + """ + Retrieves the name of the process file. + + This method returns the name of the process file, which may represent + a file used to store process-related information (e.g., PID files or + other runtime data files). + + Returns: + The name of the process file. + + Example: + ```python + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_pfile_name() + 'merlin_server.pf' + ``` + """ return self.pfile def get_pfile_path(self) -> str: - """Getter method to get the process file path""" + """ + Retrieves the full path to the process file. + + This method constructs and returns the absolute path to the process file + by combining the configuration directory and the process file name. + + Returns: + The full path to the process file. + + Example: + ```python + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_pfile_path() + '/configs/merlin_server.pf' + ``` + """ return os.path.join(self.config_dir, self.pfile) def get_pass_file_name(self) -> str: - """Getter method to get the password file name""" + """ + Retrieves the name of the password file. + + This method returns the name of the password file, which is typically used + to store sensitive information such as user credentials or authentication keys. + + Returns: + The name of the password file. + + Example: + ```python + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_pass_file_name() + 'redis.pass' + ``` + """ return self.pass_file def get_pass_file_path(self) -> str: - """Getter method to get the password file path""" + """ + Retrieves the full path to the password file. + + This method constructs and returns the absolute path to the password file + by combining the configuration directory and the password file name. + + Returns: + The full path to the password file. + + Example: + ```python + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_pass_file_path() + '/configs/redis.pass' + ``` + """ return os.path.join(self.config_dir, self.pass_file) def get_user_file_name(self) -> str: - """Getter method to get the user file name""" + """ + Retrieves the name of the user file. + + This method returns the name of the user file, which may be used to store + information related to users, such as user configurations or metadata. + + Returns: + The name of the user file. + + Example: + ```python + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_user_file_name() + 'redis.users' + ``` + """ return self.user_file def get_user_file_path(self) -> str: - """Getter method to get the user file path""" + """ + Retrieves the full path to the user file. + + This method constructs and returns the absolute path to the user file + by combining the configuration directory and the user file name. + + Returns: + The full path to the user file. + + Example: + ```python + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_user_file_path() + '/configs/redis.users' + ``` + """ return os.path.join(self.config_dir, self.user_file) def get_container_password(self) -> str: - """Getter method to get the container password""" + """ + Retrieves the password for the container. + + This method reads the container password from the password file, which is + located at the path returned by + [`get_pass_file_path`][server.server_util.ContainerConfig.get_pass_file_path]. + The password is read as plain text from the file. + + Returns: + The container password. + + Example: + ```python + >>> with open("redis.pass", "w") as passfile: + ... passfile.write("redis_password") + >>> config_data = { + ... "format": "singularity", + ... "image_type": "redis", + ... "image": "redis_latest.sif", + ... "url": "docker://redis", + ... "config": "redis.conf", + ... "config_dir": "/configs", + ... "pfile": "merlin_server.pf", + ... "pass_file": "redis.pass", + ... "user_file": "redis.users", + ... } + >>> container_config = ContainerConfig(config_data) + >>> container_config.get_container_password() + 'redis_password' + ``` + """ password = None with open(self.get_pass_file_path(), "r") as f: # pylint: disable=C0103 password = f.read() @@ -196,85 +728,356 @@ def get_container_password(self) -> str: class ContainerFormatConfig: """ - ContainerFormatConfig provides an interface for parsing and interacting with container specific - configuration files .yaml. These configuration files contain container specific - commands to run containerizers such as singularity, docker, and podman. + `ContainerFormatConfig` provides an interface for parsing and interacting with container-specific + configuration files, such as .yaml. These configuration files define + container-specific commands for containerizers like Singularity, Docker, and Podman. + + This class allows you to customize and retrieve commands for running, stopping, and pulling + container images. + + Attributes: + COMMAND (str): Default command for running the container (default is "singularity"). + RUN_COMMAND (str): Default template for the run command. + STOP_COMMAND (str): Default command for stopping the container (default is "kill"). + PULL_COMMAND (str): Default template for the pull command. + + command (str): The container command, initialized from the configuration data or defaulting to `COMMAND`. + run_command (str): The run command, initialized from the configuration data or defaulting to `RUN_COMMAND`. + stop_command (str): The stop command, initialized from the configuration data or defaulting to `STOP_COMMAND`. + pull_command (str): The pull command, initialized from the configuration data or defaulting to `PULL_COMMAND`. + + Methods: + __eq__: Compares two `ContainerFormatConfig` objects for equality. + get_command: Retrieves the container command. + get_run_command: Retrieves the run command. + get_stop_command: Retrieves the stop command. + get_pull_command: Retrieves the pull command. """ - COMMAND = "singularity" - RUN_COMMAND = "{command} run {image} {config}" - STOP_COMMAND = "kill" - PULL_COMMAND = "{command} pull {image} {url}" + COMMAND: str = "singularity" + RUN_COMMAND: str = "{command} run {image} {config}" + STOP_COMMAND: str = "kill" + PULL_COMMAND: str = "{command} pull {image} {url}" - command = COMMAND - run_command = RUN_COMMAND - stop_command = STOP_COMMAND - pull_command = PULL_COMMAND + command: str = COMMAND + run_command: str = RUN_COMMAND + stop_command: str = STOP_COMMAND + pull_command: str = PULL_COMMAND - def __init__(self, data: dict) -> None: - self.command = data["command"] if "command" in data else self.COMMAND - self.run_command = data["run_command"] if "run_command" in data else self.RUN_COMMAND - self.stop_command = data["stop_command"] if "stop_command" in data else self.STOP_COMMAND - self.pull_command = data["pull_command"] if "pull_command" in data else self.PULL_COMMAND + def __init__(self, data: Dict): + """ + Initializes a `ContainerFormatConfig` object with container-specific configuration data. + + This constructor takes a dictionary of configuration data and initializes the container + command attributes. If any of the keys are missing in the provided data, default values + are used instead. + + Args: + data: A dictionary containing configuration data. Expected keys are:\n + - `command` (str): The container command (e.g., "singularity", "docker"). + - `run_command` (str): The template for the run command. + - `stop_command` (str): The command to stop the container. + - `pull_command` (str): The template for the pull command. + """ + self.command: str = data["command"] if "command" in data else self.COMMAND + self.run_command: str = data["run_command"] if "run_command" in data else self.RUN_COMMAND + self.stop_command: str = data["stop_command"] if "stop_command" in data else self.STOP_COMMAND + self.pull_command: str = data["pull_command"] if "pull_command" in data else self.PULL_COMMAND - def __eq__(self, other: "ContainerFormatConfig"): + def __eq__(self, other: "ContainerFormatConfig") -> bool: """ - Equality magic method used for testing this class + Checks equality between two `ContainerFormatConfig` objects. + + This method compares the attributes of the current object with those of another + `ContainerFormatConfig` object to determine if they are equal. + + Args: + other: Another instance of the `ContainerFormatConfig` class to compare against. - :param other: Another ContainerFormatConfig object to check if they're the same + Returns: + True if all corresponding attributes are equal between the two objects, otherwise False. + + Example: + ```python + >>> config1 = ContainerFormatConfig({"command": "singularity"}) + >>> config2 = ContainerFormatConfig({"command": "singularity"}) + >>> config1 == config2 + True + ``` """ variables = ("command", "run_command", "stop_command", "pull_command") return all(getattr(self, attr) == getattr(other, attr) for attr in variables) + def __str__(self) -> str: + """ + Returns a human-readable string representation of the ContainerFormatConfig. + + This method provides a formatted, user-friendly view of the container format + configuration, showing the key command attributes in a readable format. + + Returns: + A human-readable string representation of the configuration. + """ + return ( + f"ContainerFormatConfig:\n" + f" Command: {self.command}\n" + f" Run Command: {self.run_command}\n" + f" Stop Command: {self.stop_command}\n" + f" Pull Command: {self.pull_command}" + ) + + def __repr__(self) -> str: + """ + Returns an unambiguous string representation of the ContainerFormatConfig. + + This method provides a string that could be used to recreate the object, + showing the constructor call with the current configuration values. + + Returns: + An unambiguous string representation that shows how to recreate the object. + """ + config_dict = { + "command": self.command, + "run_command": self.run_command, + "stop_command": self.stop_command, + "pull_command": self.pull_command, + } + return f"ContainerFormatConfig({config_dict!r})" + def get_command(self) -> str: - """Getter method to get the container command""" + """ + Retrieves the container command. + + This method returns the value of the `command` attribute, + which specifies the container command (e.g., "singularity", "docker"). + + Returns: + The container command. + + Example: + ```python + >>> config = ContainerFormatConfig( + ... command="docker", + ... run_command="docker run --name my_container", + ... stop_command="docker stop my_container", + ... pull_command="docker pull my_image" + ... ) + >>> config.get_command() + 'docker' + ``` + """ return self.command def get_run_command(self) -> str: - """Getter method to get the run command""" + """ + Retrieves the run command. + + This method returns the value of the `run_command` attribute, + which specifies the template or command used to run the container. + + Returns: + The run command. + + Example: + ```python + >>> config = ContainerFormatConfig( + ... command="docker", + ... run_command="docker run --name my_container", + ... stop_command="docker stop my_container", + ... pull_command="docker pull my_image" + ... ) + >>> config.get_run_command() + 'docker run --name my_container' + ``` + """ return self.run_command def get_stop_command(self) -> str: - """Getter method to get the stop command""" + """ + Retrieves the stop command. + + This method returns the value of the `stop_command` attribute, + which specifies the command used to stop the container. + + Returns: + The stop command. + + Example: + ```python + >>> config = ContainerFormatConfig( + ... command="docker", + ... run_command="docker run --name my_container", + ... stop_command="docker stop my_container", + ... pull_command="docker pull my_image" + ... ) + >>> config.get_stop_command() + 'docker stop my_container' + ``` + """ return self.stop_command def get_pull_command(self) -> str: - """Getter method to get the pull command""" + """ + Retrieves the pull command. + + This method returns the value of the `pull_command` attribute, + which specifies the template or command used to pull the container image. + + Returns: + The pull command. + + Example: + ```python + >>> config = ContainerFormatConfig( + ... command="docker", + ... run_command="docker run --name my_container", + ... stop_command="docker stop my_container", + ... pull_command="docker pull my_image" + ... ) + >>> config.get_pull_command() + 'docker pull my_image' + ``` + """ return self.pull_command class ProcessConfig: """ - ProcessConfig provides an interface for parsing and interacting with process config specified - in merlin_server.yaml configuration. This configuration provide commands for interfacing with - host machine while the containers are running. + `ProcessConfig` provides an interface for parsing and interacting with process configuration + specified in the `merlin_server.yaml` configuration file. This configuration defines commands + for interacting with the host machine while containers are running, such as checking the status + of processes or terminating them. + + Attributes: + STATUS_COMMAND (str): Default template for the status command, which checks if a process + is running using its parent process ID (PID). Default is "pgrep -P {pid}". + KILL_COMMAND (str): Default template for the kill command, which terminates a process + using its PID. Default is "kill {pid}". + + status (str): The status command template to check the status of a process. This is + initialized from the provided configuration or defaults to `STATUS_COMMAND`. + kill (str): The kill command template to terminate a process. This is initialized from + the provided configuration or defaults to `KILL_COMMAND`. + + Methods: + __eq__: Compares two ProcessConfig objects for equality based on their `status` and + `kill` attributes. + get_status_command: Retrieves the status command template. + get_kill_command: Retrieves the kill command template. """ - STATUS_COMMAND = "pgrep -P {pid}" - KILL_COMMAND = "kill {pid}" + STATUS_COMMAND: str = "pgrep -P {pid}" + KILL_COMMAND: str = "kill {pid}" + + status: str = STATUS_COMMAND + kill: str = KILL_COMMAND - status = STATUS_COMMAND - kill = KILL_COMMAND + def __init__(self, data: Dict): + """ + Initializes the ProcessConfig object with custom or default process commands. - def __init__(self, data: dict) -> None: - self.status = data["status"] if "status" in data else self.STATUS_COMMAND - self.kill = data["kill"] if "kill" in data else self.KILL_COMMAND + This constructor takes a dictionary containing configuration data and initializes + the `status` and `kill` attributes. If the keys `status` or `kill` are not present + in the provided dictionary, their values default to `STATUS_COMMAND` and `KILL_COMMAND`, + respectively. - def __eq__(self, other: "ProcessConfig"): + Args: + data: A dictionary containing process configuration. Expected keys are:\n + - "status": A string representing the status command template. + - "kill": A string representing the kill command template. """ - Equality magic method used for testing this class + self.status: str = data["status"] if "status" in data else self.STATUS_COMMAND + self.kill: str = data["kill"] if "kill" in data else self.KILL_COMMAND - :param other: Another ProcessConfig object to check if they're the same + def __eq__(self, other: "ProcessConfig") -> bool: + """ + Checks equality between two `ProcessConfig` objects. + + This method compares the attributes of the current object with those of another + `ProcessConfig` object to determine if they are equal. + + Args: + other: Another instance of the `ProcessConfig` class to compare with. + + Returns: + `True` if the `status` and `kill` attributes of both instances are equal, + otherwise `False`. + + Example: + ```python + >>> config1 = ProcessConfig({"status": "check_status", "kill": "terminate_process"}) + >>> config2 = ProcessConfig({"status": "check_status", "kill": "terminate_process"}) + >>> config3 = ProcessConfig({"status": "check_status", "kill": "stop_process"}) + >>> config1 == config2 + True + >>> config1 == config3 + False + ``` """ variables = ("status", "kill") return all(getattr(self, attr) == getattr(other, attr) for attr in variables) + def __str__(self) -> str: + """ + Returns a human-readable string representation of the ProcessConfig. + + This method provides a formatted, user-friendly view of the process + configuration, showing the status and kill command templates in a readable format. + + Returns: + A human-readable string representation of the configuration. + """ + return f"ProcessConfig:\n" f" Status Command: {self.status}\n" f" Kill Command: {self.kill}" + + def __repr__(self) -> str: + """ + Returns an unambiguous string representation of the ProcessConfig. + + This method provides a string that could be used to recreate the object, + showing the constructor call with the current configuration values. + + Returns: + An unambiguous string representation that shows how to recreate the object. + """ + config_dict = {"status": self.status, "kill": self.kill} + return f"ProcessConfig({config_dict!r})" + def get_status_command(self) -> str: - """Getter method to get the status command""" + """ + Retrieves the status command for the process. + + This method returns the command used to check the status of the process + managed by the `ProcessConfig` instance. + + Returns: + The status command as a string. + + Example: + ```python + >>> process_config = ProcessConfig(status="ps aux | grep process_name", kill="kill -9 process_id") + >>> process_config.get_status_command() + 'ps aux | grep process_name' + ``` + """ return self.status def get_kill_command(self) -> str: - """Getter method to get the kill command""" + """ + Retrieves the kill command for the process. + + This method returns the command used to terminate the process + managed by the `ProcessConfig` instance. + + Returns: + The kill command as a string. + + Example: + ```python + >>> process_config = ProcessConfig(status="ps aux | grep process_name", kill="kill -9 process_id") + >>> process_config.get_kill_command() + 'kill -9 process_id' + ``` + """ return self.kill @@ -282,19 +1085,101 @@ def get_kill_command(self) -> str: # classes so we can ignore it class ServerConfig: # pylint: disable=R0903 """ - ServerConfig is an interface for storing all the necessary configuration for merlin server. - These configuration container things such as ContainerConfig, ProcessConfig, and ContainerFormatConfig. + `ServerConfig` is an interface for storing all the necessary configuration for the Merlin server. + + This class encapsulates configurations related to containers, processes, and container formats, + making it easier to manage and access these settings in a structured way. + + Attributes: + container (ContainerConfig): Configuration related to the container. + process (ProcessConfig): Configuration related to the process. + container_format (ContainerFormatConfig): Configuration for the container format. """ container: ContainerConfig = None process: ProcessConfig = None container_format: ContainerFormatConfig = None - def __init__(self, data: dict) -> None: - self.container = ContainerConfig(data["container"]) if "container" in data else None - self.process = ProcessConfig(data["process"]) if "process" in data else None - container_format_data = data.get(self.container.get_format() if self.container else None) - self.container_format = ContainerFormatConfig(container_format_data) if container_format_data else None + def __init__(self, data: Dict): + """ + Initializes a ServerConfig instance with the provided configuration data. + + Args: + data: A dictionary containing configuration data. Expected keys include:\n + - `container` ([`ContainerConfig`][server.server_util.ContainerConfig]): Configuration data for the container. + - `process` ([`ProcessConfig`][server.server_util.ProcessConfig]): Configuration data for the process. + - `container_format` ([`ContainerFormatConfig`][server.server_util.ContainerFormatConfig]): Configuration + data for the container format. + """ + self.container: ContainerConfig = ContainerConfig(data["container"]) if "container" in data else None + self.process: ProcessConfig = ProcessConfig(data["process"]) if "process" in data else None + container_format_data: str = data.get(self.container.get_format() if self.container else None) + self.container_format: ContainerFormatConfig = ( + ContainerFormatConfig(container_format_data) if container_format_data else None + ) + + def __str__(self) -> str: + """ + Returns a human-readable string representation of the ServerConfig. + + This method provides a formatted, user-friendly view of the complete server + configuration, showing all nested configuration objects in a readable hierarchical format. + + Returns: + A human-readable string representation of the server configuration. + """ + lines = ["ServerConfig:"] + + if self.container: + lines.append(" Container Configuration:") + container_str = str(self.container) + # Indent each line of the container config + for line in container_str.split("\n")[1:]: # Skip the first line (class name) + lines.append(" " + line) + else: + lines.append(" Container Configuration: None") + + if self.process: + lines.append(" Process Configuration:") + process_str = str(self.process) + # Indent each line of the process config + for line in process_str.split("\n")[1:]: # Skip the first line (class name) + lines.append(" " + line) + else: + lines.append(" Process Configuration: None") + + if self.container_format: + lines.append(" Container Format Configuration:") + format_str = str(self.container_format) + # Indent each line of the container format config + for line in format_str.split("\n")[1:]: # Skip the first line (class name) + lines.append(" " + line) + else: + lines.append(" Container Format Configuration: None") + + return "\n".join(lines) + + def __repr__(self) -> str: + """ + Returns an unambiguous string representation of the ServerConfig. + + This method provides a string that shows the constructor call with the current + configuration values. Since ServerConfig contains complex nested objects, this + representation focuses on showing the structure rather than being directly evaluable. + + Returns: + An unambiguous string representation that shows the server configuration structure. + """ + container_repr = repr(self.container) if self.container else "None" + process_repr = repr(self.process) if self.process else "None" + container_format_repr = repr(self.container_format) if self.container_format else "None" + + return ( + f"ServerConfig(" + f"container={container_repr}, " + f"process={process_repr}, " + f"container_format={container_format_repr})" + ) class RedisConfig: @@ -302,20 +1187,110 @@ class RedisConfig: RedisConfig is an interface for parsing and interacing with redis.conf file that is provided by redis. This allows users to parse the given redis configuration and make edits and allow users to write those changes into a redis readable config file. + + `RedisConfig` is an interface for parsing and interacting with a Redis configuration file (`redis.conf`). + + This class allows users to: + - Parse an existing Redis configuration file. + - Modify configuration settings such as IP address, port, password, snapshot settings, directories, etc. + - Write the updated configuration back to a file in a Redis-readable format. + + Attributes: + filename (str): The path to the Redis configuration file. + changed (bool): A flag indicating whether any changes have been made to the configuration. + entry_order (List): A list maintaining the order of configuration entries as they appear in the file. + entries (Dict): A dictionary storing configuration keys and their corresponding values. + comments (Dict): A dictionary storing comments associated with each configuration entry. + trailing_comments (str): Any comments that appear at the end of the configuration file. + + Methods: + parse: Parses the Redis configuration file and populates the `entries`, `comments`, and `entry_order` + attributes. + write: Writes the current configuration (including comments) back to the file. + set_filename: Updates the filename of the configuration file. + set_config_value: Updates the value of a given configuration key. + get_config_value: Retrieves the value of a given configuration key. + changes_made: Returns whether any changes have been made to the configuration. + get_ip_address: Retrieves the IP address (`bind`) setting from the configuration. + set_ip_address: Validates and sets the IP address (`bind`) in the configuration. + get_port: Retrieves the port setting from the configuration. + set_port: Validates and sets the port in the configuration. + set_password: Sets the Redis password (`requirepass`) in the configuration. + get_password: Retrieves the Redis password (`requirepass`) from the configuration. + set_directory: Sets the save directory (`dir`) in the configuration. Creates the directory if it + does not exist. + set_snapshot: Updates the snapshot settings (`save`) for the configuration. + set_snapshot_file: Sets the snapshot file name (`dbfilename`) in the configuration. + set_append_mode: Sets the append mode (`appendfsync`) in the configuration. + set_append_file: Sets the append file name (`appendfilename`) in the configuration. """ - def __init__(self, filename) -> None: - self.filename = filename - self.changed = False - self.entry_order = [] - self.entries = {} - self.comments = {} - self.trailing_comments = "" - self.changed = False + def __init__(self, filename: str): + """ + Initializes a `RedisConfig` instance and parses the given Redis configuration file. + + Args: + filename: The path to the Redis configuration file (`redis.conf`). + + Notes: + - Parses the configuration file immediately after initialization by calling the `parse()` method. + - Populates the `entries`, `comments`, and `entry_order` attributes based on the file's contents. + """ + self.filename: str = filename + self.changed: bool = False + self.entry_order: List[str] = [] + self.entries: Dict[str, str] = {} + self.comments: Dict[str, str] = {} + self.trailing_comments: str = "" + self.changed: bool = False self.parse() - def parse(self) -> None: - """Parses the redis configuration file""" + def parse(self): + """ + Parses the Redis configuration file and populates the configuration data. + + This method reads the Redis configuration file specified by `self.filename` and extracts: + + - Configuration entries (key-value pairs). + - Associated comments for each entry. + - The order of entries as they appear in the file. + - Any trailing comments at the end of the file. + + Behavior: + - Lines starting with `#` are treated as comments and stored in the `comments` dictionary or `trailing_comments`. + - Non-comment lines are split into key-value pairs and stored in the `entries` dictionary. + - The order of configuration keys is preserved in the `entry_order` list. + - Handles duplicate keys by appending additional parts to the key (e.g., for special cases like `save`). + + Attributes Updated: + - `self.entries`: Stores configuration key-value pairs. + - `self.comments`: Stores comments associated with each configuration key. + - `self.entry_order`: Preserves the order of configuration keys as they appear in the file. + - `self.trailing_comments`: Stores any comments that appear at the end of the file. + + Example: + Assume we have a Redis configuration file named "redis.conf" with the following content: + ``` + # Redis configuration file + maxmemory 256mb + save 900 1 + # End of configuration + ``` + + Then we'd see the following: + ```python + >>> config = RedisConfig("redis.conf") + >>> config.parse() + >>> print(config.entries) + {'maxmemory': '256mb', 'save 900': 1} + >>> print(config.comments) + {'maxmemory': '# Redis configuration file\\n', 'save 900': ''} + >>> print(config.entry_order) + ['maxmemory', 'save 900'] + >>> print(config.trailing_comments) + '# End of configuration' + ``` + """ self.entries = {} self.comments = {} with open(self.filename, "r+") as f: # pylint: disable=C0103 @@ -337,20 +1312,88 @@ def parse(self) -> None: comments += line + "\n" self.trailing_comments = comments[:-1] - def write(self) -> None: - """Writes to the redis configuration file""" + def write(self): + """ + Writes the current configuration and comments back to the Redis configuration file. + + This method writes the configuration entries, their associated comments, and any + trailing comments to the file specified by `self.filename`. The order of entries is + preserved as per the `self.entry_order` list. + + Example: + Assume we start with a Redis configuration file named "redis.conf" with the following content: + ``` + # Redis configuration file + maxmemory 256mb + save 900 1 + # End of configuration + ``` + + We then update it with the write method: + ```python + >>> config = RedisConfig("redis.conf") + >>> config.set_config_value("maxmemory", "512mb") + >>> config.write() + ``` + + Which updates our "redis.conf" file to be: + ``` + # Redis configuration file + maxmemory 512mb + save 900 1 + # End of configuration + ``` + """ with open(self.filename, "w") as f: # pylint: disable=C0103 for entry in self.entry_order: f.write(self.comments[entry]) f.write(f"{entry} {self.entries[entry]}\n") f.write(self.trailing_comments) - def set_filename(self, filename: str) -> None: - """Setter method to set the filename""" + def set_filename(self, filename: str): + """ + Sets a new filename for the Redis configuration file. + + Args: + filename: The new file path to be set as the Redis configuration file. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> print(config.filename) + 'redis.conf' + >>> config.set_filename("/path/to/new/redis.conf") + >>> print(config.filename) + '/path/to/new/redis.conf' + ``` + """ self.filename = filename def set_config_value(self, key: str, value: str) -> bool: - """Changes a configuration value""" + """ + Updates the value of a specific configuration key. + + This method changes the value of an existing configuration key in the `entries` dictionary. + If the key does not exist, the method returns `False`. If the key is updated successfully, + the `changed` attribute is set to `True`. + + Args: + key: The configuration key to update. + value: The new value to set for the specified key. + + Returns: + True if the key exists and the value is successfully updated. False otherwise. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> success = config.set_config_value("maxmemory", "512mb") + >>> if success: + ... print("Configuration updated successfully!") + ... else: + ... print("Key not found in the configuration.") + ``` + """ if key not in self.entries: return False self.entries[key] = value @@ -358,21 +1401,102 @@ def set_config_value(self, key: str, value: str) -> bool: return True def get_config_value(self, key: str) -> str: - """Given an entry in the config, get the value""" + """ + Retrieves the value of a specific configuration key. + + This method looks up the value of the specified key in the `entries` dictionary + and returns it. If the key does not exist, the method returns `None`. + + Args: + key: The configuration key to retrieve the value for. + + Returns: + The value associated with the specified key as a string, or `None` if the key is not found. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> value = config.get_config_value("maxmemory") + >>> print(value) + '256mb' + >>> value = config.get_config_value("nonexistent_key") + >>> print(value) + None + ``` + """ if key in self.entries: return self.entries[key] return None def changes_made(self) -> bool: - """Getter method to get the changes made""" + """ + Checks if any changes have been made to the configuration. + + This method returns the value of the `self.changed` attribute, which indicates + whether any configuration values have been modified since the last parse or write. + + Returns: + True if changes have been made to the configuration, False otherwise. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> print(config.changes_made()) + False + >>> config.set_config_value("maxmemory", "512mb") + >>> print(config.changes_made()) + True + ``` + """ return self.changed def get_ip_address(self) -> str: - """Getter method to get the ip from the redis config""" + """ + Retrieves the IP address bound in the Redis configuration. + + This method uses the [`get_config_value`][server.server_util.RedisConfig.get_config_value] + method to fetch the value of the `bind` key from the configuration. The `bind` key typically + specifies the IP address that Redis binds to. If the `bind` key is not present in the configuration, + the method returns `None`. + + Returns: + The IP address as a string if the `bind` key exists, or `None` if it does not. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> ip_address = config.get_ip_address() + >>> print(ip_address) + '127.0.0.1' + ``` + """ return self.get_config_value("bind") def set_ip_address(self, ipaddress: str) -> bool: - """Validates and sets a given ip address""" + """ + Validates and sets the given IP address in the Redis configuration. + + This method checks if the provided IP address is a valid IPv4 address. + If valid, it updates the `bind` key in the Redis configuration with the new IP address. + If the IP address is invalid or the update fails, the method logs an error and returns `False`. + + Args: + ipaddress: The IP address to set in the Redis configuration. + + Returns: + True if the IP address is successfully validated and set, False otherwise. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> success = config.set_ip_address("192.168.1.1") + >>> print(success) + True + >>> success = config.set_ip_address("invalid_ip") + >>> print(success) + False + ``` + """ if ipaddress is None: return False # Check if ipaddress is valid @@ -388,11 +1512,52 @@ def set_ip_address(self, ipaddress: str) -> bool: return True def get_port(self) -> str: - """Getter method to get the port from the redis config""" + """ + Retrieves the port number from the Redis configuration. + + This method fetches the value of the `port` key from the configuration + using the [`get_config_value`][server.server_util.RedisConfig.get_config_value] + method. If the `port` key is not present, the method returns `None`. + + Returns: + The port number as a string if the `port` key exists, or `None` if it does not. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> port = config.get_port() + >>> print(port) + '6379' + ``` + """ return self.get_config_value("port") def set_port(self, port: int) -> bool: - """Validates and sets a given port""" + """ + Validates and sets the given port number in the Redis configuration. + + This method checks if the provided port number is valid. If valid, it updates + the `port` key in the Redis configuration with the new port number. If the port + is invalid or the update fails, the method logs an error and returns `False`. + + Args: + port: The port number to set in the Redis configuration. + + Returns: + True if the port number is successfully validated and set, False otherwise. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> success = config.set_port(6379) + >>> print(success) + True + >>> success = config.set_port(99999) + ERROR: Invalid port given + >>> print(success) + False + ``` + """ if port is None: return False # Check if port is valid @@ -408,7 +1573,26 @@ def set_port(self, port: int) -> bool: return True def set_password(self, password: str) -> bool: - """Changes the password""" + """ + Sets a new password in the Redis configuration. + + This method updates the `requirepass` key in the Redis configuration with the provided password. + If the password is `None`, the method returns `False` without making any changes. + + Args: + password: The new password to set in the Redis configuration. + + Returns: + True if the password is successfully set, False otherwise. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> success = config.set_password("my_secure_password") + >>> print(success) + True + ``` + """ if password is None: return False self.set_config_value("requirepass", password) @@ -416,13 +1600,47 @@ def set_password(self, password: str) -> bool: return True def get_password(self) -> str: - """Getter method to get the config password""" + """ + Retrieves the password from the Redis configuration. + + This method fetches the value of the `requirepass` key from the configuration + using the [`get_config_value`][server.server_util.RedisConfig.get_config_value] + method. If the `requirepass` key is not present, the method returns `None`. + + Returns: + The password as a string if the `requirepass` key exists, or `None` if it does not. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> password = config.get_password() + >>> print(password) + 'my_secure_password' + ``` + """ return self.get_config_value("requirepass") def set_directory(self, directory: str) -> bool: """ - Sets the save directory in the redis config file. - Creates the directory if necessary. + Sets the save directory in the Redis configuration file. + + This method updates the `dir` key in the Redis configuration with the provided directory path. + If the directory does not exist, it is created. If the directory is `None` or the update fails, + the method logs an error and returns `False`. + + Args: + directory: The directory path to set as the save directory in the Redis configuration. + + Returns: + True if the directory is successfully validated, created (if necessary), and set, False otherwise. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> success = config.set_directory("/var/lib/redis") + >>> print(success) + True + ``` """ if directory is None: return False @@ -439,14 +1657,36 @@ def set_directory(self, directory: str) -> bool: def set_snapshot(self, seconds: int = None, changes: int = None) -> bool: """ - Sets the 'seconds' and/or 'changes' values of the snapshot setting, - depending on what the user requests. - - :param seconds: The first value of snapshot to change. If we're leaving it the - same this will be None. - :param changes: The second value of snapshot to change. If we're leaving it the - same this will be None. - :returns: True if successful, False otherwise. + Updates the snapshot configuration in the Redis settings. + + This method allows you to set the snapshot parameters, which determine + when Redis creates a snapshot of the dataset. The snapshot is triggered + based on a combination of time (`seconds`) and the number of changes + (`changes`) made to the dataset. If either parameter is `None`, it will + remain unchanged. + + Args: + seconds: The time interval (in seconds) after which a snapshot should be created. + If `None`, the existing value for `seconds` remains unchanged. + changes: The number of changes to the dataset that trigger a snapshot. + If `None`, the existing value for `changes` remains unchanged. + + Returns: + True if the snapshot configuration is successfully updated, False otherwise. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> success = config.set_snapshot(seconds=300, changes=10) + >>> print(success) + True + >>> success = config.set_snapshot(seconds=600) + >>> print(success) + True + >>> success = config.set_snapshot() # No changes + >>> print(success) + False + ``` """ # If both values are None, this method is doing nothing @@ -479,7 +1719,27 @@ def set_snapshot(self, seconds: int = None, changes: int = None) -> bool: return True def set_snapshot_file(self, file: str) -> bool: - """Sets the snapshot file""" + """ + Sets the name of the snapshot file in the Redis configuration. + + This method updates the `dbfilename` parameter in the Redis configuration + with the provided file name. The snapshot file is where Redis saves the + dataset during a snapshot operation. + + Args: + file: The name of the snapshot file to set in the Redis configuration. + + Returns: + True if the snapshot file name is successfully set, False otherwise. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> success = config.set_snapshot_file("dump.rdb") + >>> print(success) + True + ``` + """ if file is None: return False # Set the snapshot file in the redis config @@ -491,7 +1751,30 @@ def set_snapshot_file(self, file: str) -> bool: return True def set_append_mode(self, mode: str) -> bool: - """Sets the append mode""" + """ + Sets the append mode in the Redis configuration. + + The append mode determines how Redis handles data persistence to the append-only file (AOF). + Valid modes are: + + - "always": Redis appends data to the AOF after every write operation. + - "everysec": Redis appends data to the AOF every second (default and recommended). + - "no": Disables AOF persistence. + + Args: + mode: The append mode to set. Must be one of "always", "everysec", or "no". + + Returns: + True if the append mode is successfully set, False otherwise. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> success = config.set_append_mode("everysec") + >>> print(success) + True + ``` + """ if mode is None: return False valid_modes = ["always", "everysec", "no"] @@ -510,7 +1793,27 @@ def set_append_mode(self, mode: str) -> bool: return True def set_append_file(self, file: str) -> bool: - """Sets the append file""" + """ + Sets the name of the append-only file (AOF) in the Redis configuration. + + The append-only file is used by Redis for data persistence in append-only mode. + This method updates the `appendfilename` parameter in the Redis configuration + with the provided file name. + + Args: + file: The name of the append-only file to set in the Redis configuration. + + Returns: + True if the append file name is successfully set, False otherwise. + + Example: + ```python + >>> config = RedisConfig("redis.conf") + >>> success = config.set_append_file("appendonly.aof") + >>> print(success) + True + ``` + """ if file is None: return False # Set the append file in the redis config @@ -523,35 +1826,107 @@ def set_append_file(self, file: str) -> bool: class RedisUsers: """ - RedisUsers provides an interface for parsing and interacting with redis.users configuration - file. Allow users and merlin server to create, remove, and edit users within the redis files. - Changes can be sync and push to an exisiting redis server if one is available. + `RedisUsers` provides an interface for parsing and interacting with a Redis `users` configuration + file. This class allows users and the Merlin server to create, remove, and edit user entries + within the Redis configuration files. Changes can be synchronized and pushed to an existing + Redis server if one is available. + + Attributes: + filename (str): The path to the Redis user configuration file. + users (Dict[str, User]): A dictionary + containing user data, where keys are usernames and values are instances of the + `User` class. + + Methods: + parse: Parses the Redis user configuration file and populates the `users` dictionary with + [`User`][server.server_util.RedisUsers.User] objects. + write: Writes the current users' data back to the Redis user configuration file. + add_user: Adds a new user to the `users` dictionary. + set_password: Sets a new password for a specific user. + remove_user: Removes a user from the `users` dictionary. + apply_to_redis: Applies the current user configuration to a running Redis server by synchronizing + the local user data with the Redis server's ACL configuration. """ class User: - """Embedded class to store user specific information""" + """ + An embedded class that represents an individual Redis user with attributes and methods for managing + user-specific data. + + Attributes: + status (str): The status of the user, either "on" (enabled) or "off" (disabled). + hash_password (str): The hashed password of the user. + keys (str): The keys the user has access to (e.g., "*" for all keys). + channels (str): The channels the user can access (e.g., "*" for all channels). + commands (str): The commands the user is allowed to execute (e.g., "@all" for all commands). + + Methods: + parse_dict: Parses a dictionary of user data and updates the `User` object's attributes. + get_user_dict: Returns a dictionary representation of the `User` object. + __repr__(): Returns a string representation of the `User` object. + __str__(): Returns a string representation of the `User` object (same as `__repr__`). + set_password: Sets the user's hashed password based on the provided plaintext password. + """ - status = "on" - hash_password = hashlib.sha256(b"password").hexdigest() - keys = "*" - channels = "*" - commands = "@all" + status: str = "on" + hash_password: str = hashlib.sha256(b"password").hexdigest() + keys: str = "*" + channels: str = "*" + commands: str = "@all" def __init__( # pylint: disable=R0913 - self, status="on", keys="*", channels="*", commands="@all", password=None - ) -> None: - self.status = status - self.keys = keys - self.channels = channels - self.commands = commands + self, + status: str = "on", + keys: str = "*", + channels: str = "*", + commands: str = "@all", + password: str = None, + ): + """ + Initializes a `User` object with the provided attributes. + + Args: + status: The status of the user, either "on" (enabled) or "off" (disabled). + keys: The keys the user has access to (e.g., "*" for all keys). + channels: The channels the user can access (e.g., "*" for all channels). + commands: The commands the user is allowed to execute (e.g., "@all" for all commands). + password: The plaintext password for the user. If provided, it will be hashed and stored. + """ + self.status: str = status + self.keys: str = keys + self.channels: str = channels + self.commands: str = commands if password is not None: self.set_password(password) - def parse_dict(self, dictionary: dict) -> None: + def parse_dict(self, dictionary: Dict[str, str]): """ - Given a dict of user info, parse the dict and store - the values as class attributes. - :param `dictionary`: The dict to parse + Parses a dictionary containing user information and updates the attributes of the `User` object. + + Args: + dictionary: A dictionary containing user data. Expected keys are:\n + - `status` (str): The user's status ("on" or "off"). + - `keys` (str): The keys the user can access. + - `channels` (str): The channels the user can access. + - `commands` (str): The commands the user can execute. + - `hash_password` (str): The hashed password of the user. + + Example: + ```python + >>> user_data = { + ... "status": "on", + ... "keys": "*", + ... "channels": "*", + ... "commands": "@all", + ... "hash_password": "hashed_password_value" + ... } + >>> user = User() + >>> user.parse_dict(user_data) + >>> print(user.status) + 'on' + >>> print(user.hash_password) + 'hashed_password_value' + ``` """ self.status = dictionary["status"] self.keys = dictionary["keys"] @@ -559,8 +1934,27 @@ def parse_dict(self, dictionary: dict) -> None: self.commands = dictionary["commands"] self.hash_password = dictionary["hash_password"] - def get_user_dict(self) -> dict: - """Getter method to get the user info""" + def get_user_dict(self) -> Dict: + """ + Returns a dictionary representation of the `User` object. + + Returns: + A dictionary containing the user's attributes. + + Example: + ```python + >>> user = User(status="on", keys="*", channels="*", commands="@all", password="secure_password") + >>> user_dict = user.get_user_dict() + >>> print(user_dict) + { + "status": "on", + "hash_password": "hashed_password_value", + "keys": "*", + "channels": "*", + "commands": "@all" + } + ``` + """ self.status = "on" return { "status": self.status, @@ -571,27 +1965,89 @@ def get_user_dict(self) -> dict: } def __repr__(self) -> str: - """Repr magic method for User class""" + """ + Returns a string representation of the `User` object for debugging purposes. + + Returns: + A string representation of the user's attributes in dictionary format. + """ return str(self.get_user_dict()) def __str__(self) -> str: - """Str magic method for User class""" + """ + Returns a string representation of the `User` object. + + Returns: + A string representation of the user's attributes in dictionary format. + """ return self.__repr__() - def set_password(self, password: str) -> None: - """Setter method to set the user's hash password""" + def set_password(self, password: str): + """ + Sets the user's hashed password based on the provided plaintext password. + + Args: + password: The plaintext password to hash and store. + + Example: + ```python + >>> user = User() + >>> user.set_password("secure_password") + >>> print(user.hash_password) + # Output: The hashed value of "secure_password" + ``` + """ self.hash_password = hashlib.sha256(bytes(password, "utf-8")).hexdigest() - filename = "" - users = {} + filename: str = "" + users: Dict[str, User] = {} - def __init__(self, filename) -> None: - self.filename = filename + def __init__(self, filename: str): + """ + Initializes a `RedisUsers` object and parses the Redis user configuration file if it exists. + + Args: + filename: The path to the Redis user configuration file. + """ + self.filename: str = filename if os.path.exists(self.filename): self.parse() - def parse(self) -> None: - """Parses the redis user configuration file""" + def parse(self): + """ + Parses the Redis user configuration file and populates the `users` dictionary. + + This method reads the YAML configuration file specified by `self.filename` and converts + the user data into a dictionary. Each user entry is converted into an instance of the + [`User`][server.server_util.RedisUsers.User] class, which stores the user's attributes. + + Example: + Assume `users.yaml` contains: + ``` + user1: + status: "on" + hash_password: "hashed_password_1" + keys: "*" + channels: "*" + commands: "@all" + user2: + status: "off" + hash_password: "hashed_password_2" + keys: "key1" + channels: "channel1" + commands: "command1" + ``` + + This would then be parsed like so: + ```python + >>> redis_users = RedisUsers("users.yaml") + >>> redis_users.parse() + >>> print(redis_users.users["user1"].status) + 'on' + >>> print(redis_users.users["user2"].commands) + 'command1' + ``` + """ with open(self.filename, "r") as f: # pylint: disable=C0103 self.users = yaml.load(f, yaml.Loader) for user in self.users: @@ -599,8 +2055,30 @@ def parse(self) -> None: new_user.parse_dict(self.users[user]) self.users[user] = new_user - def write(self) -> None: - """Writes to the redis user configuration file""" + def write(self): + """ + Writes the current `users` dictionary back to the Redis user configuration file. + + This method converts the `users` dictionary, which contains [`User`][server.server_util.RedisUsers.User] + objects, into a format suitable for storage in the YAML configuration file. The file + specified by `self.filename` is then updated with the current user data. + + Example: + ```python + >>> redis_users = RedisUsers("users.yaml") + >>> redis_users.add_user( + ... user="new_user", + ... status="on", + ... keys="*", + ... channels="*", + ... commands="@all", + ... password="secure_password" + ... ) + >>> redis_users.write() + ``` + + After calling `write`, the `users.yaml` file will be updated with the new user's data. + """ data = self.users.copy() for key in data: data[key] = self.users[key].get_user_dict() @@ -608,30 +2086,121 @@ def write(self) -> None: yaml.dump(data, f, yaml.Dumper) def add_user( # pylint: disable=R0913 - self, user, status="on", keys="*", channels="*", commands="@all", password=None + self, + user: str, + status: str = "on", + keys: str = "*", + channels: str = "*", + commands: str = "@all", + password: str = None, ) -> bool: - """Add a user to the dict of Redis users""" + """ + Adds a new user to the dictionary of Redis users. + + Args: + user: The username of the new user. + status: The status of the user, either "on" (enabled) or "off" (disabled). + keys: The keys the user has access to (e.g., "*" for all keys). + channels: The channels the user can access (e.g., "*" for all channels). + commands: The commands the user is allowed to execute (e.g., "@all" for all commands). + password: The plaintext password for the user. If provided, it will be hashed and stored. + + Returns: + True if the user was successfully added, False if the user already exists. + + Example: + ```python + >>> redis_users = RedisUsers("users.yaml") + >>> success = redis_users.add_user( + ... user="new_user", + ... status="on", + ... keys="*", + ... channels="*", + ... commands="@all", + ... password="secure_password" + ... ) + >>> print(success) + True + ``` + """ if user in self.users: return False self.users[user] = self.User(status, keys, channels, commands, password) return True - def set_password(self, user: str, password: str): - """Set the password for a specific user""" + def set_password(self, user: str, password: str) -> bool: + """ + Sets the password for an existing user. + + Args: + user: The username of the user whose password is to be updated. + password: The plaintext password to hash and store for the user. + + Returns: + True if the password was successfully updated, False if the user does not exist. + + Example: + ```python + >>> redis_users = RedisUsers("users.yaml") + >>> redis_users.add_user("existing_user", password="old_password") + >>> success = redis_users.set_password("existing_user", "new_password") + >>> print(success) + True + ``` + """ if user not in self.users: return False self.users[user].set_password(password) return True def remove_user(self, user: str) -> bool: - """Remove a user from the dict of users""" + """ + Removes a user from the dictionary of Redis users. + + Args: + user: The username of the user to be removed. + + Returns: + True if the user was successfully removed, False if the user does not exist. + + Example: + ```python + >>> redis_users = RedisUsers("users.yaml") + >>> redis_users.add_user("user_to_remove", password="password") + >>> success = redis_users.remove_user("user_to_remove") + >>> print(success) + True + ``` + """ if user in self.users: del self.users[user] return True return False - def apply_to_redis(self, host: str, port: int, password: str) -> None: - """Apply the changes to users to redis""" + def apply_to_redis(self, host: str, port: int, password: str): + """ + Applies the user configuration changes to a Redis instance. + + This method synchronizes the current user configuration stored in `self.users` + with the Redis instance specified by the provided connection details. It performs + the following actions: + + - Adds or updates users in Redis based on the `self.users` dictionary. + - Removes users from Redis that are not present in `self.users`. + + Args: + host: The hostname or IP address of the Redis server. + port: The port number of the Redis server. + password: The password for authenticating with the Redis server. + + Example: + ```python + >>> redis_users = RedisUsers("users.yaml") + >>> redis_users.add_user("user1", password="password1") + >>> redis_users.add_user("user2", password="password2") + >>> redis_users.apply_to_redis(host="127.0.0.1", port=6379, password="redis_password") + ``` + """ database = redis.Redis(host=host, port=port, password=password) current_users = database.acl_users() for user in self.users: @@ -653,50 +2222,146 @@ def apply_to_redis(self, host: str, port: int, password: str) -> None: class AppYaml: """ - AppYaml allows for an structured way to interact with any app.yaml main merlin configuration file. - It helps to parse each component of the app.yaml and allow users to edit, configure and write the - file. + `AppYaml` provides a structured way to interact with the main `app.yaml` configuration file for Merlin. + This class allows users to parse, edit, configure, and write the `app.yaml` file, which contains + the application's main configuration settings. + + Attributes: + default_filename (str): The default file path for the `app.yaml` configuration file. + data (Dict): A dictionary that holds the parsed configuration data from the `app.yaml` file. + broker_name (str): The key name in the configuration file representing the broker settings. + results_name (str): The key name in the configuration file representing the results backend settings. + + Methods: + apply_server_config: Updates the `data` dictionary with Redis server configuration details based on + the provided `ServerConfig` object. + update_data: Updates the `data` dictionary with new entries. + get_data: Returns the current configuration data stored in the `data` attribute. + read: Reads a YAML file and populates the `data` attribute with its contents. + write: Writes the current `data` dictionary to the specified YAML file. """ - default_filename = os.path.join(MERLIN_CONFIG_DIR, "app.yaml") - data = {} - broker_name = "broker" - results_name = "results_backend" + default_filename: str = os.path.join(CONFIG_DIR, "app.yaml") + data: Dict = {} + broker_name: str = "broker" + results_name: str = "results_backend" + + def __init__(self, filename: str = default_filename): + """ + Initializes the `AppYaml` object and loads the configuration file. - def __init__(self, filename: str = default_filename) -> None: + Args: + filename: The path to the `app.yaml` file. If the file does not exist, + the default file path (`default_filename`) is used. + """ if not os.path.exists(filename): filename = self.default_filename + LOG.debug(f"Reading configuration from {filename}") self.read(filename) def apply_server_config(self, server_config: ServerConfig): - """Store the redis configuration""" + """ + Updates the `data` dictionary with Redis server configuration details + based on the provided `ServerConfig` object. + + Args: + server_config: An object containing the server configuration. + + Example: + ```python + >>> server_config = ServerConfig(...) + >>> app_yaml = AppYaml() + >>> app_yaml.apply_server_config(server_config) + ``` + """ redis_config = RedisConfig(server_config.container.get_config_path()) + if self.broker_name not in self.data: + self.data[self.broker_name] = {} + self.data[self.broker_name]["name"] = server_config.container.get_image_type() self.data[self.broker_name]["username"] = "default" self.data[self.broker_name]["password"] = server_config.container.get_pass_file_path() self.data[self.broker_name]["server"] = redis_config.get_ip_address() self.data[self.broker_name]["port"] = redis_config.get_port() + if self.results_name not in self.data: + self.data[self.results_name] = {} + self.data[self.results_name]["name"] = server_config.container.get_image_type() self.data[self.results_name]["username"] = "default" self.data[self.results_name]["password"] = server_config.container.get_pass_file_path() self.data[self.results_name]["server"] = redis_config.get_ip_address() self.data[self.results_name]["port"] = redis_config.get_port() - def update_data(self, new_data: dict): - """Update the data dict with new entries""" + def update_data(self, new_data: Dict): + """ + Updates the `data` dictionary with new entries. + + Args: + new_data: A dictionary containing the new data to be merged + into the existing `data` dictionary. + + Example: + ```python + >>> new_data = {"custom_key": {"sub_key": "value"}} + >>> app_yaml = AppYaml() + >>> app_yaml.update_data(new_data) + ``` + """ self.data.update(new_data) - def get_data(self): - """Getter method to obtain the data""" + def get_data(self) -> Dict: + """ + Retrieves the current configuration data stored in the `data` attribute. + + Returns: + The current configuration data. + + Example: + ```python + >>> app_yaml = AppYaml() + >>> current_data = app_yaml.get_data() + ``` + """ return self.data def read(self, filename: str = default_filename): - """Load in a yaml file and save it to the data attribute""" - self.data = merlin.utils.load_yaml(filename) + """ + Reads a YAML file and populates the `data` attribute with its contents. + + Args: + filename: The path to the YAML file to be read. If not provided, + the default file path (`default_filename`) is used. + + Example: + ```python + >>> app_yaml = AppYaml() + >>> app_yaml.read("custom_app.yaml") + ``` + """ + try: + self.data = merlin.utils.load_yaml(filename) + except FileNotFoundError: + self.data = {} def write(self, filename: str = default_filename): - """Given a filename, dump the data to the file""" - with open(filename, "w+") as f: # pylint: disable=C0103 - yaml.dump(self.data, f, yaml.Dumper) + """ + Writes the current `data` dictionary to the specified YAML file. + + Args: + filename: The path to the YAML file where the data should be written. + If not provided, the default file path (`default_filename`) is used. + + Example: + ```python + >>> app_yaml = AppYaml() + >>> app_yaml.write("output_app.yaml") + ``` + """ + try: + with open(filename, "w+") as f: # pylint: disable=C0103 + yaml.dump(self.data, f, yaml.Dumper) + except FileNotFoundError: + with open(filename, "w") as f: # pylint: disable=C0103 + yaml.dump(self.data, f, yaml.Dumper) diff --git a/merlin/spec/__init__.py b/merlin/spec/__init__.py index 37cabcad1..de04a002b 100644 --- a/merlin/spec/__init__.py +++ b/merlin/spec/__init__.py @@ -1,29 +1,22 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +The `spec` package defines the structure, defaults, and functionality for working with +Merlin specification files. + +Modules: + all_keys.py: Defines all the valid keys for each block in a Merlin specification file, + ensuring consistency and validation. + defaults.py: Provides the default values for each block in a spec file, enabling workflows + to execute even when fields are omitted. + expansion.py: Handles the expansion of variables within a spec file, including user-defined, + environment, and reserved variables, as well as parameter substitutions. + override.py: Supports overriding variables in a spec file via the command-line interface, + with functions for validation and replacement. + specification.py: Contains the `MerlinSpec` class, which represents the raw data from a + Merlin specification file and provides methods for interacting with it. +""" diff --git a/merlin/spec/all_keys.py b/merlin/spec/all_keys.py index 706111851..66c679ce3 100644 --- a/merlin/spec/all_keys.py +++ b/merlin/spec/all_keys.py @@ -1,33 +1,19 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### -"""This module defines all the keys possible in each block of a merlin spec file""" +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module defines all the possible keys for each block in a Merlin specification file. + +Merlin specification files are used to configure and manage studies, workflows, and tasks +in Merlin. Each block in the spec file corresponds to a specific aspect of the workflow, +such as batch settings, environment variables, study steps, parameters, and resources. +This module provides sets of valid keys for each block to ensure consistency and validation. + +This module serves as a reference for the structure and valid keys of Merlin specification files. +""" DESCRIPTION = {"description", "name"} diff --git a/merlin/spec/defaults.py b/merlin/spec/defaults.py index 1aa700607..3ab7c2fe1 100644 --- a/merlin/spec/defaults.py +++ b/merlin/spec/defaults.py @@ -1,33 +1,19 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### -"""This module defines the default values of every block in the merlin spec""" +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module defines the default values for every block in a Merlin specification file. + +Merlin specification files are used to configure workflows, tasks, and resources in +Merlin. This module provides the default values for each block in the spec file, ensuring +that workflows can execute even if certain fields are omitted by the user. These defaults +serve as a fallback mechanism to maintain consistency and simplify the configuration process. + +This module serves as a reference for the default configuration of Merlin specification files. +""" DESCRIPTION = {"description": {}} diff --git a/merlin/spec/expansion.py b/merlin/spec/expansion.py index b9f9eee65..a2fa173d9 100644 --- a/merlin/spec/expansion.py +++ b/merlin/spec/expansion.py @@ -1,40 +1,25 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### -"""This module handles expanding variables in the merlin spec""" +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module handles the expansion of variables in a Merlin spec file. + +It provides functionality to expand user-defined variables, environment variables, and reserved variables +within a spec file. The module also supports variable substitution for specific use cases, such as parameter +substitutions for samples and commands, and allows for the processing of override variables provided via +the command-line interface. +""" import logging from collections import ChainMap from copy import deepcopy from os.path import expanduser, expandvars +from typing import Dict, List, Tuple -from merlin.common.abstracts.enums import ReturnCode +from merlin.common.enums import ReturnCode from merlin.spec.override import error_override_vars, replace_override_vars from merlin.spec.specification import MerlinSpec from merlin.utils import contains_shell_ref, contains_token, verify_filepath @@ -66,10 +51,27 @@ LOG = logging.getLogger(__name__) -def var_ref(string): +def var_ref(string: str) -> str: """ - Given a string , return that string surrounded - by $(). + Format a string as a variable reference. + + This function takes a string, converts it to uppercase, and returns it + wrapped in the format `$()`. If the string already contains + a token (e.g., it is already formatted as a variable reference), a warning + is logged and the original string is returned unchanged. + + Args: + string: The input string to format as a variable reference. + + Returns: + The formatted variable reference, or the original string if it + already contains a token. + + Example: + ```python + >>> var_ref("example") + '$(EXAMPLE)' + ``` """ string = string.upper() if contains_token(string): @@ -78,10 +80,31 @@ def var_ref(string): return f"$({string})" -def expand_line(line, var_dict, env_vars=False): +def expand_line(line: str, var_dict: Dict[str, str], env_vars: bool = False) -> str: """ - Expand one line of text by substituting user variables, - optionally environment variables, as well as variables in 'var_dict'. + Expand a single line of text by substituting variables. + + This function replaces variable references in a given line of text with + their corresponding values from a provided dictionary. Optionally, it can + also expand environment variables and user home directory shortcuts (e.g., `~`). + + Args: + line: The input line of text to expand. + var_dict: A dictionary of variable names and their corresponding values + to substitute in the line. + env_vars: If True, environment variables and home directory shortcuts + will also be expanded. + + Returns: + The expanded line of text with all applicable substitutions applied. + + Example: + ```python + >>> line = "Path: $(VAR1) and $(VAR2)" + >>> var_dict = {"VAR1": "/path/to/dir", "VAR2": "/another/path"} + >>> expand_line(line, var_dict) + 'Path: /path/to/dir and /another/path' + ``` """ # fmt: off if ( @@ -99,10 +122,30 @@ def expand_line(line, var_dict, env_vars=False): return line -def expand_by_line(text, var_dict): +def expand_by_line(text: str, var_dict: Dict[str, str]) -> str: """ - Given a text (yaml spec), and a dictionary of variable names - and values, expand variables in the text line by line. + Expand variables in a text line by line. + + This function processes a multi-line text (e.g., a YAML specification) and + replaces variable references in each line using a provided dictionary of + variable names and their corresponding values. + + Args: + text: The input multi-line text to process. + var_dict: A dictionary of variable names and their corresponding values + to substitute in the text. + + Returns: + The text with all applicable variable substitutions applied, processed + line by line. + + Example: + ```python + >>> text = "Path is $(VAR1) and stores $(VAR2)" + >>> var_dict = {"VAR1": "/path/to/dir", "VAR2": "value"} + >>> expand_by_line(text, var_dict) + 'Path is /path/to/dir and stores value' + ``` """ text = text.splitlines() result = "" @@ -112,11 +155,23 @@ def expand_by_line(text, var_dict): return result -def expand_env_vars(spec): +def expand_env_vars(spec: MerlinSpec) -> MerlinSpec: """ - Expand environment variables for all sections of a spec, except - for values with the key 'cmd' or 'restart' (these are executable - shell scripts, so environment variable expansion would be redundant). + Expand environment variables in all sections of a spec. + + This function processes all sections of a given spec object and expands + environment variables (e.g., `$HOME` or `~`) in string values. It skips + expansion for values associated with the keys 'cmd' or 'restart', as these + are typically shell scripts where environment variable expansion would + already occur during execution. + + Args: + spec (spec.specification.MerlinSpec): The spec object + containing sections to process. + + Returns: + (spec.specification.MerlinSpec): The updated spec object with environment variables expanded in all + applicable sections. """ def recurse(section): @@ -139,25 +194,35 @@ def recurse(section): return spec -def determine_user_variables(*user_var_dicts): +def determine_user_variables(*user_var_dicts: List[Dict]) -> Dict: """ - Given an arbitrary number of dictionaries, determine them - in order. - - param `user_var_dicts`: A list of dictionaries of user variables. - For example: - [variables, labels] - - A single user var dict may look like: - {'OUTPUT_PATH':'./studies', 'N_SAMPLES':10} - - This user var dict: - {'TARGET': 'target_dir', - 'PATH': '$(SPECROOT)/$(TARGET)'} - - ...would be determined as: - {'TARGET': 'target_dir', - 'PATH': '$(SPECROOT)/target_dir'} + Determine user-defined variables from multiple dictionaries. + + This function takes an arbitrary number of dictionaries containing user-defined + variables and resolves them in order, handling variable references and expansions + (e.g., environment variables, user home directory shortcuts). Variable names are + converted to uppercase, and reserved words cannot be reassigned. + + Args: + user_var_dicts: One or more dictionaries of user variables. Each dictionary + contains key-value pairs where the key is the variable name, and the value + is the variable's definition. + + Returns: + A dictionary of resolved user variables, with variable names in uppercase + and all references expanded. + + Raises: + ValueError: If a reserved word is attempted to be reassigned. + + Example: + ```python + >>> user_vars_1 = {'OUTPUT_PATH': './studies', 'N_SAMPLES': 10} + >>> user_vars_2 = {'TARGET': 'target_dir', 'PATH': '$(SPECROOT)/$(TARGET)'} + >>> determine_user_variables(user_vars_1, user_vars_2) + {'OUTPUT_PATH': './studies', 'N_SAMPLES': '10', + 'TARGET': 'target_dir', 'PATH': '$(SPECROOT)/target_dir'} + ``` """ all_var_dicts = dict(ChainMap(*user_var_dicts)) determined_results = {} @@ -175,15 +240,41 @@ def determine_user_variables(*user_var_dicts): return determined_results -def parameter_substitutions_for_sample(sample, labels, sample_id, relative_path_to_sample): +def parameter_substitutions_for_sample( + sample: List[str], labels: List[str], sample_id: int, relative_path_to_sample: str +) -> List[Tuple[str, str]]: """ - :param sample : The sample to do substitution for. - :param labels : The column labels of the sample. - :param sample_id : The merlin sample id for this sample. - :param relative_path_to_sample : The relative path to this sample. - - :return : list of pairs indicating what needs to be substituted for a - merlin sample + Generate parameter substitutions for a specific sample. + + This function creates a list of substitution pairs for a given sample, + mapping variable references (e.g., `$(LABEL)`) to their corresponding + values in the sample. It also includes metadata substitutions such as + the sample ID and the relative path to the sample. + + Args: + sample: A list of values representing the sample. + labels: A list of column labels corresponding to the sample values. + sample_id: The unique integer ID of the sample. + relative_path_to_sample: The relative path to the sample. + + Returns: + A list of tuples, where each tuple contains a variable reference + (e.g., `$(LABEL)`) and its corresponding value. + + Example: + ```python + >>> sample = [10, 20] + >>> labels = ["X", "Y"] + >>> sample_id = 0 + >>> relative_path_to_sample = "/0/3/4/8/9/" + >>> parameter_substitutions_for_sample(sample, labels, sample_id, relative_path_to_sample) + [ + ("$(X)", "10"), + ("$(Y)", "20"), + ("$(MERLIN_SAMPLE_ID)", "0"), + ("$(MERLIN_SAMPLE_PATH)", "/0/3/4/8/9/") + ] + ``` """ substitutions = [] for label, axis in zip(labels, sample): @@ -198,13 +289,50 @@ def parameter_substitutions_for_sample(sample, labels, sample_id, relative_path_ return substitutions -def parameter_substitutions_for_cmd(glob_path, sample_paths): +def parameter_substitutions_for_cmd(glob_path: str, sample_paths: str) -> List[Tuple[str, str]]: """ - :param glob_path: a glob that should yield the paths to all merlin samples - :param sample_paths: a delimited list of all of the samples - - :return : list of pairs indicating what needs to be substituted for a - merlin cmd + Generate parameter substitutions for a Merlin command. + + This function creates a list of substitution pairs for a Merlin command, + mapping variable references to their corresponding values. It also includes + predefined return codes for various Merlin states. + + Substitutions: + - `$(MERLIN_GLOB_PATH)`: The provided `glob_path`. + - `$(MERLIN_PATHS_ALL)`: The provided `sample_paths`. + - `$(MERLIN_SUCCESS)`: The return code for a successful operation. + - `$(MERLIN_RESTART)`: The return code for a restart operation. + - `$(MERLIN_SOFT_FAIL)`: The return code for a soft failure. + - `$(MERLIN_HARD_FAIL)`: The return code for a hard failure. + - `$(MERLIN_RETRY)`: The return code for a retry operation. + - `$(MERLIN_STOP_WORKERS)`: The return code for stopping workers. + - `$(MERLIN_RAISE_ERROR)`: The return code for raising an error. + + Args: + glob_path: A glob pattern that yields the paths to all Merlin samples. + sample_paths: A delimited string containing the paths to all samples. + + Returns: + A list of tuples, where each tuple contains a variable reference and its + corresponding value. + + Example: + ```python + >>> glob_path = "/path/to/samples/*" + >>> sample_paths = "/path/to/sample1:/path/to/sample2" + >>> parameter_substitutions_for_cmd(glob_path, sample_paths) + [ + ("$(MERLIN_GLOB_PATH)", "/path/to/samples/*"), + ("$(MERLIN_PATHS_ALL)", "/path/to/sample1:/path/to/sample2"), + ("$(MERLIN_SUCCESS)", "0"), + ("$(MERLIN_RESTART)", "100"), + ("$(MERLIN_SOFT_FAIL)", "101"), + ("$(MERLIN_HARD_FAIL)", "102"), + ("$(MERLIN_RETRY)", "104"), + ("$(MERLIN_STOP_WORKERS)", "105"), + ("$(MERLIN_RAISE_ERROR)", "106") + ] + ``` """ substitutions = [] substitutions.append(("$(MERLIN_GLOB_PATH)", glob_path)) @@ -223,12 +351,26 @@ def parameter_substitutions_for_cmd(glob_path, sample_paths): # There's similar code inside study.py but the whole point of this function is to not use # the MerlinStudy object so we disable this pylint error # pylint: disable=duplicate-code -def expand_spec_no_study(filepath, override_vars=None): +def expand_spec_no_study(filepath: str, override_vars: Dict[str, str] = None) -> str: """ Get the expanded text of a spec without creating a MerlinStudy. Expansion is limited to user variables (the ones defined inside the yaml spec or at the command line). + + Expand a spec without creating a [`MerlinStudy`][study.study.MerlinStudy]. + + This function processes a spec file to expand user-defined variables (those defined + in the YAML spec or provided via `override_vars`) without creating a `MerlinStudy` + object. It returns the expanded text of the specification. + + Args: + filepath: The path to the YAML specification file. + override_vars: A dictionary of variable overrides to apply during the expansion. + These overrides replace or supplement the variables defined in the spec. + + Returns: + The expanded YAML specification as a string, with user-defined variables resolved. """ error_override_vars(override_vars, filepath) spec = MerlinSpec.load_specification(filepath) @@ -248,10 +390,22 @@ def expand_spec_no_study(filepath, override_vars=None): # pylint: enable=duplicate-code -def get_spec_with_expansion(filepath, override_vars=None): +def get_spec_with_expansion(filepath: str, override_vars: Dict[str, str] = None) -> MerlinSpec: """ - Return a MerlinSpec with overrides and expansion, without - creating a MerlinStudy. + Load and expand a Merlin YAML specification with overrides, without creating a + [`MerlinStudy`][study.study.MerlinStudy] object. + + This function returns a [`MerlinSpec`][spec.specification.MerlinSpec] object with + variables expanded and overrides applied. It processes the YAML specification file + and resolves user-defined variables without creating a `MerlinStudy` object. + + Args: + filepath: The path to the YAML specification file. + override_vars: A dictionary of variable overrides to apply during the expansion. + These overrides replace or supplement the variables defined in the YAML spec. + + Returns: + (spec.specification.MerlinSpec): A `MerlinSpec` object with expanded variables and applied overrides. """ filepath = verify_filepath(filepath) expanded_spec_text = expand_spec_no_study(filepath, override_vars) diff --git a/merlin/spec/override.py b/merlin/spec/override.py index 049d8030f..b0266e0e8 100644 --- a/merlin/spec/override.py +++ b/merlin/spec/override.py @@ -1,44 +1,37 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### -"""This module handles overriding variables in a spec file via the CLI""" +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module provides functionality to handle overriding variables in a spec file +via the command-line interface. It includes functions to validate and replace variables +in the spec file or environment block based on user-provided overrides. +""" import logging from copy import deepcopy +from typing import Dict LOG = logging.getLogger(__name__) -def error_override_vars(override_vars, spec_filepath): +def error_override_vars(override_vars: Dict[str, str], spec_filepath: str): """ - Warn user if any given variable name isn't found in the original spec file. + Warns the user if any given variable name in the override list is not found + in the original spec file. + + Args: + override_vars: A dictionary of variables to override, where keys are + variable names and values are their corresponding new values. Can + be `None` if no overrides are provided. + spec_filepath: The file path to the original spec file. + + Raises: + ValueError: If any variable name in `override_vars` is not found in the + spec file. """ if override_vars is None: return @@ -49,8 +42,21 @@ def error_override_vars(override_vars, spec_filepath): raise ValueError(f"Command line override variable '{variable}' not found in spec file '{spec_filepath}'.") -def replace_override_vars(env, override_vars): - """Replace override variables in the environment block""" +def replace_override_vars(env: Dict[str, str], override_vars: Dict[str, str]) -> Dict[str, str]: + """ + Replaces variables in the given environment block with the provided override + values. + + Args: + env: The environment block, represented as a dictionary where keys are + environment variable names and values are their corresponding values. + override_vars: A dictionary of variables to override, where keys are + variable names and values are their corresponding new values. Can be + `None` if no overrides are provided. + + Returns: + A new environment block with the override variables replaced. + """ if override_vars is None: return env result = deepcopy(env) diff --git a/merlin/spec/specification.py b/merlin/spec/specification.py index 141ebf297..80bde0065 100644 --- a/merlin/spec/specification.py +++ b/merlin/spec/specification.py @@ -1,36 +1,13 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ -This module contains a class, MerlinSpec, which holds the unchanged +This module contains a class, `MerlinSpec`, which holds the unchanged data from the Merlin specification file. + To see examples of yaml specifications, run `merlin example`. """ import json @@ -40,9 +17,10 @@ from copy import deepcopy from datetime import timedelta from io import StringIO -from typing import Dict, List +from typing import Any, Dict, List, Set, TextIO, Union import yaml +from maestrowf.datastructures.core.parameters import ParameterGenerator from maestrowf.specification import YAMLSpecification from merlin.spec import all_keys, defaults @@ -55,32 +33,62 @@ # Pylint complains we have too many instance attributes but it's fine class MerlinSpec(YAMLSpecification): # pylint: disable=R0902 """ - This class represents the logic for parsing the Merlin yaml - specification. - - Example spec_file contents: - - --spec_file.yaml-- - ... - merlin: - resources: - task_server: celery - samples: - generate: - cmd: python make_samples.py -outfile=$(OUTPUT_PATH)/merlin_info/samples.npy - file: $(OUTPUT_PATH)/merlin_info/samples.npy - column_labels: [X0, X1] + A class to represent and manage the specifications for a Merlin workflow. + + This class provides methods to verify, load, and process various sections of a + workflow specification file, including the merlin block, batch block, and user block. + It also handles default values and parameter mapping. + + Attributes: + batch (Dict): A dictionary representing the batch section of the spec file. + description (Dict): A dictionary representing the description section of the spec file. + environment (Dict): A dictionary representing the environment section of the spec file. + globals (Dict): A dictionary representing global parameters in the spec file. + merlin (Dict): A dictionary representing the merlin section of the spec file. + sections (Dict): A dictionary of all sections in the spec file. + study (Dict): A dictionary representing the study section of the spec file. + user (Dict): A dictionary representing the user section of the spec file. + yaml_sections (Dict): A dictionary for YAML representation of the sections. + + Methods: + check_section: Checks sections of the spec file for unrecognized keys. + dump: Dumps the current spec to a pretty YAML string. + fill_missing_defaults: Merges default values into an object. + get_queue_list: Returns a sorted set of queues for specified steps. + get_queue_step_relationship: Maps task queues to their associated steps. + get_step_param_map: Creates a mapping of parameters used for each step. + get_step_worker_map: Maps step names to associated workers. + get_study_step_names: Returns a list of the names of the steps in the spec file. + get_task_queues: Maps steps to their corresponding task queues. + get_tasks_per_step: Returns the number of tasks needed for each step. + get_worker_names: Returns a list of worker names. + get_worker_step_map: Maps worker names to associated steps. + load_merlin_block: Loads the merlin block from a YAML stream. + load_spec_from_string: Creates a `MerlinSpec` object from a string (or stream) representing + a spec file. + load_specification: Creates a `MerlinSpec` object based on the contents of a spec file. + load_user_block: Loads the user block from a YAML stream. + make_queue_string: Returns a unique queue string for specified steps. + process_spec_defaults: Fills in default values for missing sections. + verify: Verify the spec against a valid schema. + verify_batch_block: Validates the batch block against a predefined schema. + verify_merlin_block: Validates the merlin block against a predefined schema. + warn_unrecognized_keys: Checks for unrecognized keys in the spec file. """ # Pylint says this call to super is useless but we'll leave it in case we want to add to __init__ in the future def __init__(self): # pylint: disable=W0246 + """Initializes a MerlinSpec object.""" super().__init__() @property - def yaml_sections(self): + def yaml_sections(self) -> Dict: """ - Returns a nested dictionary of all sections of the specification - as used in a yaml spec. + Returns a nested dictionary of all sections of the specification as used in a YAML + specification. The structure is tailored for YAML representation. + + Returns: + A dictionary containing the sections of the specification formatted for YAML. """ return { "description": self.description, @@ -93,10 +101,14 @@ def yaml_sections(self): } @property - def sections(self): + def sections(self) -> Dict: """ - Returns a nested dictionary of all sections of the specification - as referenced by Maestro's YAMLSpecification class. + Returns a nested dictionary of all sections of the specification as referenced by + [Maestro's `YAMLSpecification` class](https://maestrowf.readthedocs.io/en/latest/Maestro/reference_guide/api_reference/specification/yamlspecification.html). + The structure is aligned with the expectations of Maestro's `YAMLSpecification` class. + + Returns: + A dictionary containing the sections of the specification formatted for Maestro. """ return { "description": self.description, @@ -109,7 +121,6 @@ def sections(self): } def __str__(self): - """Magic method to print an instance of our MerlinSpec class.""" env = "" globs = "" merlin = "" @@ -129,14 +140,23 @@ def __str__(self): return result @classmethod - def load_specification(cls, path, suppress_warning=True): + def load_specification(cls, path: str, suppress_warning: bool = True) -> "MerlinSpec": """ - Load in a spec file and create a MerlinSpec object based on its' contents. + Load a specification file and create a `MerlinSpec` object based on its contents. + + This method reads a YAML specification file from the provided path, + processes its contents, and returns a `MerlinSpec` object. It can also + suppress warnings about unrecognized keys in the specification. - :param `cls`: The class reference (like self) - :param `path`: A path to the spec file we're loading in - :param `suppress_warning`: A bool representing whether to warn the user about unrecognized keys - :returns: A MerlinSpec object + Args: + path: The path to the specification file to be loaded. + suppress_warning: Whether to suppress warnings about unrecognized keys. + + Returns: + A `MerlinSpec` object created from the contents of the specification file. + + Raises: + Exception: If there is an error loading the specification file. """ LOG.info("Loading specification from path: %s", path) try: @@ -157,16 +177,23 @@ def load_specification(cls, path, suppress_warning=True): return spec @classmethod - def load_spec_from_string(cls, string, needs_IO=True, needs_verification=False): # pylint: disable=C0103 + def load_spec_from_string( + cls, string: Union[str, TextIO], needs_IO: bool = True, needs_verification: bool = False + ) -> "MerlinSpec": # pylint: disable=C0103 """ - Read in a spec file from a string (or stream) and create a MerlinSpec object from it. + Read a specification from a string (or stream) and create a `MerlinSpec` object from it. + + This method processes a string or stream containing the specification + and returns a `MerlinSpec` object. It can also verify the specification + if required. + + Args: + string: A string or stream of the specification content. + needs_IO: Whether to treat the string as a file object. + needs_verification: Whether to verify the specification after loading. - :param `cls`: The class reference (like self) - :param `string`: A string or stream of the file we're reading in - :param `needs_IO`: A bool representing whether we need to turn the string into a file - object or not - :param `needs_verification`: A bool representing whether we need to verify the spec - :returns: A MerlinSpec object + Returns: + A `MerlinSpec` object created from the provided specification content. """ LOG.debug("Creating Merlin spec object...") # Create and populate the MerlinSpec object @@ -197,19 +224,28 @@ def load_spec_from_string(cls, string, needs_IO=True, needs_verification=False): return spec @classmethod - def _populate_spec(cls, data): + def _populate_spec(cls, data: TextIO) -> "MerlinSpec": """ - Helper method to load a study spec and populate it's fields. - - NOTE: This is basically a direct copy of YAMLSpecification's - load_specification method from Maestro just without the call to verify. - The verify method was breaking our code since we have no way of modifying - Maestro's schema that they use to verify yaml files. The work around - is to load the yaml file ourselves and create our own schema to verify - against. - - :param data: Raw text stream to study YAML spec data - :returns: A MerlinSpec object containing information from the path + Helper method to load a study specification and populate its fields. + + This method reads a YAML specification from a raw text stream and + populates the fields of a `MerlinSpec` object. It is a modified version + of the `load_specification` method from Maestro's YAMLSpecification class, + excluding the verification step due to compatibility issues with Maestro's schema. + + Note: + This is basically a direct copy of YAMLSpecification's + load_specification method from Maestro just without the call to verify. + The verify method was breaking our code since we have no way of modifying + Maestro's schema that they use to verify yaml files. The work around + is to load the yaml file ourselves and create our own schema to verify + against. + + Args: + data: A raw text stream containing the study YAML specification data. + + Returns: + A `MerlinSpec` object populated with information extracted from the YAML specification. """ # Read in the spec file try: @@ -244,12 +280,22 @@ def _populate_spec(cls, data): def verify(self): """ - Verify the spec against a valid schema. Similar to YAMLSpecification's verify - method from Maestro but specific for Merlin yaml specs. - - NOTE: Maestro v2.0 may add the ability to customize the schema files it - compares against. If that's the case then we can convert this file back to - using Maestro's verification. + Verify the specification against a valid schema. + + This method checks the current `MerlinSpec` object against a predefined + schema to ensure that it adheres to the expected structure and + constraints. It is similar to the verify method from Maestro's + YAMLSpecification class but is tailored specifically for Merlin YAML + specifications. + + Note: + Maestro v2.0 may introduce the ability to customize the schema files + used for verification. If that feature becomes available, then we can + convert this file back to using Maestro's verification. + + Raises: + Exception: If the specification does not conform to the schema, + appropriate exceptions will be raised during the verification process. """ # Load the MerlinSpec schema file dir_path = os.path.dirname(os.path.abspath(__file__)) @@ -267,11 +313,16 @@ def verify(self): self.verify_merlin_block(schema["MERLIN"]) self.verify_batch_block(schema["BATCH"]) - def get_study_step_names(self): + def get_study_step_names(self) -> List[str]: """ - Get a list of the names of steps in our study. + Retrieve the names of steps in the study. + + This method iterates through the study steps and collects their names + into a list. The returned list is unsorted. - :returns: an unsorted list of study step names + Returns: + An unsorted list of strings representing the names of the + study steps. """ names = [] for step in self.study: @@ -280,8 +331,16 @@ def get_study_step_names(self): def _verify_workers(self): """ - Helper method to verify the workers section located within the Merlin block - of our spec file. + Verify the workers section in the Merlin block of the specification. + + This helper method checks that the steps referenced in the workers + section of the Merlin block exist in the study steps. It raises a + ValueError if any step specified for a worker does not match the + defined study steps. + + Raises: + ValueError: If a step specified in the workers section does not + exist in the list of study step names. """ # Retrieve the names of the steps in our study actual_steps = self.get_study_step_names() @@ -301,24 +360,35 @@ def _verify_workers(self): except Exception: # pylint: disable=W0706 raise - def verify_merlin_block(self, schema): + def verify_merlin_block(self, schema: Dict): """ - Method to verify the merlin section of our spec file. - :param schema: The section of the predefined schema (merlinspec.json) to check - our spec file against. + Verify the Merlin section of the specification file against a schema. + + This method validates the Merlin block of the specification file + against a predefined JSON schema and verifies the workers section + to ensure that all specified steps are defined in the study. + + Args: + schema: The section of the predefined schema (merlinspec.json) to + check the Merlin block against. """ # Validate merlin block against the json schema YAMLSpecification.validate_schema("merlin", self.merlin, schema) # Verify the workers section within merlin block self._verify_workers() - def verify_batch_block(self, schema): + def verify_batch_block(self, schema: Dict): """ - Method to verify the batch section of our spec file. + Verify the batch section of the specification file against a schema. - :param schema: The section of the predefined schema (merlinspec.json) to check - our spec file against. + This method validates the batch block of the specification file + against a predefined JSON schema and performs additional checks + related to the walltime parameter for the LSF batch type. + + Args: + schema: The section of the predefined schema (merlinspec.json) to + check the batch block against. """ # Validate batch block against the json schema YAMLSpecification.validate_schema("batch", self.batch, schema) @@ -328,8 +398,23 @@ def verify_batch_block(self, schema): LOG.warning("The walltime argument is not available in lsf.") @staticmethod - def load_merlin_block(stream): - """Loads in the merlin block of the spec file""" + def load_merlin_block(stream: TextIO) -> Dict: + """ + Load the Merlin block from a specification file stream. + + This static method reads a YAML stream and attempts to extract + the 'merlin' section. If the 'merlin' section is missing, it + logs a warning and returns an empty dictionary, indicating that + the default configuration will be used without sampling. + + Args: + stream: A file-like object or string stream containing the + YAML specification. + + Returns: + The Merlin block extracted from the YAML stream. If the 'merlin' + section is not found, an empty dictionary is returned. + """ try: merlin_block = yaml.safe_load(stream)["merlin"] except KeyError: @@ -343,8 +428,22 @@ def load_merlin_block(stream): return merlin_block @staticmethod - def load_user_block(stream): - """Loads in the user block of the spec file""" + def load_user_block(stream: TextIO) -> Dict: + """ + Load the user block from a specification file stream. + + This static method reads a YAML stream and attempts to extract + the 'user' section. If the 'user' section is not present, it + returns an empty dictionary. + + Args: + stream: A file-like object or string stream containing the + YAML specification. + + Returns: + The user block extracted from the YAML stream. If the 'user' + section is not found, an empty dictionary is returned. + """ try: user_block = yaml.safe_load(stream)["user"] except KeyError: @@ -352,7 +451,23 @@ def load_user_block(stream): return user_block def process_spec_defaults(self): - """Fills in the default values if they aren't there already""" + """ + Fill in default values for specification sections if they are missing. + + This method iterates through the sections of the specification and + populates any that are `None` with empty dictionaries. It then fills + in default values for various sections, including batch, environment, + global parameters, and step sections within the study. + + The method also handles specific cases for the VLAUNCHER variables + in the command of each step, ensuring that default values are set + if they are not defined by the user. Additionally, it ensures that + workers are assigned to steps appropriately, filling in defaults + where necessary. + + The method modifies the instance's attributes directly, ensuring that + the specification is complete and ready for further processing. + """ for name, section in self.sections.items(): if section is None: setattr(self, name, {}) @@ -415,15 +530,52 @@ def process_spec_defaults(self): # no defaults for user block @staticmethod - def fill_missing_defaults(object_to_update, default_dict): + def fill_missing_defaults(object_to_update: Dict, default_dict: Dict): """ - Merge keys and values from a dictionary of defaults - into a parallel object that may be missing attributes. - Only adds missing attributes to object; does not overwrite - existing ones. + Merge default values into an object, filling in missing attributes. + + This static method takes an object and a dictionary of default values, + and merges the defaults into the object. It only adds missing attributes + to the object and does not overwrite any existing attributes. If an + attribute is present in the object but its value is `None`, it will be + updated with the corresponding value from the defaults. + + The method works recursively, allowing for nested dictionaries. + + The method modifies the `object_to_update` in place. + + Args: + object_to_update: The object (as a dictionary) that needs to be + updated with default values. + default_dict: A dictionary containing default values to merge into + the object. + + Example: + ```python + >>> obj = {'a': 1, 'b': None} + >>> defaults = {'a': 2, 'b': 3, 'c': 4} + >>> fill_missing_defaults(obj, defaults) + >>> print(obj) + {'a': 1, 'b': 3, 'c': 4} + ``` """ - def recurse(result, recurse_defaults): + def recurse(result: Dict, recurse_defaults: Dict): + """ + Recursively merge default values into the result object. + + This helper function checks if the current level of the `recurse_defaults` + dictionary is a dictionary itself. If it is, it iterates through each key-value + pair. If a key is not present in the `result` or its value is `None`, it + assigns the value from `recurse_defaults`. If the key exists and has a value, + it recursively calls itself to handle nested dictionaries. + + The function modifies the `result` in place. + + Args: + result: The current state of the object being updated. + recurse_defaults: The current level of defaults to merge. + """ if not isinstance(recurse_defaults, dict): return for key, val in recurse_defaults.items(): @@ -440,7 +592,16 @@ def recurse(result, recurse_defaults): # ***Unsure if this method is still needed after adding json schema verification*** def warn_unrecognized_keys(self): - """Checks if there are any unrecognized keys in the spec file""" + """ + Check for unrecognized keys in the specification file. + + This method verifies that all keys present in the specification file + conform to the expected structure defined by the `MerlinSpec` class. + It checks various sections of the specification, including "description", + "batch", "env", "global parameters", "steps", and "merlin". For each + section, it calls the `check_section` method to ensure that the keys + are recognized and valid according to predefined criteria. + """ # check description MerlinSpec.check_section("description", self.description, all_keys.DESCRIPTION) @@ -470,8 +631,21 @@ def warn_unrecognized_keys(self): # user block is not checked @staticmethod - def check_section(section_name, section, known_keys): - """Checks a section of the spec file to see if there are any unrecognized keys""" + def check_section(section_name: str, section: Dict, known_keys: Set[str]): + """ + Check a section of the specification file for unrecognized keys. + + This static method compares the keys present in a specified section + of the specification file against a set of known keys. If any keys + are found that are not recognized, a warning is logged indicating + the unrecognized key and the section in which it was found. + + Args: + section_name: The name of the section being checked. + section: The section of the specification file to validate. + known_keys: A set of keys that are recognized as valid for + the specified section. + """ diff = set(section.keys()).difference(known_keys) # TODO: Maybe add a check here for required keys @@ -479,9 +653,23 @@ def check_section(section_name, section, known_keys): for extra in diff: LOG.warning(f"Unrecognized key '{extra}' found in spec section '{section_name}'.") - def dump(self): + def dump(self) -> str: """ - Dump this MerlinSpec to a pretty yaml string. + Dump the `MerlinSpec` instance to a formatted YAML string. + + This method converts the current state of the `MerlinSpec` instance + into a YAML formatted string. It utilizes the `_dict_to_yaml` + method to handle the conversion and prettification of the data. + Additionally, it ensures that the resulting YAML string is valid + by attempting to parse it with `yaml.safe_load`. If parsing fails, + a ValueError is raised with details about the error. + + Returns: + A pretty formatted YAML string representation of the + `MerlinSpec` instance. + + Raises: + ValueError: If there is an error while parsing the YAML string. """ tab = 3 * " " result = self._dict_to_yaml(self.yaml_sections, "", [], tab) @@ -493,9 +681,26 @@ def dump(self): raise ValueError(f"Error parsing provenance spec:\n{e}") from e return result - def _dict_to_yaml(self, obj, string, key_stack, tab): + def _dict_to_yaml(self, obj: Any, string: str, key_stack: List[str], tab: int) -> str: """ - The if-else ladder for sorting the yaml string prettification of dump(). + Convert a Python object to a formatted YAML string. + + This private method handles the conversion of various Python data + types (strings, booleans, lists, and dictionaries) into a + formatted YAML string. It uses an if-else structure to determine + the type of the input object and calls the appropriate processing + methods for each type. The method also manages indentation based + on the current level of nesting. + + Args: + obj: The object to convert to YAML format. + string: The current string representation being built. + key_stack: A stack of keys representing the current level of + nesting in the YAML structure. + tab: The number of spaces to use for indentation. + + Returns: + A formatted YAML string representation of the input object. """ if obj is None: return "" @@ -512,18 +717,57 @@ def _dict_to_yaml(self, obj, string, key_stack, tab): return self._process_dict(obj, string, key_stack, lvl, tab) return obj - def _process_string(self, obj, lvl, tab): + def _process_string(self, obj: str, lvl: int, tab: int) -> str: """ - Processes strings for _dict_to_yaml() in the dump() method. + Process a string for YAML formatting in the dump method. + + This private method takes a string and formats it for inclusion + in a YAML output. If the string contains multiple lines, it + transforms the string into a block scalar format using the pipe + (`|`) character, which is suitable for YAML representation. + The indentation is adjusted based on the current level of + nesting. + + Args: + obj: The string to be processed. + lvl: The current level of indentation for the YAML output. + tab: The number of spaces to use for indentation. + + Returns: + The formatted string ready for YAML output. """ split = obj.splitlines() if len(split) > 1: obj = "|\n" + tab * (lvl + 1) + ("\n" + tab * (lvl + 1)).join(split) return obj - def _process_list(self, obj, string, key_stack, lvl, tab): # pylint: disable=R0913 + def _process_list( + self, + obj: List[Any], + string: str, + key_stack: List[str], + lvl: int, + tab: int, + ) -> str: """ - Processes lists for _dict_to_yaml() in the dump() method. + Process a list for YAML formatting in the dump method. + + This private method handles the conversion of a list into a + YAML formatted string. It determines whether to use hyphens + for list items based on the context provided by the key stack. + The method recursively processes each element in the list and + manages indentation based on the current level of nesting. + + Args: + obj: The list to be processed. + string: The current string representation being built. + key_stack: A stack of keys representing the current + level of nesting in the YAML structure. + lvl: The current level of indentation for the YAML output. + tab: The number of spaces to use for indentation. + + Returns: + A formatted YAML string representation of the input list. """ num_entries = len(obj) use_hyphens = key_stack[-1] in ["paths", "sources", "git", "study"] or key_stack[0] in ["user"] @@ -545,9 +789,33 @@ def _process_list(self, obj, string, key_stack, lvl, tab): # pylint: disable=R0 string += "]" return string - def _process_dict(self, obj, string, key_stack, lvl, tab): # pylint: disable=R0913 + def _process_dict( + self, + obj: Dict, + string: str, + key_stack: List[str], + lvl: int, + tab: int, + ) -> str: # pylint: disable=R0913 """ - Processes dicts for _dict_to_yaml() in the dump() method + Process a dictionary for YAML formatting in the dump method. + + This private method converts a dictionary into a YAML formatted + string. It iterates over the dictionary's key-value pairs, + formatting each pair according to YAML syntax. The method + handles indentation and manages the key stack to maintain the + correct nesting level in the output. + + Args: + obj: The dictionary to be processed. + string: The current string representation being built. + key_stack: A stack of keys representing the current + level of nesting in the YAML structure. + lvl: The current level of indentation for the YAML output. + tab: The number of spaces to use for indentation. + + Returns: + A formatted YAML string representation of the input dictionary. """ list_offset = 2 * " " if len(key_stack) > 0 and key_stack[-1] != "elem": @@ -570,10 +838,18 @@ def _process_dict(self, obj, string, key_stack, lvl, tab): # pylint: disable=R0 def get_step_worker_map(self) -> Dict[str, List[str]]: """ - Creates a dictionary with step names as keys and a list of workers - associated with each step as values. The inverse of get_worker_step_map(). - - :returns: A dict mapping step names to workers + Create a mapping of step names to associated workers. + + This method constructs a dictionary where each key is a step name + and the corresponding value is a list of workers assigned to that + step. Workers can either be associated with all steps or with + specific steps. This method serves as the inverse of the + [`get_worker_step_map`][spec.specification.MerlinSpec.get_worker_step_map] + method. + + Returns: + A dictionary mapping step names to lists of worker names + associated with each step. """ steps = self.get_study_step_names() step_worker_map = {step_name: [] for step_name in steps} @@ -590,10 +866,18 @@ def get_step_worker_map(self) -> Dict[str, List[str]]: def get_worker_step_map(self) -> Dict[str, List[str]]: """ - Creates a dictionary with worker names as keys and a list of steps - associated with each worker as values. The inverse of get_step_worker_map(). - - :returns: A dict mapping workers to the steps they watch + Create a mapping of worker names to associated steps. + + This method constructs a dictionary where each key is a worker name + and the corresponding value is a list of steps that the worker is + assigned to monitor. Workers can either be assigned to all steps or + to specific steps. It serves as the inverse of the + [`get_step_worker_map`][spec.specification.MerlinSpec.get_step_worker_map] + method. + + Returns: + A dictionary mapping worker names to lists of step names that each + worker monitors. """ worker_step_map = {} steps = self.get_study_step_names() @@ -608,13 +892,23 @@ def get_worker_step_map(self) -> Dict[str, List[str]]: worker_step_map[worker_name].append(step) return worker_step_map - def get_task_queues(self, omit_tag=False): + def get_task_queues(self, omit_tag: bool = False) -> Dict[str, str]: """ - Creates a dictionary of steps and their corresponding task queues. - This is the inverse of get_queue_step_relationship() + Create a mapping of steps to their corresponding task queues. - :param `omit_tag`: If True, omit the celery queue tag. - :returns: A dict of steps and their corresponding task queues + This method constructs a dictionary where each key is a step name + and the corresponding value is the associated task queue. The + `omit_tag` parameter allows for the optional exclusion of the Celery + queue tag from the queue names. It serves as the inverse of the + [`get_queue_step_relationship`][spec.specification.MerlinSpec.get_queue_step_relationship] + method. + + Args: + omit_tag: If True, the Celery queue tag will be omitted + from the task queue names. Default is False. + + Returns: + A dictionary mapping step names to their corresponding task queues. """ from merlin.config.configfile import CONFIG # pylint: disable=C0415 @@ -629,10 +923,17 @@ def get_task_queues(self, omit_tag=False): def get_queue_step_relationship(self) -> Dict[str, List[str]]: """ - Builds a dictionary of task queues and their associated steps. - This returns the inverse of get_task_queues(). + Build a mapping of task queues to their associated steps. - :returns: A dict of task queues and their associated steps + This method constructs a dictionary where each key is a task queue + name and the corresponding value is a list of steps that are + associated with that queue. It serves as the inverse of the + [`get_task_queues`][spec.specification.MerlinSpec.get_task_queues] + method. + + Returns: + A dictionary mapping task queue names to lists of step names + associated with each queue. """ from merlin.config.configfile import CONFIG # pylint: disable=C0415 @@ -654,13 +955,28 @@ def get_queue_step_relationship(self) -> Dict[str, List[str]]: return relationship_tracker - def get_queue_list(self, steps, omit_tag=False) -> set: + def get_queue_list(self, steps: Union[List[str], str], omit_tag: bool = False) -> Set[str]: """ - Return a sorted set of queues corresponding to spec steps - - :param `steps`: a list of step names or ['all'] - :param `omit_tag`: If True, omit the celery queue tag. - :returns: A sorted set of queues corresponding to spec steps + Return a sorted set of queues corresponding to specified steps. + + This method retrieves a list of task queues associated with the + given steps. If the `steps` parameter is set to ['all'], it will + return all available queues. The `omit_tag` parameter allows for + the optional exclusion of the Celery queue tag from the queue names. + + Args: + steps: A list of step names or a list containing the string 'all' + to represent all steps, or the name of a single step. + omit_tag: If True, the Celery queue tag will be omitted from the + task queue names. + + Returns: + A sorted set of unique task queues corresponding to the specified + steps. + + Raises: + KeyError: If any of the specified steps do not exist in the + task queues. """ queues = self.get_task_queues(omit_tag=omit_tag) if steps[0] == "all": @@ -677,17 +993,34 @@ def get_queue_list(self, steps, omit_tag=False) -> set: raise return sorted(set(task_queues)) - def make_queue_string(self, steps): + def make_queue_string(self, steps: List[str]) -> str: """ - Return a unique queue string for the steps + Return a unique queue string for the specified steps. - param steps: a list of step names + This method constructs a comma-separated string of unique task + queues associated with the provided steps. The resulting string + is suitable for use in command-line contexts. + + Args: + steps: A list of step names for which to generate the + queue string. + + Returns: + A quoted string of unique task queues, separated by commas. """ queues = ",".join(set(self.get_queue_list(steps))) return shlex.quote(queues) - def get_worker_names(self): - """Builds a list of workers""" + def get_worker_names(self) -> List[str]: + """ + Build a list of worker names. + + This method retrieves the names of all workers defined in the + Merlin resources and returns them as a list. + + Returns: + A list of worker names. + """ result = [] for worker in self.merlin["resources"]["workers"]: result.append(worker) @@ -695,8 +1028,17 @@ def get_worker_names(self): def get_tasks_per_step(self) -> Dict[str, int]: """ - Get the number of tasks needed to complete each step, formatted as a dictionary. - :returns: A dict where the keys are the step names and the values are the number of tasks required for that step + Get the number of tasks needed to complete each step. + + This method calculates the number of tasks required for each + step in the study based on the number of samples and parameters. + It returns a dictionary where the keys are the step names and + the values are the corresponding number of tasks required for + that step. + + Returns: + A dictionary mapping step names to the number of tasks + required for each step. """ # Get the number of samples used samples = [] @@ -729,20 +1071,33 @@ def get_tasks_per_step(self) -> Dict[str, int]: return tasks_per_step - def _create_param_maps(self, param_gen: "ParameterGenerator", expanded_labels: Dict, label_param_map: Dict): # noqa: F821 + def _create_param_maps(self, param_gen: ParameterGenerator, expanded_labels: Dict, label_param_map: Dict): """ - Given a parameters block like so: + Create mappings of tokens to expanded labels and labels to parameter values. + + This private method processes a parameter generator to create two mappings: + + 1. `expanded_labels`: Maps tokens to their expanded labels based on the + provided parameter values. + 2. `label_param_map`: Maps expanded labels to their corresponding parameter + values. + + The expected structure for the parameter block is: + + ``` global.parameters: TOKEN: values: [param_val_1, param_val_2] label: label.%% - Expanded labels will map tokens to their expanded labels (e.g. {'TOKEN': ['label.param_val_1', 'label.param_val_2']}) - Label param map will map labels to parameter values - (e.g. {'label.param_val_1': {'TOKEN': 'param_val_1'}, 'label.param_val_2': {'TOKEN': 'param_val_2'}}) - - :param `param_gen`: A ParameterGenerator object from Maestro - :param `expanded_labels`: A dict to store the map from tokens to expanded labels - :param `label_param_map`: A dict to store the map from labels to parameter values + ``` + + Args: + param_gen: A `ParameterGenerator` object from Maestro containing the + parameter definitions. + expanded_labels: A dictionary to store the mapping from tokens to their + expanded labels. + label_param_map: A dictionary to store the mapping from labels to their + corresponding parameter values. """ for token, orig_label in param_gen.labels.items(): for param in param_gen.parameters[token]: @@ -755,9 +1110,13 @@ def _create_param_maps(self, param_gen: "ParameterGenerator", expanded_labels: D def get_step_param_map(self) -> Dict: # pylint: disable=R0914 """ - Create a mapping of parameters used for each step. Each step will have a cmd - to search for parameters in and could also have a restart cmd to check, too. - This creates a mapping of the form: + Create a mapping of parameters used for each step in the study. + + This method generates a mapping of parameters for each step, where each + step may have a command (`cmd`) and a restart command (`restart_cmd`). + The resulting mapping has a structure similar to the following: + + ```python step_name_with_parameters: { "cmd": { TOKEN_1: param_1_value_1, @@ -768,8 +1127,11 @@ def get_step_param_map(self) -> Dict: # pylint: disable=R0914 TOKEN_3: param_3_value_1, } } + ``` - :returns: A dict mapping between steps and params of the form shown above + Returns: + A dictionary mapping step names (with parameters) to their + respective command and restart command parameter mappings. """ # Get the steps and the parameters in the study study_steps = self.get_study_steps() diff --git a/merlin/study/__init__.py b/merlin/study/__init__.py index 37cabcad1..be03206ec 100644 --- a/merlin/study/__init__.py +++ b/merlin/study/__init__.py @@ -1,29 +1,28 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +The `study` package contains functionality for defining, managing, and monitoring studies +in Merlin. A study represents a collection of tasks, steps, and workflows that can be +executed on distributed systems using various schedulers. + +Modules: + batch.py: Parses the batch section of the YAML specification, supporting worker + launches for schedulers like Slurm, LSF, and Flux. + celeryadapter.py: Provides an adapter for integrating with the Celery Distributed + Task Queue, enabling distributed task execution. + dag.py: Defines the Merlin `DAG` class, which represents the Directed Acyclic Graph + structure of a study's workflow. + script_adapter.py: Contains functionality for adapting bash scripts to work with + supported schedulers, including Flux, LSF, and Slurm. + status_constants.py: Defines constants used by the `status` module and its renderers, + helping to avoid circular import issues. + status_renderers.py: Handles the creation of formatted, task-by-task status displays + for studies. + status.py: Implements functionality for retrieving and displaying the statuses of studies. + step.py: Contains the logic for representing and managing individual steps in a study. + study.py: Implements the core logic for defining and managing a study as a whole. +""" diff --git a/merlin/study/batch.py b/merlin/study/batch.py index cf2c53116..61ba48709 100644 --- a/merlin/study/batch.py +++ b/merlin/study/batch.py @@ -1,62 +1,57 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ This module parses the batch section of the yaml specification. Currently only the batch worker launch for slurm, lsf or flux are implemented. - """ import logging import os import subprocess -from typing import Dict, Optional, Union +from typing import Dict, Union +from merlin.spec.specification import MerlinSpec from merlin.utils import convert_timestring, get_flux_alloc, get_flux_version, get_yaml_var LOG = logging.getLogger(__name__) -def batch_check_parallel(spec): +def batch_check_parallel(spec: MerlinSpec) -> bool: """ - Check for a parallel batch section in the yaml file. + Check for a parallel batch section in the provided MerlinSpec object. + + This function examines the 'batch' section of the given specification to determine + whether it is configured for parallel execution. It checks the 'type' attribute + within the batch section, defaulting to 'local' if not specified. If the type + is anything other than 'local', the function will return True, indicating that + parallel processing is enabled. + + Args: + spec (spec.specification.MerlinSpec): An instance of the + [`MerlinSpec`][spec.specification.MerlinSpec] class that contains the + configuration details, including the batch section. + + Returns: + Returns True if the batch type is set to a value other than 'local', + indicating that parallel processing is enabled; otherwise, returns False. + + Raises: + AttributeError: If the 'batch' section is not present in the specification, + an error is logged and an AttributeError is raised. """ parallel = False try: batch = spec.batch - except AttributeError: + except AttributeError as exc: LOG.error("The batch section is required in the specification file.") - raise + raise exc btype = get_yaml_var(batch, "type", "local") if btype != "local": @@ -65,14 +60,29 @@ def batch_check_parallel(spec): return parallel -def check_for_scheduler(scheduler, scheduler_legend): +def check_for_scheduler(scheduler: str, scheduler_legend: Dict[str, str]) -> bool: """ - Check which scheduler (Flux, Slurm, LSF, or PBS) is the main - scheduler for the cluster. - :param `scheduler`: A string representing the scheduler to check for - Options: flux, slurm, lsf, or pbs - :param `scheduler_legend`: A dict of information related to each scheduler - :returns: A bool representing whether `scheduler` is the main scheduler for the cluster + Check which scheduler (Flux, Slurm, LSF, or PBS) is the main scheduler for the cluster. + + This function verifies if the specified scheduler is the main scheduler by executing + a command associated with it from the provided scheduler legend. It returns a boolean + indicating whether the specified scheduler is active. + + Args: + scheduler: A string representing the scheduler to check for. Options include 'flux', + 'slurm', 'lsf', or 'pbs'. + scheduler_legend: A dictionary containing information related to each scheduler, + including the command to check its status and the expected output. See + [`construct_scheduler_legend`][study.batch.construct_scheduler_legend] + for more information on all the settings this dict contains. + + Returns: + Returns True if the specified scheduler is the main scheduler for the + cluster, otherwise returns False. + + Raises: + FileNotFoundError: If the command associated with the scheduler cannot be found. + PermissionError: If there are insufficient permissions to execute the command. """ # Check for invalid scheduler if scheduler not in ("flux", "slurm", "lsf", "pbs"): @@ -96,14 +106,26 @@ def check_for_scheduler(scheduler, scheduler_legend): return False -def get_batch_type(scheduler_legend, default=None): +def get_batch_type(scheduler_legend: Dict[str, str], default: str = None) -> str: """ Determine which batch scheduler to use. - :param scheduler_legend: A dict storing info related to each scheduler - :param default: (str) The default batch scheduler to use if a scheduler - can't be determined. The default is None. - :returns: (str) The batch name (available options: slurm, flux, lsf, pbs). + This function checks a predefined list of batch schedulers in a specific order + to determine which one is available for use. If none of the schedulers are found, + it checks the system type environment variable to suggest a default scheduler. + If no suitable scheduler is determined, it returns the specified default value. + + Args: + scheduler_legend: A dictionary storing information related to each + scheduler, including commands and expected outputs for checking their + availability. See [`construct_scheduler_legend`][study.batch.construct_scheduler_legend] + for more information on all the settings this dict contains. + default: The default batch scheduler to use if a scheduler cannot be determined. + + Returns: + The name of the available batch scheduler. Possible options include + 'slurm', 'flux', 'lsf', or 'pbs'. If no scheduler is found, returns + the specified default value. """ # These schedulers are listed in order of which should be checked for first # 1. Flux should be checked first due to slurm emulation scripts @@ -126,13 +148,29 @@ def get_batch_type(scheduler_legend, default=None): return default -def get_node_count(parsed_batch: Dict, default=1): +def get_node_count(parsed_batch: Dict, default: int = 1) -> int: """ Determine a default node count based on the environment. - :param default: (int) The number of nodes to return if a node count from - the environment cannot be determined. - :param returns: (int) The number of nodes to use. + This function checks the environment and the Flux version to determine the + appropriate number of nodes to use for batch processing. It first verifies + the Flux version, then attempts to retrieve the node count from the Flux + allocation or environment variables specific to Slurm or LSF. If no valid + node count can be determined, it returns a specified default value. + + Args: + parsed_batch: A dictionary containing parsed batch configurations. + See [`parse_batch_block`][study.batch.parse_batch_block] for more + information on all the settings in this dictionary. + default: The number of nodes to return if a node count from the + environment cannot be determined. + + Returns: + The number of nodes to use for the batch job. This value is determined + based on the environment and scheduler specifics. + + Raises: + ValueError: If the Flux version is too old (below 0.17.0). """ # Flux version check @@ -166,9 +204,38 @@ def get_node_count(parsed_batch: Dict, default=1): def parse_batch_block(batch: Dict) -> Dict: """ - A function to parse the batch block of the yaml file. - :param `batch`: The batch block to read in - :returns: A dict with all the info (or defaults) from the batch block + Parse the batch block of a YAML configuration file. + + This function extracts relevant information from the provided batch block + dictionary, including paths, execution options, and defaults. It retrieves + the Flux executable path and allocation details, and populates a dictionary + with the parsed values. + + Args: + batch: A dictionary representing the batch block from the YAML + configuration file. + + Returns: + A dictionary containing parsed information from the batch block, + including:\n + - `btype`: The type of batch job (default is 'local'). + - `nodes`: The number of nodes to use (default is None). + - `shell`: The shell to use (default is 'bash'). + - `bank`: The bank to charge for the job (default is an empty string). + - `queue`: The queue to submit the job to (default is an empty string). + - `walltime`: The maximum wall time for the job (default is an empty string). + - `launch pre`: Any commands to run before launching (default is an empty string). + - `launch args`: Arguments for the launch command (default is an empty string). + - `launch command`: Custom command to launch workers. This will override the + default launch command (default is an empty string). + - `flux path`: Optional path to flux bin. + - `flux exe`: The full path to the Flux executable. + - `flux exec`: Optional flux exec command to launch workers on all nodes if + `flux_exec_workers` is True (default is None). + - `flux alloc`: The Flux allocation retrieved from the executable. + - `flux opts`: Optional flux start options (default is an empty string). + - `flux exec workers`: Optional flux argument to launch workers + on all nodes (default is True). """ flux_path: str = get_yaml_var(batch, "flux_path", "") if "/" in flux_path: @@ -204,9 +271,20 @@ def parse_batch_block(batch: Dict) -> Dict: def get_flux_launch(parsed_batch: Dict) -> str: """ - Build the flux launch command based on the batch section of the yaml. - :param `parsed_batch`: A dict of batch configurations - :returns: The flux launch command + Build the Flux launch command based on the batch section of the YAML configuration. + + This function constructs the command to launch a Flux job using the parameters + specified in the parsed batch configuration. It determines the appropriate + execution command for Flux workers and integrates it with the launch command + provided in the batch configuration. + + Args: + parsed_batch: A dictionary containing batch configuration parameters. + See [`parse_batch_block`][study.batch.parse_batch_block] for more information + on all the settings in this dictionary. + + Returns: + The constructed Flux launch command, ready to be executed. """ default_flux_exec = "flux exec" if parsed_batch["launch command"] else f"{parsed_batch['flux exe']} exec" flux_exec: str = "" @@ -225,20 +303,37 @@ def get_flux_launch(parsed_batch: Dict) -> str: def batch_worker_launch( - spec: Dict, + spec: MerlinSpec, com: str, - nodes: Optional[Union[str, int]] = None, - batch: Optional[Dict] = None, + nodes: Union[str, int] = None, + batch: Dict = None, ) -> str: """ - The configuration in the batch section of the merlin spec - is used to create the worker launch line, which may be - different from a simulation launch. - - : param spec : (Dict) workflow specification - : param com : (str): The command to launch with batch configuration - : param nodes : (Optional[Union[str, int]]): The number of nodes to use in the batch launch - : param batch : (Optional[Dict]): An optional batch override from the worker config + Create the worker launch command based on the batch configuration in the + workflow specification. + + This function constructs a command to launch a worker process using the + specified batch configuration. It handles different batch types and + integrates any necessary pre-launch commands, launch arguments, and + node specifications. + + Args: + spec (spec.specification.MerlinSpec): An instance of the + [`MerlinSpec`][spec.specification.MerlinSpec] class that contains the + configuration details, including the batch section. + com: The command to launch with the batch configuration. + nodes: The number of nodes to use in the batch launch. If not specified, + it will default to the value in the batch configuration. + batch: An optional batch override from the worker configuration. If not + provided, the function will attempt to retrieve the batch section from + the specification. + + Returns: + The constructed worker launch command, ready to be executed. + + Raises: + AttributeError: If the batch section is missing in the specification. + TypeError: If the `nodes` parameter is of an invalid type. """ if batch is None: try: @@ -289,17 +384,34 @@ def batch_worker_launch( def construct_scheduler_legend(parsed_batch: Dict, nodes: int) -> Dict: """ - Constructs a legend of relevant information needed for each scheduler. This includes: - - bank (str): The flag to add a bank to the launch command - - check cmd (list): The command to run to check if this is the main scheduler for the cluster - - expected check output (str): The expected output from running the check cmd - - launch (str): The initial launch command for the scheduler - - queue (str): The flag to add a queue to the launch command - - walltime (str): The flag to add a walltime to the launch command - - :param `parsed_batch`: A dict of batch configurations - :param `nodes`: An int representing the number of nodes to use in a launch command - :returns: A dict of scheduler related information + Constructs a legend of relevant information needed for each scheduler. + + This function generates a dictionary containing configuration details for various + job schedulers based on the provided batch configuration. The returned dictionary + includes flags for bank, queue, and walltime, as well as commands to check the + scheduler and the initial launch command. + + Args: + parsed_batch: A dictionary of batch configurations, which must include `bank`, + `queue`, `walltime`, and `flux alloc`. See + [`parse_batch_block`][study.batch.parse_batch_block] for more information on + all the settings in this dictionary. + nodes: The number of nodes to use in the launch command. + + Returns: + A dictionary containing scheduler-related information, structured as + follows:\n + - For each scheduler (e.g., 'flux', 'lsf', 'pbs', 'slurm'):\n + - `bank` (str): The flag to add a bank to the launch command. + - `check cmd` (List[str]): The command to run to check if this is the main + scheduler for the cluster. + - `expected check output` (bytes): The expected output from running + the check command. + - `launch` (str): The initial launch command for the scheduler. + - `queue` (str): The flag to add a queue to the launch command (if + applicable). + - `walltime` (str): The flag to add a walltime to the launch command + (if applicable). """ scheduler_legend = { "flux": { @@ -338,11 +450,26 @@ def construct_scheduler_legend(parsed_batch: Dict, nodes: int) -> Dict: def construct_worker_launch_command(parsed_batch: Dict, nodes: int) -> str: """ - If no 'worker_launch' is found in the batch yaml, this method constructs the needed launch command. - - :param `parsed_batch`: A dict of batch configurations - :param `nodes`:: The number of nodes to use in the batch launch - :returns: The launch command + Constructs the worker launch command based on the provided batch configuration. + + This function generates a launch command for a worker process when no + 'worker_launch' command is specified in the batch configuration. It + utilizes the scheduler legend to incorporate necessary flags such as + bank, queue, and walltime, depending on the workload manager. + + Args: + parsed_batch: A dictionary of batch configurations, which must include + `btype`, `bank`, `queue`, and `walltime`. See + [`parse_batch_block`][study.batch.parse_batch_block] for more information + on all the settings in this dictionary. + nodes: The number of nodes to use in the batch launch. + + Returns: + The constructed launch command for the worker process. + + Raises: + TypeError: If the PBS scheduler is enabled for a batch type other than 'flux'. + KeyError: If the workload manager is not found in the scheduler legend. """ # Initialize launch_command and get the scheduler_legend and workload_manager launch_command: str = "" diff --git a/merlin/study/celeryadapter.py b/merlin/study/celeryadapter.py index 60bded1b2..2840f5fc8 100644 --- a/merlin/study/celeryadapter.py +++ b/merlin/study/celeryadapter.py @@ -1,32 +1,8 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ This module provides an adapter to the Celery Distributed Task Queue. @@ -38,7 +14,8 @@ import time from contextlib import suppress from datetime import datetime -from typing import Dict, List, Optional, Tuple +from types import SimpleNamespace +from typing import Dict, List, Set, Tuple from amqp.exceptions import ChannelError from celery import Celery @@ -46,7 +23,9 @@ from merlin.common.dumper import dump_handler from merlin.config import Config +from merlin.spec.specification import MerlinSpec from merlin.study.batch import batch_check_parallel, batch_worker_launch +from merlin.study.study import MerlinStudy from merlin.utils import apply_list_of_regex, check_machines, get_procs, get_yaml_var, is_running @@ -55,10 +34,20 @@ # TODO figure out a better way to handle the import of celery app and CONFIG -def run_celery(study, run_mode=None): +def run_celery(study: MerlinStudy, run_mode: str = None): """ - Run the given MerlinStudy object. If the run mode is set to "local" - configure Celery to run locally (without workers). + Run the given [`MerlinStudy`][study.study.MerlinStudy] object with optional + Celery configuration. + + This function executes the provided [`MerlinStudy`][study.study.MerlinStudy] + object. If the `run_mode` is set to "local", it configures Celery to run in + local mode (without utilizing workers). Otherwise, it connects to the Celery + server to queue tasks. + + Args: + study (study.study.MerlinStudy): The study object to be executed. + run_mode: The mode in which to run the study. If set to "local", + Celery runs locally. """ # Only import celery stuff if we want celery in charge # Pylint complains about circular import between merlin.common.tasks -> merlin.router -> merlin.study.celeryadapter @@ -81,14 +70,25 @@ def run_celery(study, run_mode=None): def get_running_queues(celery_app_name: str, test_mode: bool = False) -> List[str]: """ - Check for running celery workers by looking at the currently running processes. - If there are running celery workers, we'll pull the queues from the -Q tag in the - process command. The list returned here will contain only unique celery queue names. - This must be run on the allocation where the workers are running. + Check for running Celery workers and retrieve their associated queues. - :param `celery_app_name`: The name of the celery app (typically merlin here unless testing) - :param `test_mode`: If True, run this function in test mode - :returns: A unique list of celery queues with workers attached to them + This function inspects currently running processes to identify active + Celery workers. It extracts queue names from the `-Q` tag in the + command line of the worker processes. The returned list contains + only unique Celery queue names. This function must be executed + on the allocation where the workers are running. + + Note: + Unlike [`get_active_celery_queues`][study.celeryadapter.get_active_celery_queues], + this function does _not_ go through the application's server. + + Args: + celery_app_name: The name of the Celery app (typically "merlin" + unless in test mode). + test_mode: If True, the function runs in test mode. + + Returns: + A unique list of Celery queue names with workers attached to them. """ running_queues = [] @@ -111,26 +111,36 @@ def get_running_queues(celery_app_name: str, test_mode: bool = False) -> List[st return running_queues -def get_active_celery_queues(app): - """Get all active queues and workers for a celery application. - - Unlike get_running_queues, this goes through the application's server. - Also returns a dictionary with entries for each worker attached to - the given queues. - - :param `celery.Celery` app: the celery application - - :return: queues dictionary with connected workers, all workers - :rtype: (dict of lists of strings, list of strings) - - :example: - - >>> from merlin.celery import app - >>> queues, workers = get_active_celery_queues(app) - >>> queue_names = [*queues] - >>> workers_on_q0 = queues[queue_names[0]] - >>> workers_not_on_q0 = [worker for worker in workers - if worker not in workers_on_q0] +def get_active_celery_queues(app: Celery) -> Tuple[Dict[str, List[str]], List[str]]: + """ + Retrieve all active queues and their associated workers for a Celery application. + + This function queries the application's server to obtain a comprehensive + view of active queues and the workers connected to them. It returns a + dictionary where each key is a queue name and the value is a list of + workers attached to that queue. Additionally, it provides a list of all + active workers in the application. + + Note: + Unlike [`get_running_queues`][study.celeryadapter.get_running_queues], + this function goes through the application's server. + + Args: + app: The Celery application instance. + + Returns: + A tuple containing:\n + - A dictionary mapping queue names to lists of workers connected to them. + - A list of all active workers in the application. + + Example: + ```python + from merlin.celery import app + queues, workers = get_active_celery_queues(app) + queue_names = list(queues) + workers_on_q0 = queues[queue_names[0]] + workers_not_on_q0 = [worker for worker in workers if worker not in workers_on_q0] + ``` """ i = app.control.inspect() active_workers = i.active_queues() @@ -146,14 +156,22 @@ def get_active_celery_queues(app): return queues, [*active_workers] -def get_active_workers(app): +def get_active_workers(app: Celery) -> Dict[str, List[str]]: """ - This is the inverse of get_active_celery_queues() defined above. This function - builds a dict where the keys are worker names and the values are lists - of queues attached to the worker. + Retrieve a mapping of active workers to their associated queues for a Celery application. + + This function serves as the inverse of + [`get_active_celery_queues()`][study.celeryadapter.get_active_celery_queues]. It constructs + a dictionary where each key is a worker's name and the corresponding value is a + list of queues that the worker is connected to. This allows for easy identification + of which queues are being handled by each worker. + + Args: + app: The Celery application instance. - :param `app`: The celery application - :returns: A dict mapping active workers to queues + Returns: + A dictionary mapping active worker names to lists of queue names they are + attached to. If no active workers are found, an empty dictionary is returned. """ # Get the information we need from celery i = app.control.inspect() @@ -173,14 +191,19 @@ def get_active_workers(app): return worker_queue_map -def celerize_queues(queues: List[str], config: Optional[Dict] = None): +def celerize_queues(queues: List[str], config: SimpleNamespace = None): """ - Celery requires a queue tag to be prepended to their - queues so this function will 'celerize' every queue in - a list you provide it by prepending the queue tag. + Prepend a queue tag to each queue in the provided list to conform to Celery's + queue naming requirements. - :param `queues`: A list of queues that need the queue tag prepended. - :param `config`: A dict of configuration settings + This function modifies the input list of queues by adding a specified queue tag + from the configuration. If no configuration is provided, it defaults to using + the global configuration settings. + + Args: + queues: A list of queue names that need the queue tag prepended. + config: A SimpleNamespace of configuration settings. If not provided, the + function will use the default configuration. """ if config is None: from merlin.config.configfile import CONFIG as config # pylint: disable=C0415 @@ -189,14 +212,18 @@ def celerize_queues(queues: List[str], config: Optional[Dict] = None): queues[i] = f"{config.celery.queue_tag}{queue}" -def _build_output_table(worker_list, output_table): +def _build_output_table(worker_list: List[str], output_table: List[Tuple[str, str]]): """ - Helper function for query-status that will build a table - that we'll use as output. + Construct an output table for displaying the status of workers and their associated queues. + + This helper function populates the provided output table with entries for each worker + in the given worker list. It retrieves the mapping of active workers to their queues + and formats the data accordingly. - :param `worker_list`: A list of workers to add to the table - :param `output_table`: A list of tuples where each entry is - of the form (worker name, associated queues) + Args: + worker_list: A list of worker names to be included in the output table. + output_table: A list of tuples where each entry will be of the form + (worker name, associated queues). """ from merlin.celery import app # pylint: disable=C0415 @@ -211,15 +238,22 @@ def _build_output_table(worker_list, output_table): output_table.append((worker, ", ".join(worker_queue_map[worker]))) -def query_celery_workers(spec_worker_names, queues, workers_regex): +def query_celery_workers(spec_worker_names: List[str], queues: List[str], workers_regex: List[str]): """ - Look for existing celery workers. Filter by spec, queues, or - worker names if provided by user. At the end, print a table - of workers and their associated queues. - - :param `spec_worker_names`: The worker names defined in a spec file - :param `queues`: A list of queues to filter by - :param `workers_regex`: A list of regexs to filter by + Query and filter existing Celery workers based on specified criteria, + and print a table of the workers along with their associated queues. + + This function retrieves the list of active Celery workers and filters them + according to the provided specifications, including worker names from a + spec file, specific queues, and regular expressions for worker names. + It then constructs and displays a table of the matching workers and their + associated queues. + + Args: + spec_worker_names: A list of worker names defined in a spec file + to filter the workers. + queues: A list of queues to filter the workers by. + workers_regex: A list of regular expressions to filter the worker names. """ from merlin.celery import app # pylint: disable=C0415 @@ -280,11 +314,21 @@ def query_celery_workers(spec_worker_names, queues, workers_regex): def build_csv_queue_info(query_return: List[Tuple[str, int, int]], date: str) -> Dict[str, List]: """ - Build the lists of column labels and queue info to write to the csv file. + Construct a dictionary containing queue information and column labels + for writing to a CSV file. - :param query_return: The output of `query_queues` - :param date: A timestamp for us to mark when this status occurred - :returns: A dict of queue information to dump to a csv file + This function processes the output from the [`query_queues`][router.query_queues] + function and organizes the data into a format suitable for CSV export. It includes + a timestamp to indicate when the status was recorded. + + Args: + query_return: The output from the [`query_queues`][router.query_queues] function, + containing queue names and their associated statistics. + date: A timestamp indicating when the queue status was recorded. + + Returns: + A dictionary where keys are column labels and values are lists containing the + corresponding queue information, formatted for CSV output. """ # Build the list of labels if necessary csv_to_dump = {"time": [date]} @@ -297,11 +341,22 @@ def build_csv_queue_info(query_return: List[Tuple[str, int, int]], date: str) -> def build_json_queue_info(query_return: List[Tuple[str, int, int]], date: str) -> Dict: """ - Build the dict of queue info to dump to the json file. - - :param query_return: The output of `query_queues` - :param date: A timestamp for us to mark when this status occurred - :returns: A dictionary that's ready to dump to a json outfile + Construct a dictionary containing queue information for JSON export. + + This function processes the output from the [`query_queues`][router.query_queues] + function and organizes the data into a structured format suitable for JSON + serialization. It includes a timestamp to indicate when the queue status was + recorded. + + Args: + query_return: The output from the [`query_queues`][router.query_queues] + function, containing queue names and their associated statistics. + date: A timestamp indicating when the queue status was recorded. + + Returns: + A dictionary structured for JSON output, where the keys are timestamps + and the values are dictionaries containing queue names and their + corresponding statistics (tasks and consumers). """ # Get the datetime so we can track different entries and initalize a new json entry json_to_dump = {date: {}} @@ -315,11 +370,17 @@ def build_json_queue_info(query_return: List[Tuple[str, int, int]], date: str) - def dump_celery_queue_info(query_return: List[Tuple[str, int, int]], dump_file: str): """ - Format the information we're going to dump in a way that the Dumper class can - understand and add a timestamp to the info. + Format and dump Celery queue information to a specified file. + + This function processes the output from the `query_queues` function, formats + the data according to the file type (CSV or JSON), and adds a timestamp + to the information before writing it to the specified file. - :param query_return: The output of `query_queues` - :param dump_file: The filepath of the file we'll dump queue info to + Args: + query_return: The output from the [`query_queues`][router.query_queues] + function, containing queue names and their associated statistics. + dump_file: The filepath of the file where the queue information + will be written. The file extension determines the format (CSV or JSON). """ # Get a timestamp for this dump date = datetime.now().strftime("%Y-%m-%d %H:%M:%S") @@ -336,16 +397,25 @@ def dump_celery_queue_info(query_return: List[Tuple[str, int, int]], dump_file: dump_handler(dump_file, dump_info) -def _get_specific_queues(queues: set, specific_queues: List[str], spec: "MerlinSpec", verbose=True) -> set: # noqa: F821 +def _get_specific_queues(queues: Set[str], specific_queues: List[str], spec: MerlinSpec, verbose: bool = True) -> Set[str]: """ - Search for specific queues that the user asked for. The queues that cannot be found will not - be returned. The queues that can be found will be added to a set and returned. - - :param queues: Either an empty set or a set of queues from `spec` - :param specific_queues: The list of queues that we're going to search for - :param spec: A `MerlinSpec` object or None - :param verbose: If True, display log messages. Otherwise, don't. - :returns: A set of the specific queues that were found to exist. + Retrieve a set of specific queues requested by the user, filtering out those that do not exist. + + This function checks a provided list of specific queues against a set of existing queues + (from a [`MerlinSpec`][spec.specification.MerlinSpec] object) and returns a set of queues + that are found. If a queue is not found in the existing set, it will be excluded from the + results. The function also logs messages based on the verbosity setting. + + Args: + queues: A set of existing queues, which may be empty or populated from the `spec` + object. + specific_queues: A list of specific queue names to search for. + spec (spec.specification.MerlinSpec): A [`MerlinSpec`][spec.specification.MerlinSpec] + object that may provide context for the search. Can be None. + verbose: If True, log messages will be displayed. + + Returns: + A set containing the specific queues that were found in the existing queues. """ if verbose: LOG.info(f"Filtering queues to query by these specific queues: {specific_queues}") @@ -376,21 +446,30 @@ def _get_specific_queues(queues: set, specific_queues: List[str], spec: "MerlinS def build_set_of_queues( - spec: "MerlinSpec", # noqa: F821 + spec: MerlinSpec, steps: List[str], specific_queues: List[str], - verbose: Optional[bool] = True, - app: Optional["Celery"] = None, # noqa: F821 -) -> set: + verbose: bool = True, + app: Celery = None, +) -> Set[str]: """ - Build a set of queues to query based on the parameters given here. - - :param spec: A `MerlinSpec` object or None - :param steps: Spaced-separated list of stepnames to query. Default is all - :param specific_queues: A list of queue names to query or None - :param verbose: A bool to determine whether to output log statements or not - :param app: A celery app object, if left out we'll just import it - :returns: A set of queues to investigate + Construct a set of queues to query based on the provided parameters. + + This function builds a set of queues by querying a [`MerlinSpec`][spec.specification.MerlinSpec] + object for queues associated with specified steps and/or filtering for specific queue names. + If no spec or specific queues are provided, it defaults to querying active queues from the Celery + application. + + Args: + spec (spec.specification.MerlinSpec): A [`MerlinSpec`][spec.specification.MerlinSpec] + object that defines the context for the query. Can be None. + steps: A list of step names to query. If empty, all steps are considered. + specific_queues: A list of specific queue names to filter. Can be None. + verbose: If True, log statements will be output. Defaults to True. + app: A Celery application instance. If None, it will be imported. + + Returns: + A set of queue names to investigate based on the provided parameters. """ if app is None: from merlin.celery import app # pylint: disable=C0415 @@ -424,15 +503,32 @@ def build_set_of_queues( return queues -def query_celery_queues(queues: List[str], app: Celery = None, config: Config = None) -> Dict[str, List[str]]: +def query_celery_queues(queues: List[str], app: Celery = None, config: Config = None) -> Dict[str, Dict[str, int]]: """ - Build a dict of information about the number of jobs and consumers attached - to specific queues that we want information on. - - :param queues: A list of the queues we want to know about - :param app: The celery application (this will be none unless testing) - :param config: The configuration object that has the broker name (this will be none unless testing) - :returns: A dict of info on the number of jobs and consumers for each queue in `queues` + Retrieve information about the number of jobs and consumers for specified Celery queues. + + This function constructs a dictionary containing details about the number of jobs + and consumers associated with each queue provided in the input list. It connects + to the Celery application to gather this information, handling both Redis and + RabbitMQ brokers. + + Notes: + - If the specified queue does not exist or has no jobs, it will be handled gracefully. + - For Redis brokers, the function counts consumers by inspecting active queues + since Redis does not track consumers like RabbitMQ does. + + Args: + queues: A list of queue names for which to gather information. + app: The Celery application instance. Defaults to None, which triggers an import + for testing purposes. + config (config.Config): A configuration object containing broker details. + Defaults to None, which also triggers an import for testing. + + Returns: + A dictionary where each key is a queue name and the value is another dictionary + containing:\n + - `jobs`: The number of jobs in the queue. + - `consumers`: The number of consumers attached to the queue. """ if app is None: from merlin.celery import app # pylint: disable=C0415 @@ -474,12 +570,17 @@ def query_celery_queues(queues: List[str], app: Celery = None, config: Config = return queue_info -def get_workers_from_app(): - """Get all workers connected to a celery application. +def get_workers_from_app() -> List[str]: + """ + Retrieve a list of all workers connected to the Celery application. + + This function uses the Celery control interface to inspect the current state + of the application and returns a list of workers that are currently connected. + If no workers are found, an empty list is returned. - :param `celery.Celery` app: the celery application - :return: A list of all connected workers - :rtype: list + Returns: + A list of worker names that are currently connected to the Celery application. + If no workers are connected, an empty list is returned. """ from merlin.celery import app # pylint: disable=C0415 @@ -492,11 +593,19 @@ def get_workers_from_app(): def check_celery_workers_processing(queues_in_spec: List[str], app: Celery) -> bool: """ - Query celery to see if any workers are still processing tasks. + Check if any Celery workers are currently processing tasks from specified queues. + + This function queries the Celery application to determine if there are any active + tasks being processed by workers for the given list of queues. It returns a boolean + indicating whether any tasks are currently active. - :param queues_in_spec: A list of queues to check if tasks are still active in - :param app: The celery app that we're querying - :returns: True if workers are still processing tasks, False otherwise + Args: + queues_in_spec: A list of queue names to check for active tasks. + app: The Celery application instance used for querying. + + Returns: + True if any workers are processing tasks in the specified queues; False + otherwise. """ # Query celery for active tasks active_tasks = app.control.inspect().active() @@ -511,15 +620,23 @@ def check_celery_workers_processing(queues_in_spec: List[str], app: Celery) -> b return False -def _get_workers_to_start(spec, steps): +def _get_workers_to_start(spec: MerlinSpec, steps: List[str]) -> Set[str]: """ - Helper function to return a set of workers to start based on - the steps provided by the user. + Determine the set of workers to start based on the specified steps. + + This helper function retrieves a mapping of steps to their corresponding workers + from a [`MerlinSpec`][spec.specification.MerlinSpec] object and returns a unique + set of workers that should be started for the provided list of steps. If a step + is not found in the mapping, a warning is logged. - :param `spec`: A MerlinSpec object - :param `steps`: A list of steps to start workers for + Args: + spec (spec.specification.MerlinSpec): An instance of the + [`MerlinSpec`][spec.specification.MerlinSpec] class that contains the + mapping of steps to workers. + steps: A list of steps for which workers need to be started. - :returns: A set of workers to start + Returns: + A set of unique workers to be started based on the specified steps. """ workers_to_start = [] step_worker_map = spec.get_step_worker_map() @@ -535,14 +652,25 @@ def _get_workers_to_start(spec, steps): return workers_to_start -def _create_kwargs(spec): +def _create_kwargs(spec: MerlinSpec) -> Tuple[Dict[str, str], Dict]: """ - Helper function to handle creating the kwargs dict that - we'll pass to subprocess.Popen when we launch the worker. - - :param `spec`: A MerlinSpec object - :returns: A tuple where the first entry is the kwargs and - the second entry is variables defined in the spec + Construct the keyword arguments for launching a worker process. + + This helper function creates a dictionary of keyword arguments that will be + passed to `subprocess.Popen` when launching a worker. It retrieves the + environment variables defined in a [`MerlinSpec`][spec.specification.MerlinSpec] + object and updates the shell environment accordingly. + + Args: + spec (spec.specification.MerlinSpec): An instance of the MerlinSpec class + that contains environment specifications. + + Returns: + A tuple containing: + - A dictionary of keyword arguments for `subprocess.Popen`, including + the updated environment. + - A dictionary of variables defined in the spec, or None if no variables + were defined. """ # Get the environment from the spec and the shell spec_env = spec.environment @@ -563,15 +691,24 @@ def _create_kwargs(spec): return kwargs, yaml_vars -def _get_steps_to_start(wsteps, steps, steps_provided): +def _get_steps_to_start(wsteps: List[str], steps: List[str], steps_provided: bool) -> List[str]: """ - Determine which steps to start workers for. - - :param `wsteps`: A list of steps associated with a worker - :param `steps`: A list of steps to start provided by the user - :param `steps`: A bool representing whether the user gave specific - steps to start or not - :returns: A list of steps to start workers for + Identify the steps for which workers should be started. + + This function determines which steps to initiate based on the steps + associated with a worker and the user-provided steps. If specific steps + are provided by the user, only those steps that match the worker's steps + will be included. If no specific steps are provided, all worker-associated + steps will be returned. + + Args: + wsteps: A list of steps that are associated with a worker. + steps: A list of steps specified by the user to start workers for. + steps_provided: A boolean indicating whether the user provided + specific steps to start. + + Returns: + A list of steps for which workers should be started. """ steps_to_start = [] if steps_provided: @@ -584,31 +721,48 @@ def _get_steps_to_start(wsteps, steps, steps_provided): return steps_to_start -def start_celery_workers(spec, steps, celery_args, disable_logs, just_return_command): # pylint: disable=R0914,R0915 - """Start the celery workers on the allocation - - :param MerlinSpec spec: A MerlinSpec object representing our study - :param list steps: A list of steps to start workers for - :param str celery_args: A string of arguments to provide to the celery workers - :param bool disable_logs: A boolean flag to turn off the celery logs for the workers - :param bool just_return_command: When True, workers aren't started and just the launch command(s) - are returned - :side effect: Starts subprocesses for each worker we launch - :returns: A string of all the worker launch commands - ... - - example config: - - merlin: - resources: - task_server: celery - overlap: False - workers: - simworkers: - args: -O fair --prefetch-multiplier 1 -E -l info --concurrency 4 - steps: [run, data] - nodes: 1 - machine: [hostA, hostB] +def start_celery_workers( + spec: MerlinSpec, steps: List[str], celery_args: str, disable_logs: bool, just_return_command: bool +) -> str: # pylint: disable=R0914,R0915 + """ + Start Celery workers based on the provided specifications and steps. + + This function initializes and starts Celery workers for the specified steps + in the given [`MerlinSpec`][spec.specification.MerlinSpec]. It constructs + the necessary command-line arguments and handles the launching of subprocesses + for each worker. If the `just_return_command` flag is set to `True`, it will + return the command(s) to start the workers without actually launching them. + + Args: + spec (spec.specification.MerlinSpec): A [`MerlinSpec`][spec.specification.MerlinSpec] + object representing the study configuration. + steps: A list of steps for which to start workers. + celery_args: A string of additional arguments to pass to the Celery workers. + disable_logs: A flag to disable logging for the Celery workers. + just_return_command: If `True`, returns the launch command(s) without starting the workers. + + Returns: + A string containing all the worker launch commands. + + Side Effects: + - Starts subprocesses for each worker that is launched, so long as `just_return_command` + is not True. + + Example: + Below is an example configuration for Merlin workers: + + ```yaml + merlin: + resources: + task_server: celery + overlap: False + workers: + simworkers: + args: -O fair --prefetch-multiplier 1 -E -l info --concurrency 4 + steps: [run, data] + nodes: 1 + machine: [hostA, hostB] + ``` """ if not just_return_command: LOG.info("Starting workers") @@ -701,10 +855,25 @@ def start_celery_workers(spec, steps, celery_args, disable_logs, just_return_com return str(worker_list) -def examine_and_log_machines(worker_val, yenv) -> bool: +def examine_and_log_machines(worker_val: Dict, yenv: Dict[str, str]) -> bool: """ - Examines whether a worker should be skipped in a step of start_celery_workers(), logs errors in output path for a celery - worker. + Determine if a worker should be skipped based on machine availability and log any errors. + + This function checks the specified machines for a worker and determines + whether the worker can be started. If the machines are not available, + it logs an error message regarding the output path for the Celery worker. + If the environment variables (`yenv`) are not provided or do not specify + an output path, a warning is logged. + + Args: + worker_val: A dictionary containing worker configuration, including + the list of machines associated with the worker. + yenv: A dictionary of environment variables that may include the + output path for logging. + + Returns: + Returns `True` if the worker should be skipped (i.e., machines are + unavailable), otherwise returns `False`. """ worker_machines = get_yaml_var(worker_val, "machines", None) if worker_machines: @@ -725,8 +894,27 @@ def examine_and_log_machines(worker_val, yenv) -> bool: return False -def verify_args(spec, worker_args, worker_name, overlap, disable_logs=False): - """Examines the args passed to a worker for completeness.""" +def verify_args(spec: MerlinSpec, worker_args: str, worker_name: str, overlap: bool, disable_logs: bool = False) -> str: + """ + Validate and enhance the arguments passed to a Celery worker for completeness. + + This function checks the provided worker arguments to ensure that they include + recommended settings for running parallel tasks. It adds default values for + concurrency, prefetch multiplier, and logging level if they are not specified. + Additionally, it generates a unique worker name based on the current time if + the `-n` argument is not provided. + + Args: + spec (spec.specification.MerlinSpec): A [`MerlinSpec`][spec.specification.MerlinSpec] + object containing the study configuration. + worker_args: A string of arguments passed to the worker that may need validation. + worker_name: The name of the worker, used for generating a unique worker identifier. + overlap: A flag indicating whether multiple workers can overlap in their queue processing. + disable_logs: A flag to disable logging configuration for the worker. + + Returns: + The validated and potentially modified worker arguments string. + """ parallel = batch_check_parallel(spec) if parallel: if "--concurrency" not in worker_args: @@ -750,30 +938,57 @@ def verify_args(spec, worker_args, worker_name, overlap, disable_logs=False): return worker_args -def launch_celery_worker(worker_cmd, worker_list, kwargs): +def launch_celery_worker(worker_cmd: str, worker_list: List[str], kwargs: Dict): """ - Using the worker launch command provided, launch a celery worker. - :param str worker_cmd: The celery command to launch a worker - :param list worker_list: A list of worker launch commands - :param dict kwargs: A dictionary containing additional keyword args to provide - to subprocess.Popen - - :side effect: Launches a celery worker via a subprocess + Launch a Celery worker using the specified command and parameters. + + This function executes the provided Celery command to start a worker as a + subprocess. It appends the command to the given list of worker commands + for tracking purposes. If the worker fails to start, an error is logged. + + Args: + worker_cmd: The command string used to launch the Celery worker. + worker_list: A list that will be updated to include the launched + worker command for tracking active workers. + kwargs: A dictionary of additional keyword arguments to pass to + `subprocess.Popen`, allowing for customization of the subprocess + behavior. + + Raises: + Exception: If the worker fails to start, an error is logged, and the + exception is re-raised. + + Side Effects: + - Launches a Celery worker process in the background. + - Modifies the `worker_list` by appending the launched worker command. """ try: - _ = subprocess.Popen(worker_cmd, **kwargs) # pylint: disable=R1732 + subprocess.Popen(worker_cmd, **kwargs) # pylint: disable=R1732 worker_list.append(worker_cmd) except Exception as e: # pylint: disable=C0103 LOG.error(f"Cannot start celery workers, {e}") raise -def get_celery_cmd(queue_names, worker_args="", just_return_command=False): +def get_celery_cmd(queue_names: str, worker_args: str = "", just_return_command: bool = False) -> str: """ - Get the appropriate command to launch celery workers for the specified MerlinStudy. - queue_names The name(s) of the queue(s) to associate a worker with - worker_args Optional celery arguments for the workers - just_return_command Don't execute, just return the command + Construct the command to launch Celery workers for the specified queues. + + This function generates a command string that can be used to start Celery + workers associated with the provided queue names. It allows for optional + worker arguments to be included and can return the command without executing it. + + Args: + queue_names: A comma-separated string of the queue name(s) to which the worker + will be associated. + worker_args: Additional command-line arguments for the Celery worker. + just_return_command: If True, the function will return the constructed command + without executing it. + + Returns: + The constructed command string for launching the Celery worker. If + `just_return_command` is True, returns the command; otherwise, returns an + empty string. """ worker_command = " ".join(["celery -A merlin worker", worker_args, "-Q", queue_names]) if just_return_command: @@ -783,12 +998,24 @@ def get_celery_cmd(queue_names, worker_args="", just_return_command=False): return "" -def purge_celery_tasks(queues, force): +def purge_celery_tasks(queues: str, force: bool) -> int: """ - Purge celery tasks for the specified spec file. - - queues Which queues to purge - force Purge without asking for confirmation + Purge Celery tasks from the specified queues. + + This function constructs and executes a command to purge tasks from the + specified Celery queues. If the `force` parameter is set to True, the + purge operation will be executed without prompting for confirmation. + + Args: + queues: A comma-separated string of the queue name(s) from which + tasks should be purged. + force: If True, the purge operation will be executed without asking + for user confirmation. + + Returns: + The return code from the subprocess execution. A return code of + 0 indicates success, while any non-zero value indicates an error + occurred during the purge operation. """ # This version will purge all queues. # from merlin.celery import app @@ -801,24 +1028,33 @@ def purge_celery_tasks(queues, force): return subprocess.run(purge_command, shell=True).returncode -def stop_celery_workers(queues=None, spec_worker_names=None, worker_regex=None): # pylint: disable=R0912 - """Send a stop command to celery workers. - - Default behavior is to stop all connected workers. - As options can downselect to only workers on certain queues and/or that - match a regular expression. - - :param list queues: The queues to send stop signals to. If None: stop all - :param list spec_worker_names: Worker names read from a spec to stop, in addition to worker_regex matches. - :param str worker_regex: The regex string to match worker names. If None: - :return: Return code from stop command - - :example: - - >>> stop_celery_workers(queues=['hello'], worker_regex='celery@*my_machine*') - - >>> stop_celery_workers() - +def stop_celery_workers( + queues: List[str] = None, spec_worker_names: List[str] = None, worker_regex: List[str] = None +): # pylint: disable=R0912 + """ + Send a stop command to Celery workers. + + This function sends a shutdown command to Celery workers associated with + specified queues. By default, it stops all connected workers, but it can + be configured to target specific workers based on queue names or regular + expression patterns. + + Args: + queues: A list of queue names to which the stop command will be sent. + If None, all connected workers across all queues will be stopped. + spec_worker_names: A list of specific worker names to stop, in addition + to those matching the `worker_regex`. + worker_regex: A regular expression string used to match worker names. + If None, no regex filtering will be applied. + + Side Effects: + - Broadcasts a shutdown signal to Celery workers + + Example: + ```python + stop_celery_workers(queues=['hello'], worker_regex='celery@*my_machine*') + stop_celery_workers() + ``` """ from merlin.celery import app # pylint: disable=C0415 @@ -865,37 +1101,7 @@ def stop_celery_workers(queues=None, spec_worker_names=None, worker_regex=None): if workers_to_stop: LOG.info(f"Sending stop to these workers: {workers_to_stop}") + # Send the shutdown signal app.control.broadcast("shutdown", destination=workers_to_stop) else: LOG.warning("No workers found to stop") - - -def create_celery_config(config_dir, data_file_name, data_file_path): - """ - Command to setup default celery merlin config. - - :param `config_dir`: The directory to create the config file. - :param `data_file_name`: The name of the config file. - :param `data_file_path`: The full data file path. - """ - # This will need to come from the server interface - MERLIN_CONFIG = os.path.join(config_dir, data_file_name) # pylint: disable=C0103 - - if os.path.isfile(MERLIN_CONFIG): - from merlin.common.security import encrypt # pylint: disable=C0415 - - encrypt.init_key() - LOG.info(f"The config file already exists, {MERLIN_CONFIG}") - return - - with open(MERLIN_CONFIG, "w") as outfile, open(data_file_path, "r") as infile: - outfile.write(infile.read()) - - if not os.path.isfile(MERLIN_CONFIG): - LOG.error(f"Cannot create config file {MERLIN_CONFIG}") - - LOG.info(f"The file {MERLIN_CONFIG} is ready to be edited for your system.") - - from merlin.common.security import encrypt # pylint: disable=C0415 - - encrypt.init_key() diff --git a/merlin/study/dag.py b/merlin/study/dag.py index 06f75beac..969d125f7 100644 --- a/merlin/study/dag.py +++ b/merlin/study/dag.py @@ -1,85 +1,111 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ -Holds DAG class. TODO make this an interface, separate from Maestro. +Holds the Merlin Directed Acyclic Graph (DAG) class. """ from collections import OrderedDict +from typing import Dict, List from merlin.study.step import Step +# TODO make this an interface, separate from Maestro. class DAG: """ This class provides methods on a task graph that Merlin needs for staging - tasks in celery. It is initialized from am maestro ExecutionGraph, and the + tasks in Celery. It is initialized from a Maestro `ExecutionGraph`, and the major entry point is the group_tasks method, which provides groups of independent chains of tasks. + + Attributes: + backwards_adjacency (Dict): A dictionary mapping each task to its parent tasks for reverse + traversal. + column_labels (List[str]): A list of column labels provided in the spec file. + maestro_adjacency_table (OrderedDict): An ordered dict showing adjacency of nodes. Comes from + a maestrowf `ExecutionGraph`. + maestro_values (OrderedDict): An ordered dict of the values at each node. Comes from a maestrowf + `ExecutionGraph`. + parameter_info (Dict): A dict containing information about parameters in the study. + study_name (str): The name of the study. + + Methods: + calc_backwards_adjacency: Initializes the backwards adjacency table. + calc_depth: Calculate the depth of the given node and its children. + children: Return the children of the task. + compatible_merlin_expansion: Check if two tasks are compatible for Merlin expansion. + find_chain: Find the chain containing the task. + find_independent_chains: Finds independent chains and adjusts with the groups of chains + to maximize parallelism. + group_by_depth: Group Directed Acyclic Graph (DAG) tasks by depth. + group_tasks: Group independent tasks in a DAG. + num_children: Find the number of children for the given task in the DAG. + num_parents: Find the number of parents for the given task in the DAG. + parents: Return the parents of the task. + step: Return a [`Step`][study.step.Step] object for the given task name. """ def __init__( - self, maestro_adjacency_table, maestro_values, column_labels, study_name, parameter_info + self, + maestro_adjacency_table: OrderedDict, + maestro_values: OrderedDict, + column_labels: List[str], + study_name: str, + parameter_info: Dict, ): # pylint: disable=R0913 """ - :param `maestro_adjacency_table`: An ordered dict showing adjacency of nodes. Comes from a maestrowf ExecutionGraph. - :param `maestro_values`: An ordered dict of the values at each node. Comes from a maestrowf ExecutionGraph. - :param `column_labels`: A list of column labels provided in the spec file. - :param `study_name`: The name of the study - :param `parameter_info`: A dict containing information about parameters in the study + Initializes a Directed Acyclic Graph (DAG) object, which represents a task graph used by Merlin + for staging tasks in Celery. The DAG is initialized from a Maestro `ExecutionGraph` by unpacking + its adjacency table and node values. + + Args: + maestro_adjacency_table: An ordered dictionary representing the adjacency + relationships between tasks in the graph. This comes from a Maestro `ExecutionGraph`. + maestro_values: An ordered dictionary containing the values or metadata + associated with each task in the graph. This also comes from a Maestro `ExecutionGraph`. + column_labels: A list of column labels provided in the specification file, + typically used to identify parameters or task attributes. + study_name: The name of the study to which this DAG belongs. + parameter_info: A dictionary containing information about the parameters in the study, + such as their names and values. """ # We used to store the entire maestro ExecutionGraph here but now it's # unpacked so we're only storing the 2 attributes from it that we use: # the adjacency table and the values. This had to happen to get pickle # to work for Celery. - self.maestro_adjacency_table = maestro_adjacency_table - self.maestro_values = maestro_values - self.column_labels = column_labels - self.study_name = study_name - self.parameter_info = parameter_info - self.backwards_adjacency = {} + self.maestro_adjacency_table: OrderedDict = maestro_adjacency_table + self.maestro_values: OrderedDict = maestro_values + self.column_labels: List[str] = column_labels + self.study_name: str = study_name + self.parameter_info: Dict = parameter_info + self.backwards_adjacency: Dict = {} self.calc_backwards_adjacency() - def step(self, task_name): - """Return a Step object for the given task name + def step(self, task_name: str) -> Step: + """ + Return a Step object for the given task name. + + Args: + task_name: The task name. - :param `task_name`: The task name. - :return: A Merlin Step object. + Returns: + A Merlin [`Step`][study.step.Step] object representing the + task's configuration and parameters. """ return Step(self.maestro_values[task_name], self.study_name, self.parameter_info) - def calc_depth(self, node, depths, current_depth=0): - """Calculate the depth of the given node and its children. + def calc_depth(self, node: str, depths: Dict, current_depth: int = 0): + """ + Calculate the depth of the given node and its children. This recursive + method will update `depths` in place. - :param `node`: The node (str) to start at. - :param `depths`: the dictionary of depths to update. - :param `current_depth`: the current depth in the graph traversal. + Args: + node: The node to start at. + depths: The dictionary of depths to update. + current_depth: The current depth in the graph traversal. """ if node not in depths: depths[node] = current_depth @@ -90,22 +116,30 @@ def calc_depth(self, node, depths, current_depth=0): self.calc_depth(child, depths, current_depth=depths[node] + 1) @staticmethod - def group_by_depth(depths): - """Group DAG tasks by depth. + def group_by_depth(depths: Dict) -> List[List[List]]: + """ + Group Directed Acyclic Graph (DAG) tasks by depth. + + This method only groups by depth, and has one task in every chain. + [`find_independent_chains`][study.dag.DAG.find_independent_chains] is used + to figure out how to coalesce chains across depths. - :param `depths`: the dictionary of depths to group by + Args: + depths: The dictionary of depths to group by. - :return: a list of lists of lists ordered by depth + Returns: + A list of lists of lists ordered by depth. - ([[["tasks"],["with"],["Depth 0"]],[["tasks"],["with"],["Depth 1"]]]) + Example: + This method will return a list that could look something like this: - The outer index of this list is the depth, the middle index is which - chain of tasks in that depth, and the inner index is the task id in - that chain. + ```python + [[["tasks"], ["with"], ["Depth 0"]], [["tasks"], ["with"], ["Depth 1"]]] + ``` - This method only groups by depth, and has one task in every chain. - find_independent_chains is used to figure out how to coalesce chains - across depths. + Here, the outer index of this list is the depth, the middle index is + which chain of tasks in that depth, and the inner index is the task + id in that chain. """ groups = {} for node in depths: @@ -123,44 +157,66 @@ def group_by_depth(depths): return list_of_groups_of_chains - def children(self, task_name): - """Return the children of the task. - :param `task_name`: The name of the task to get the children of. + def children(self, task_name: str) -> List: + """ + Return the children of the task. + + Args: + task_name: The name of the task to get the children of. - :return: list of children of this task. + Returns: + List of children of this task. """ return self.maestro_adjacency_table[task_name] - def num_children(self, task_name): - """Find the number of children for the given task in the dag. - :param `task_name`: The name of the task to count the children of. + def num_children(self, task_name: str) -> int: + """ + Find the number of children for the given task in the Directed Acyclic Graph (DAG). + + Args: + task_name: The name of the task to count the children of. - :return : number of children this task has + Returns: + Number of children this task has. """ return len(self.children(task_name)) - def parents(self, task_name): - """Return the parents of the task. - :param `task_name` : The name of the task to get the parents of. + def parents(self, task_name: str) -> List: + """ + Return the parents of the task. + + Args: + task_name: The name of the task to get the parents of. - :return : list of parents of this task""" + Returns: + List of parents of this task. + """ return self.backwards_adjacency[task_name] - def num_parents(self, task_name): - """find the number of parents for the given task in the dag - :param `task_name` : The name of the task to count the parents of + def num_parents(self, task_name: str) -> int: + """ + Find the number of parents for the given task in the Directed Acyclic Graph (DAG). + + Args: + task_name: The name of the task to count the parents of. - :return : number of parents this task has""" + Returns: + Number of parents this task has. + """ return len(self.parents(task_name)) @staticmethod - def find_chain(task_name, list_of_groups_of_chains): - """find the chain containing the task - :param `task_name` : The task to search for. - :param `list_of_groups_of_chains` : list of groups of chains to search - for the task + def find_chain(task_name: str, list_of_groups_of_chains: List[List[List]]) -> List: + """ + Find the chain containing the task. - :return : the list representing the chain containing task_name""" + Args: + task_name: The task to search for. + list_of_groups_of_chains: List of groups of chains to search for the task. + + Returns: + The list representing the chain containing task_name, or None if not found. + """ for group in list_of_groups_of_chains: for chain in group: if task_name in chain: @@ -168,7 +224,42 @@ def find_chain(task_name, list_of_groups_of_chains): return None def calc_backwards_adjacency(self): - """initializes our backwards adjacency table""" + """ + Initializes the backwards adjacency table. + + This method constructs a mapping of each task to its parent tasks in the Directed + Acyclic Graph (DAG). The backwards adjacency table allows for reverse traversal + of the graph, enabling the identification of dependencies for each task. + + The method iterates through each parent task in the `maestro_adjacency_table` + and updates the `backwards_adjacency` dictionary. For each task that is a child + of a parent, it adds the parent to the list of that task's parents in the + `backwards_adjacency` table. + + This is essential for operations that require knowledge of a task's dependencies, + such as determining the order of execution or identifying independent tasks. + + Example: + If the `maestro_adjacency_table` is structured as follows: + + ```python + { + 'A': ['B', 'C'], + 'B': ['D'], + 'C': ['D'] + } + ``` + + After calling this method, the `backwards_adjacency` will be: + + ```python + { + 'B': ['A'], + 'C': ['A'], + 'D': ['B', 'C'] + } + ``` + """ for parent in self.maestro_adjacency_table: for task_name in self.maestro_adjacency_table[parent]: if task_name in self.backwards_adjacency: @@ -176,34 +267,52 @@ def calc_backwards_adjacency(self): else: self.backwards_adjacency[task_name] = [parent] - def compatible_merlin_expansion(self, task1, task2): + def compatible_merlin_expansion(self, task1: str, task2: str) -> bool: """ - TODO + Check if two tasks are compatible for Merlin expansion. + + This method compares the expansion needs of two tasks to determine + if they can be expanded together. + + Args: + task1: The first task. + task2: The second task. + + Returns: + True if compatible, False otherwise. """ step1 = self.step(task1) step2 = self.step(task2) return step1.check_if_expansion_needed(self.column_labels) == step2.check_if_expansion_needed(self.column_labels) - def find_independent_chains(self, list_of_groups_of_chains): + def find_independent_chains(self, list_of_groups_of_chains: List[List[List]]) -> List[List[List]]: """ - Finds independent chains and adjusts with the groups of chains to - maximalize parallelism + Finds independent chains and adjusts with the groups of chains to maximize parallelism. - :param list_of_groups_of_chains: List of list of lists, as returned by - self.group_by_depth + This method looks for opportunities to move tasks in deeper groups of chains + into chains in shallower groups, thus increasing available parallelism in execution. - e.g., + Args: + list_of_groups_of_chains: List of list of lists, as returned by + [`group_by_depth`][study.dag.DAG.group_by_depth]. - ([[["task1"],["with"],["Depth 0"]],[["task2"],["has"],["Depth 1"]]]) + Returns: + Adjusted list of groups of chains to maximize parallelism. - :return : This takes the groups of chains and looks for opportunities - to move tasks in deeper groups of chains into chains in shallower - groups, thus increasing available parallelism in the execution. + Example: + Given input chains, the method may return a modified structure that allows + for more tasks to be executed in parallel. For example, we might start with + this: - Depending on the precise parental relationships between the tasks - in the graph the output may be something like: + ```python + [[["task1"], ["with"], ["Depth 0"]], [["task2"], ["has"], ["Depth 1"]]] + ``` - ([[["task1", "has"],["with","task2"],["Depth 0"]],["Depth 1"]]]) + and finish with this: + + ```python + [[["task1", "has"], ["with", "task2"], ["Depth 0"]], ["Depth 1"]]] + ``` """ for group in list_of_groups_of_chains: for chain in group: @@ -220,14 +329,18 @@ def find_independent_chains(self, list_of_groups_of_chains): return new_list_2 - def group_tasks(self, source_node): - """Group independent tasks in a directed acyclic graph (DAG). + def group_tasks(self, source_node: str) -> List[List[List]]: + """ + Group independent tasks in a Directed Acyclic Graph (DAG). + + Starts from a source node and works down, grouping tasks by depth, + then identifies independent parallel chains in those groups. - Starts from a source node and works down, grouping tasks by - depth, then identify independent parallel chains in those groups. + Args: + source_node: The source node from which to start grouping tasks. - :param dag : The DAG - :param source_node: The source node. + Returns: + A list of independent chains of tasks. """ depths = {} self.calc_depth(source_node, depths) diff --git a/merlin/study/script_adapter.py b/merlin/study/script_adapter.py index b76eb2172..da4ca6283 100644 --- a/merlin/study/script_adapter.py +++ b/merlin/study/script_adapter.py @@ -1,62 +1,47 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ -Merlin script adapter module +This module stores the functionality for adapting bash scripts to use schedulers. + +Supported schedulers are currently: Flux, LSF, and Slurm. """ import logging import os -from typing import Dict, List, Set +from typing import Dict, List, Set, Tuple, Union +from maestrowf.abstracts.enums import StepPriority +from maestrowf.abstracts.interfaces.scriptadapter import ScriptAdapter +from maestrowf.datastructures.core.study import StudyStep from maestrowf.interfaces.script import SubmissionRecord from maestrowf.interfaces.script.localscriptadapter import LocalScriptAdapter from maestrowf.interfaces.script.slurmscriptadapter import SlurmScriptAdapter from maestrowf.utils import start_process -from merlin.common.abstracts.enums import ReturnCode +from merlin.common.enums import ReturnCode from merlin.utils import convert_timestring, find_vlaunch_var LOG = logging.getLogger(__name__) -def setup_vlaunch(step_run: str, batch_type: str, gpu_config: bool) -> None: +def setup_vlaunch(step_run: str, batch_type: str, gpu_config: bool): """ - Check for the VLAUNCHER keyword int the step run string, find - the MERLIN variables and configure VLAUNCHER. + Check for the VLAUNCHER keyword in the step run string and configure VLAUNCHER settings. + + This function examines the provided step run command string for the presence of the + VLAUNCHER keyword. If found, it replaces the keyword with the LAUNCHER keyword and + extracts relevant MERLIN variables such as nodes, processes, and cores per task. + It also configures GPU settings based on the provided boolean flag. - :param `step_run`: the step.run command string - :param `batch_type`: the batch type string - :param `gpu_config`: bool to determin if gpus should be configured - :returns: None + Args: + step_run: The step.run command string that may contain the VLAUNCHER keyword. + batch_type: A string representing the type of batch processing being used. + gpu_config: A boolean indicating whether GPUs should be configured. """ if "$(VLAUNCHER)" in step_run["cmd"]: step_run["cmd"] = step_run["cmd"].replace("$(VLAUNCHER)", "$(LAUNCHER)") @@ -74,20 +59,34 @@ def setup_vlaunch(step_run: str, batch_type: str, gpu_config: bool) -> None: class MerlinLSFScriptAdapter(SlurmScriptAdapter): """ - A SchedulerScriptAdapter class for slurm blocking parallel launches, - the LSFScriptAdapter uses non-blocking submits. + A `SchedulerScriptAdapter` class for SLURM blocking parallel launches. + The `MerlinLSFScriptAdapter` uses non-blocking submits for executing LSF parallel jobs + in a Celery worker. + + Attributes: + key (str): A unique key identifier for the adapter. + _cmd_flags (Dict[str, str]): A dictionary containing command flags for LSF execution. + _unsupported (Set[str]): A set of parameters that are unsupported by this adapter. + + Methods: + get_header: Generates the header for LSF execution scripts. + get_parallelize_command: Generates the LSF parallelization segment of the command line. + get_priority: Overrides the abstract method to fix a pylint error. + write_script: Overwrites the write_script method from the base ScriptAdapter class. """ - key = "merlin-lsf" + key: str = "merlin-lsf" - def __init__(self, **kwargs): + def __init__(self, **kwargs: Dict): """ - Initialize an instance of the MerinLSFScriptAdapter. - The MerlinLSFScriptAdapter is the adapter that is used for workflows that + Initialize an instance of the `MerinLSFScriptAdapter`. + + The `MerlinLSFScriptAdapter` is the adapter that is used for workflows that will execute LSF parallel jobs in a celery worker. The only configurable aspect to this adapter is the shell that scripts are executed in. - :param **kwargs: A dictionary with default settings for the adapter. + Args: + **kwargs: A dictionary with default settings for the adapter. """ super().__init__(**kwargs) @@ -123,27 +122,43 @@ def __init__(self, **kwargs): "walltime", } - def get_priority(self, priority): - """This is implemented to override the abstract method and fix a pylint error""" + def get_priority(self, priority: StepPriority): + """ + This is implemented to override the abstract method and fix a pylint error. + + Args: + priority: Float or + [`StepPriority`](https://maestrowf.readthedocs.io/en/latest/Maestro/reference_guide/api_reference/abstracts/enums/index.html#maestrowf.abstracts.enums.StepPriority) + enum representing priorty. + """ - def get_header(self, step): + def get_header(self, step: StudyStep) -> str: """ Generate the header present at the top of LSF execution scripts. - :param step: A StudyStep instance. - :returns: A string of the header based on internal batch parameters and - the parameter step. + Args: + step: A Maestro StudyStep instance that contains parameters relevant to the execution. + + Returns: + A string of the header based on internal batch parameters and the parameter step. """ return f"#!{self._exec}" - def get_parallelize_command(self, procs, nodes=None, **kwargs): + def get_parallelize_command(self, procs: int, nodes: int = None, **kwargs: Dict) -> str: """ - Generate the LSF parallelization segement of the command line. - :param procs: Number of processors to allocate to the parallel call. - :param nodes: Number of nodes to allocate to the parallel call - (default = 1). - :returns: A string of the parallelize command configured using nodes - and procs. + Generate the LSF parallelization segment of the command line. + + This method constructs a command line segment for parallel execution in LSF. + It allows specifying the number of processors and nodes to be allocated for the parallel call, + along with additional command flags through keyword arguments. + + Args: + procs: Number of processors to allocate to the parallel call. + nodes: Number of nodes to allocate to the parallel call. Defaults to 1. + **kwargs: Additional command flags that may be supported by the LSF command. + + Returns: + A string representing the parallelization command configured using nodes and procs. """ if not nodes: nodes = 1 @@ -180,18 +195,21 @@ def get_parallelize_command(self, procs, nodes=None, **kwargs): return " ".join(args) - def write_script(self, ws_path, step): + def write_script(self, ws_path: str, step: StudyStep) -> Tuple[bool, str, str]: """ - This will overwrite the write_script in method from Maestro's base ScriptAdapter + This will overwrite the `write_script` method from Maestro's base ScriptAdapter class but will eventually call it. This is necessary for the VLAUNCHER to work. - :param `ws_path`: the path to the workspace where we'll write the scripts - :param `step`: the Maestro StudyStep object containing info for our step - :returns: a tuple containing: - - a boolean representing whether this step is to be scheduled or not - - Merlin can ignore this - - a path to the script for the cmd - - a path to the script for the restart cmd + Args: + ws_path: The path to the workspace where the scripts will be written. + step: The Maestro StudyStep object containing information for the step. + + Returns: + A tuple containing:\n + - bool: A boolean indicating whether this step is to be scheduled or not. + (Merlin can ignore this value.) + - str: The path to the script for the command. + - str: The path to the script for the restart command. """ setup_vlaunch(step.run, "lsf", False) @@ -200,23 +218,42 @@ class but will eventually call it. This is necessary for the VLAUNCHER to work. class MerlinSlurmScriptAdapter(SlurmScriptAdapter): """ - A SchedulerScriptAdapter class for slurm blocking parallel launches, - the SlurmScriptAdapter uses non-blocking submits. + A `SchedulerScriptAdapter` class for SLURM blocking parallel launches. + + This class extends the `SlurmScriptAdapter` to provide support for blocking parallel + launches in SLURM. Unlike the base class, which uses non-blocking submits, this adapter + is designed for workflows that execute SLURM parallel jobs in a Celery worker. + + Attributes: + key (str): A unique identifier for the adapter, set to "merlin-slurm". + _cmd_flags (Dict[str, str]): A dictionary containing command flags for SLURM. + _unsupported (Set[str]): A set of command flags that are not supported by this adapter. + + Methods: + get_header: Generates the header for SLURM execution scripts. + get_parallelize_command: Generates the SLURM parallelization segment of the command line. + get_priority: Overrides the abstract method to fix a pylint error. + time_format: Converts a timestring to HH:MM:SS format. + write_script: Overwrites the write_script method from the base class to ensure VLAUNCHER compatibility. """ key: str = "merlin-slurm" - def __init__(self, **kwargs): + def __init__(self, **kwargs: Dict): """ - Initialize an instance of the MerinSlurmScriptAdapter. - The MerlinSlurmScriptAdapter is the adapter that is used for workflows that + Initialize an instance of the `MerinSlurmScriptAdapter`. + + The `MerlinSlurmScriptAdapter` is the adapter that is used for workflows that will execute SLURM parallel jobs in a celery worker. The only configurable aspect to this adapter is the shell that scripts are executed in. - :param **kwargs: A dictionary with default settings for the adapter. + Args: + **kwargs: A dictionary with default settings for the adapter. """ super().__init__(**kwargs) + self._cmd_flags: Dict[str, str] + self._cmd_flags["slurm"] = "" self._cmd_flags["walltime"] = "-t" @@ -236,33 +273,63 @@ def __init__(self, **kwargs): ] self._unsupported: Set[str] = set(list(self._unsupported) + new_unsupported) - def get_priority(self, priority): - """This is implemented to override the abstract method and fix a pylint error""" + def get_priority(self, priority: StepPriority): + """ + This is implemented to override the abstract method and fix a pylint error. + + Args: + priority: Float or + [`StepPriority`](https://maestrowf.readthedocs.io/en/latest/Maestro/reference_guide/api_reference/abstracts/enums/index.html#maestrowf.abstracts.enums.StepPriority) + enum representing priorty. + """ - def get_header(self, step): + def get_header(self, step: StudyStep) -> str: """ Generate the header present at the top of Slurm execution scripts. - :param step: A StudyStep instance. - :returns: A string of the header based on internal batch parameters and - the parameter step. + Args: + step: A Maestro StudyStep instance that contains parameters relevant to the execution. + + Returns: + A string of the header based on internal batch parameters and the parameter step. """ return f"#!{self._exec}" - def time_format(self, val): + def time_format(self, val: Union[str, int]) -> str: """ - Convert the timestring to HH:MM:SS + Convert the input timestring or integer to HH:MM:SS format. + + This method utilizes the [`convert_timestring`][utils.convert_timestring] + function to convert a given timestring or integer (representing seconds) + into a formatted string in the 'hours:minutes:seconds' (HH:MM:SS) format. + + Args: + val: A timestring in the format '[days]:[hours]:[minutes]:seconds' or + an integer representing time in seconds. + + Returns: + A string representation of the input time formatted as 'HH:MM:SS'. """ return convert_timestring(val, format_method="HMS") - def get_parallelize_command(self, procs, nodes=None, **kwargs): + def get_parallelize_command(self, procs: int, nodes: int = None, **kwargs: Dict) -> str: """ - Generate the SLURM parallelization segement of the command line. - :param procs: Number of processors to allocate to the parallel call. - :param nodes: Number of nodes to allocate to the parallel call - (default = 1). - :returns: A string of the parallelize command configured using nodes - and procs. + Generate the SLURM parallelization segment of the command line. + + This method constructs the command line segment required for parallel execution + in SLURM, including the number of processors and nodes to allocate. It also + incorporates any additional supported command flags provided in `kwargs`. + + Args: + procs: The number of processors to allocate for the parallel call. + nodes: The number of nodes to allocate for the parallel call (default is 1). + **kwargs: Additional command flags to customize the SLURM command. + Supported flags include 'walltime' and others defined in the + `_cmd_flags` attribute, excluding those in the `_unsupported` set. + + Returns: + A string representing the SLURM parallelization command, formatted with the + specified number of processors, nodes, and any additional flags. """ args = [ # SLURM srun command @@ -297,18 +364,21 @@ def get_parallelize_command(self, procs, nodes=None, **kwargs): return " ".join(args) - def write_script(self, ws_path, step): + def write_script(self, ws_path: str, step: StudyStep) -> Tuple[bool, str, str]: """ - This will overwrite the write_script in method from Maestro's base ScriptAdapter + This will overwrite the `write_script` method from Maestro's base ScriptAdapter class but will eventually call it. This is necessary for the VLAUNCHER to work. - :param `ws_path`: the path to the workspace where we'll write the scripts - :param `step`: the Maestro StudyStep object containing info for our step - :returns: a tuple containing: - - a boolean representing whether this step is to be scheduled or not - - Merlin can ignore this - - a path to the script for the cmd - - a path to the script for the restart cmd + Args: + ws_path: The path to the workspace where the scripts will be written. + step: The Maestro `StudyStep` object containing information for the step. + + Returns: + A tuple containing:\n + - bool: A boolean indicating whether this step is to be scheduled or not. + (Merlin can ignore this value.) + - str: The path to the script for the command. + - str: The path to the script for the restart command. """ setup_vlaunch(step.run, "slurm", False) @@ -317,26 +387,41 @@ class but will eventually call it. This is necessary for the VLAUNCHER to work. class MerlinFluxScriptAdapter(MerlinSlurmScriptAdapter): """ - A SchedulerScriptAdapter class for flux blocking parallel launches, - the FluxScriptAdapter uses non-blocking submits. + A `SchedulerScriptAdapter` class for flux blocking parallel launches. + + The `MerlinFluxScriptAdapter` is designed for workflows that execute flux parallel jobs + in a Celery worker. It utilizes non-blocking submits and allows for configuration of the + shell in which scripts are executed. + + Attributes: + key (str): A unique identifier for the adapter, set to "merlin-flux". + _cmd_flags (Dict[str, str]): A dictionary containing command-line flags for the flux command. + _unsupported (Set[str]): A set of command flags that are not supported by this adapter. + + Methods: + get_priority: Retrieves the priority of the step. + time_format: Converts a time format to flux standard designation. + write_script: Writes the script for the specified step and returns relevant paths. """ - key = "merlin-flux" + key: str = "merlin-flux" - def __init__(self, **kwargs): + def __init__(self, **kwargs: Dict): """ - Initialize an instance of the MerinFluxScriptAdapter. - The MerlinFluxScriptAdapter is the adapter that is used for workflows that + Initialize an instance of the `MerinFluxScriptAdapter`. + + The `MerlinFluxScriptAdapter` is the adapter that is used for workflows that will execute flux parallel jobs in a celery worker. The only configurable aspect to this adapter is the shell that scripts are executed in. - :param **kwargs: A dictionary with default settings for the adapter. + Args: + **kwargs: A dictionary with default settings for the adapter. """ # The flux_command should always be overriden by the study object's flux_command property flux_command = kwargs.pop("flux_command", "flux run") super().__init__(**kwargs) - self._cmd_flags = { + self._cmd_flags: Dict[str, str] = { "cmd": flux_command, "ntasks": "-n", "nodes": "-N", @@ -366,29 +451,51 @@ def __init__(self, **kwargs): "lsf", "slurm", ] - self._unsupported = set(new_unsupported) # noqa + self._unsupported: Set[str] = set(new_unsupported) # noqa + + def get_priority(self, priority: StepPriority): + """ + This is implemented to override the abstract method and fix a pylint error. - def get_priority(self, priority): - """This is implemented to override the abstract method and fix a pylint error""" + Args: + priority: Float or + [`StepPriority`](https://maestrowf.readthedocs.io/en/latest/Maestro/reference_guide/api_reference/abstracts/enums/index.html#maestrowf.abstracts.enums.StepPriority) + enum representing priorty. + """ - def time_format(self, val): + def time_format(self, val: Union[str, int]) -> str: """ - Convert a time format to flux standard designation. + Convert a time format to Flux Standard Duration (FSD). + + This method takes a time value and converts it into a format that is compatible + with Flux's standard time representation. The conversion is performed using the + [`convert_timestring`][utils.convert_timestring] function with the specified format + method. + + Args: + val: The time value to be converted. This can be a string representing a time + duration or an integer representing a time value. + + Returns: + The time formatted according to Flux Standard Duration (FSD). """ return convert_timestring(val, format_method="FSD") - def write_script(self, ws_path, step): + def write_script(self, ws_path: str, step: StudyStep) -> Tuple[bool, str, str]: """ - This will overwrite the write_script in method from Maestro's base ScriptAdapter + This will overwrite the `write_script` method from Maestro's base ScriptAdapter class but will eventually call it. This is necessary for the VLAUNCHER to work. - :param `ws_path`: the path to the workspace where we'll write the scripts - :param `step`: the Maestro StudyStep object containing info for our step - :returns: a tuple containing: - - a boolean representing whether this step is to be scheduled or not - - Merlin can ignore this - - a path to the script for the cmd - - a path to the script for the restart cmd + Args: + ws_path: The path to the workspace where the scripts will be written. + step: The Maestro `StudyStep` object containing information for the step. + + Returns: + A tuple containing:\n + - bool: A boolean indicating whether this step is to be scheduled or not. + (Merlin can ignore this value.) + - str: The path to the script for the command. + - str: The path to the script for the restart command. """ setup_vlaunch(step.run, "flux", True) @@ -397,23 +504,40 @@ class but will eventually call it. This is necessary for the VLAUNCHER to work. class MerlinScriptAdapter(LocalScriptAdapter): """ - A ScriptAdapter class for interfacing for execution in Merlin + A `ScriptAdapter` class for interfacing with execution in Merlin. + + This class serves as an adapter for executing scripts in a Celery worker + environment. It allows for configuration of the execution environment and + manages the execution of scripts with appropriate logging and error handling. + + Attributes: + batch_adapter (ScriptAdapter): An instance of a batch adapter used for executing scripts + based on the specified batch type. + batch_type (str): The type of batch processing to be used, derived from + the provided keyword arguments. + key (str): A unique identifier for the adapter, set to "merlin-local". + + Methods: + submit: Executes a workflow step locally. + write_script: Writes a script using the batch adapter. """ - key = "merlin-local" + key: str = "merlin-local" - def __init__(self, **kwargs): + def __init__(self, **kwargs: Dict): """ - Initialize an instance of the MerinScriptAdapter. - The MerlinScriptAdapter is the adapter that is used for workflows that + Initialize an instance of the `MerinScriptAdapter`. + + The `MerlinScriptAdapter` is the adapter that is used for workflows that will execute in a celery worker. The only configurable aspect to this adapter is the shell that scripts are executed in. - :param **kwargs: A dictionary with default settings for the adapter. + Args: + **kwargs: A dictionary with default settings for the adapter. """ super().__init__(**kwargs) - self.batch_type = "merlin-" + kwargs.get("batch_type", "local") + self.batch_type: str = "merlin-" + kwargs.get("batch_type", "local") if "host" not in kwargs: kwargs["host"] = "None" @@ -423,33 +547,51 @@ def __init__(self, **kwargs): kwargs["queue"] = "None" # Using super prevents recursion. - self.batch_adapter = super() + self.batch_adapter: ScriptAdapter = super() if self.batch_type != "merlin-local": self.batch_adapter = MerlinScriptAdapterFactory.get_adapter(self.batch_type)(**kwargs) - def write_script(self, *args, **kwargs): + def write_script(self, *args, **kwargs) -> Tuple[bool, str, str]: """ - TODO + Generate a script for execution using the batch adapter. + + This method delegates the script writing process to the associated + batch adapter and returns the generated script along with a restart + script if applicable. + + Returns: + A tuple containing:\n + - bool: A boolean indicating whether this step is to be scheduled or not. + (Merlin can ignore this value.) + - str: The path to the script for the command. + - str: The path to the script for the restart command. """ _, script, restart_script = self.batch_adapter.write_script(*args, **kwargs) return True, script, restart_script # Pylint complains that there's too many arguments but it's fine in this case - def submit(self, step, path, cwd, job_map=None, env=None): # pylint: disable=R0913 - """ - Execute the step locally. - If cwd is specified, the submit method will operate outside of the path - specified by the 'cwd' parameter. - If env is specified, the submit method will set the environment - variables for submission to the specified values. The 'env' parameter - should be a dictionary of environment variables. - - :param step: An instance of a StudyStep. - :param path: Path to the script to be executed. - :param cwd: Path to the current working directory. - :param job_map: A map of workflow step names to their job identifiers. - :param env: A dict containing a modified environment for execution. - :returns: The return code of the command and processID of the command. + def submit( + self, step: StudyStep, path: str, cwd: str, job_map: Dict = None, env: Dict = None + ) -> SubmissionRecord: # pylint: disable=R0913 + """ + Execute a workflow step locally. + + This method runs a specified script in the local environment, allowing for + customization of the working directory and environment variables. It handles + the execution of the script and logs the results, including any errors or + specific return codes. + + Args: + step: An instance of the StudyStep that contains information about the + workflow step being executed. + path: The file path to the script that is to be executed. + cwd: The current working directory from which the script will be executed. + job_map: A mapping of workflow step names to their job identifiers. + env: A dictionary containing environment variables to be set for the execution. + + Returns: + An object containing the return code of the command, the process ID of the + command, and any additional information about the execution. """ LOG.debug("cwd = %s", cwd) LOG.debug("Script to execute: %s", path) @@ -488,21 +630,28 @@ def submit(self, step, path, cwd, job_map=None, env=None): # pylint: disable=R0 # TODO is there currently ever a scenario where join output is True? We should look into this # Pylint is complaining there's too many local variables and args but it makes this function cleaner so ignore - def _execute_subprocess(self, output_name, script_path, cwd, env=None, join_output=False): # pylint: disable=R0913,R0914 - """ - Execute the subprocess script locally. - If cwd is specified, the submit method will operate outside of the path - specified by the 'cwd' parameter. - If env is specified, the submit method will set the environment - variables for submission to the specified values. The 'env' parameter - should be a dictionary of environment variables. - - :param output_name: Output name for stdout and stderr (output_name.out). If None, don't write. - :param script_path: Path to the script to be executed. - :param cwd: Path to the current working directory. - :param env: A dict containing a modified environment for execution. - :param join_output: If True, append stderr to stdout - :returns: The return code of the submission command and job identifier (SubmissionRecord). + def _execute_subprocess( + self, output_name: str, script_path: str, cwd: str, env: Dict = None, join_output: bool = False + ) -> SubmissionRecord: # pylint: disable=R0913,R0914 + """ + Execute a subprocess script locally and manage output. + + This method runs a specified script in a subprocess, capturing its output + and error streams. It allows for customization of the working directory, + environment variables, and output handling. The output can be saved to + files, and error messages can be appended to the standard output if desired. + + Args: + output_name: The base name for the output files (stdout and stderr). + If None, no output files will be created. + script_path: The file path to the script that is to be executed. + cwd: The current working directory from which the script will be executed. + env: A dictionary containing environment variables to be set for the execution. + join_output: If True, appends stderr to stdout in the output file. + + Returns: + An object containing the return code of the command the process ID of the + command, and any additional information about the execution. """ script_bn = os.path.basename(script_path) new_output_name = os.path.splitext(script_bn)[0] @@ -537,9 +686,23 @@ def _execute_subprocess(self, output_name, script_path, cwd, env=None, join_outp class MerlinScriptAdapterFactory: - """This class routes to the correct ScriptAdapter""" + """ + This class routes to the correct `ScriptAdapter`. + + The `MerlinScriptAdapterFactory` is responsible for providing the appropriate + `ScriptAdapter` based on the specified adapter ID. It maintains a mapping of + available adapters and offers methods to retrieve them. + + Attributes: + factories: A dictionary mapping adapter IDs (str) to their corresponding + `ScriptAdapter` classes. + + Methods: + get_adapter: Returns the appropriate `ScriptAdapter` class for the given adapter ID. + get_valid_adapters: Returns a list of valid adapter IDs that can be used with this factory. + """ - factories = { + factories: Dict[str, ScriptAdapter] = { "merlin-flux": MerlinFluxScriptAdapter, "merlin-lsf": MerlinLSFScriptAdapter, "merlin-lsf-srun": MerlinSlurmScriptAdapter, @@ -548,8 +711,23 @@ class MerlinScriptAdapterFactory: } @classmethod - def get_adapter(cls, adapter_id): - """Returns the appropriate ScriptAdapter to use""" + def get_adapter(cls, adapter_id: str) -> ScriptAdapter: + """ + Returns the appropriate `ScriptAdapter` to use. + + This method retrieves the `ScriptAdapter` class associated with the given + adapter ID. If the adapter ID is not found in the factory's mapping, + a ValueError is raised. + + Args: + adapter_id: The ID of the desired `ScriptAdapter`. + + Returns: + The corresponding `ScriptAdapter` class. + + Raises: + ValueError: If the specified adapter_id is not found in the factories. + """ if adapter_id.lower() not in cls.factories: msg = f"""Adapter '{str(adapter_id)}' not found. Specify an adapter that exists or implement a new one mapping to the '{str(adapter_id)}'""" @@ -559,6 +737,15 @@ def get_adapter(cls, adapter_id): return cls.factories[adapter_id] @classmethod - def get_valid_adapters(cls): - """Returns the valid ScriptAdapters""" + def get_valid_adapters(cls) -> List[str]: + """ + Returns the valid ScriptAdapters. + + This method provides a list of all valid adapter IDs that can be used + with this factory. The IDs are derived from the keys of the factories + dictionary. + + Returns: + A list of valid adapter IDs. + """ return cls.factories.keys() diff --git a/merlin/study/status.py b/merlin/study/status.py index b785ee140..37b7b7baf 100644 --- a/merlin/study/status.py +++ b/merlin/study/status.py @@ -1,33 +1,10 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2 -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### -"""This module handles all the functionality of getting the statuses of studies""" +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +"""This module handles all the functionality of getting the statuses of studies.""" import json import logging import os @@ -37,7 +14,7 @@ from datetime import datetime from glob import glob from traceback import print_exception -from typing import Any, Dict, List, Optional, Tuple, Union +from typing import Any, Dict, List, Set, Tuple, Union import numpy as np from filelock import FileLock, Timeout @@ -47,6 +24,7 @@ from merlin.common.dumper import dump_handler from merlin.display import ANSI_COLORS, display_status_summary, display_status_task_by_task from merlin.spec.expansion import get_spec_with_expansion +from merlin.spec.specification import MerlinSpec from merlin.study.status_constants import ( ALL_VALID_FILTERS, CELERY_KEYS, @@ -72,15 +50,58 @@ class Status: """ - This class handles everything to do with status besides displaying it. - Display functionality is handled in display.py. + Handles the management and retrieval of status information for studies. + + This class is responsible for loading specifications, tracking the status of steps, + calculating runtime statistics, and formatting status information for output in + various formats (JSON, CSV). It interacts with the file system to read status files + and provides methods to display and dump status information. + + Attributes: + args (Namespace): Command-line arguments provided by the user. + full_step_name_map (Dict[str, Set[str]]): A mapping of overall step names to full step names. + num_requested_statuses (int): Counts the number of task statuses in the `requested_statuses` + dictionary. + requested_statuses (Dict): A dictionary storing the statuses that the user wants to view. + run_time_info (Dict[str, Dict]): A dictionary storing runtime statistics for each step. + spec (spec.specification.MerlinSpec): A [`MerlinSpec`][spec.specification.MerlinSpec] + object loaded from the workspace or spec file. + step_tracker (Dict[str, List[str]]): A dictionary tracking started and unstarted steps. + tasks_per_step (Dict[str, int]): A mapping of tasks per step for accurate totals. + workspace (str): The path to the workspace containing study data. + + Methods: + display: Displays a high-level summary of the status. + dump: Dumps the status information to a specified file. + format_csv_dump: Prepares the dictionary of statuses for CSV output. + format_json_dump: Prepares the dictionary of statuses for JSON output. + format_status_for_csv: Reformats statuses into a dictionary suitable for CSV output. + get_runtime_avg_std_dev: Calculates and stores the average and standard deviation of + runtimes for a step. + get_step_statuses: Reads and returns the statuses for a given step. + get_steps_to_display: Generates a list of steps to display the status for. + load_requested_statuses: Populates the `requested_statuses` dictionary with statuses + from the study. """ def __init__(self, args: Namespace, spec_display: bool, file_or_ws: str): + """ + Initializes the `Status` object, which manages and retrieves status information for studies. + + Args: + args: Command-line arguments provided by the user, including filters and options + for displaying or dumping status information. + spec_display: A flag indicating whether the status should be loaded from a specification + file (`True`) or from a workspace (`False`). + file_or_ws: The path to the specification file or workspace, depending on the value of + `spec_display`. + """ # Save the args to this class instance and check if the steps filter was given - self.args = args + self.args: Namespace = args # Load in the workspace path and spec object + self.workspace: str + self.spec: MerlinSpec if spec_display: self.workspace, self.spec = self._load_from_spec(file_or_ws) else: @@ -91,26 +112,31 @@ def __init__(self, args: Namespace, spec_display: bool, file_or_ws: str): self._verify_filter_args() # Create a step tracker that will tell us which steps have started/not started - self.step_tracker = self.get_steps_to_display() + self.step_tracker: Dict[str, List[str]] = self.get_steps_to_display() # Create a tasks per step mapping in order to give accurate totals for each step - self.tasks_per_step = self.spec.get_tasks_per_step() + self.tasks_per_step: Dict[str, int] = self.spec.get_tasks_per_step() # This attribute will store a map between the overall step name and the full step names # that are created with parameters (e.g. step name is hello and uses a "GREET: hello" parameter # so the real step name is hello_GREET.hello) - self.full_step_name_map = {} + self.full_step_name_map: Dict[str, Set[str]] = {} # Variable to store run time information for each step - self.run_time_info = {} + self.run_time_info: Dict[str, Dict] = {} # Variable to store the statuses that the user wants - self.requested_statuses = {} + self.requested_statuses: Dict = {} self.load_requested_statuses() def _print_requested_statuses(self): """ - Helper method to print out the requested statuses dict. + Print the requested statuses stored in the `requested_statuses` dictionary. + + This helper method iterates through the `requested_statuses` attribute, which contains + information about the statuses of various steps. It prints the step names along with + their corresponding status information. Non-workspace keys are printed directly, while + workspace-related keys are further detailed by their status keys and values. """ print("self.requested_statuses:") for step_name, overall_step_info in self.requested_statuses.items(): @@ -125,16 +151,38 @@ def _print_requested_statuses(self): def _verify_filter_args(self): """ - This is an abstract method since we'll need to verify filter args for DetailedStatus - but not for Status. + Verify the filter arguments for the status retrieval. + + This is an abstract method intended to be implemented in subclasses, such as + [`DetailedStatus`][study.status.DetailedStatus]. The method will ensure that + the filter arguments provided for retrieving statuses are valid and meet the + necessary criteria. The implementation details will depend on the specific + requirements of the subclass. """ def _get_latest_study(self, studies: List[str]) -> str: """ - Given a list of studies, get the latest one. + Retrieve the latest study from a list of studies. + + This method examines a list of study identifiers and determines which one is the latest + based on the timestamp embedded in the study names. It assumes that the newest study is + represented by the last entry in the list but verifies this assumption by comparing the + timestamps of all studies. + + The method extracts the timestamp from the last 15 characters of each study identifier, + converts it to a datetime object, and compares it to find the most recent study. - :param `studies`: A list of studies to sort through - :returns: The latest study in the list provided + Args: + studies: A list of study identifiers to evaluate. + + Returns: + The identifier of the latest study. + + Example: + ```python + >>> self._get_latest_study(["study_20231101-174102", "study_20231101-182044", "study_20231101-163327"]) + 'study_20231101-182044' + ``` """ # We can assume the newest study is the last one to be added to the list of potential studies newest_study = studies[-1] @@ -155,12 +203,22 @@ def _obtain_study(self, study_output_dir: str, num_studies: int, potential_studi """ Grab the study that the user wants to view the status of based on a list of potential studies provided. - :param `study_output_dir`: A string representing the output path of a study; equivalent to $(OUTPUT_PATH) - :param `num_studies`: The number of potential studies we found - :param `potential_studies`: The list of potential studies we found; - Each entry is of the form (index, potential_study_name) - :returns: A directory path to the study that the user wants - to view the status of ("study_output_dir/selected_potential_study") + This method checks the number of potential studies found and either selects the latest study + automatically or prompts the user to choose from the available options. It constructs the + directory path to the selected study. + + Args: + study_output_dir: A string representing the output path of a study; equivalent to $(OUTPUT_PATH). + num_studies: The number of potential studies found. + potential_studies: A list of potential studies found, where each entry is of the form (index, + potential_study_name). + + Returns: + A directory path to the study that the user wants to view the status of, formatted as + "study_output_dir/selected_potential_study". + + Raises: + ValueError: If no potential studies are found or if the user input is invalid. """ study_to_check = f"{study_output_dir}/" if num_studies == 0: @@ -197,14 +255,25 @@ def _obtain_study(self, study_output_dir: str, num_studies: int, potential_studi return study_to_check - def _load_from_spec(self, filepath: str) -> Tuple[str, "MerlinSpec"]: # noqa: F821 pylint: disable=R0914 + def _load_from_spec(self, filepath: str) -> Tuple[str, MerlinSpec]: # pylint: disable=R0914 """ - Get the desired workspace from the user and load up it's yaml spec - for further processing. + Get the desired workspace from the user and load its YAML spec for further processing. + + This method verifies the output path based on user input or the spec file and builds a list + of potential study output directories. It then calls another method to obtain the study to + check the status for and loads the corresponding spec. + + Args: + filepath: The filepath to a spec provided by the user. - :param `filepath`: The filepath to a spec given by the user - :returns: The workspace of the study we'll check the status for and a MerlinSpec - object loaded in from the workspace's merlin_info subdirectory. + Returns: + A tuple containing the workspace of the study to check the status for and a + [`MerlinSpec`][spec.specification.MerlinSpec] object loaded from the workspace's + merlin_info subdirectory. + + Raises: + ValueError: If the specified output directory does not contain a merlin_info subdirectory, + or if multiple or no expanded spec options are found in the directory. """ # If the user provided a new output path to look in, use that if self.args.output_path is not None: @@ -263,11 +332,17 @@ def _load_from_spec(self, filepath: str) -> Tuple[str, "MerlinSpec"]: # noqa: F return study_to_check, actual_spec - def _load_from_workspace(self) -> "MerlinSpec": # noqa: F821 + def _load_from_workspace(self) -> MerlinSpec: """ - Create a MerlinSpec object based on the spec file in the workspace. + Create a [`MerlinSpec`][spec.specification.MerlinSpec] object based on the expanded spec file + in the workspace. + + Returns: + spec.specification.MerlinSpec: A [`MerlinSpec`][spec.specification.MerlinSpec] object loaded + from the workspace provided by the user. - :returns: A MerlinSpec object loaded from the workspace provided by the user + Raises: + ValueError: If multiple or no expanded spec options are found in the workspace's merlin_info directory. """ # Grab the spec file from the directory provided expanded_spec_options = glob(f"{self.workspace}/merlin_info/*.expanded.yaml") @@ -285,11 +360,19 @@ def _load_from_workspace(self) -> "MerlinSpec": # noqa: F821 def _create_step_tracker(self, steps_to_check: List[str]) -> Dict[str, List[str]]: """ - Creates a dictionary of started and unstarted steps that we - will display the status for. + Creates a dictionary of started and unstarted steps to display their status. + + This method checks the workspace for steps that have been started and compares them + against a provided list of steps to determine which steps are started and which are + unstarted. It returns a dictionary categorizing the steps accordingly. + + Args: + steps_to_check: A list of step names to check the status of. - :param `steps_to_check`: A list of steps to view the status of - :returns: A dictionary mapping of started and unstarted steps. Values are lists of step names. + Returns: + A dictionary with two keys:\n + - "started_steps": A list of steps that have been started. + - "unstarted_steps": A list of steps that have not been started. """ step_tracker = {"started_steps": [], "unstarted_steps": []} started_steps = next(os.walk(self.workspace))[1] @@ -310,10 +393,22 @@ def _create_step_tracker(self, steps_to_check: List[str]) -> Dict[str, List[str] def get_steps_to_display(self) -> Dict[str, List[str]]: """ - Generates a list of steps to display the status for based on information - provided to the merlin status command by the user. - - :returns: A dictionary of started and unstarted steps for us to display the status of + Generates a dictionary of steps to display their status based on user input + provided to the merlin status command. + + This method retrieves the names of existing steps from the study specification + and creates a step tracker to categorize them into started and unstarted steps. + + Returns: + A dictionary with two keys:\n + - `started_steps`: A list of steps that have been started. + - `unstarted_steps`: A list of steps that have not been started. + + Example: + ```python + >>> self.get_steps_to_display() + {"started_steps": ["step1"], "unstarted_steps": ["step2", "step3"]} + ``` """ existing_steps = self.spec.get_study_step_names() @@ -328,10 +423,13 @@ def get_steps_to_display(self) -> Dict[str, List[str]]: return step_tracker @property - def num_requested_statuses(self): + def num_requested_statuses(self) -> int: """ - Count the number of task statuses in a the requested_statuses dict. - We need to ignore non workspace keys when we count. + Counts the number of task statuses in the requested_statuses dictionary, + excluding non-workspace keys. + + Returns: + The count of requested task statuses that are not non-workspace keys. """ num_statuses = 0 for overall_step_info in self.requested_statuses.values(): @@ -341,12 +439,20 @@ def num_requested_statuses(self): def get_step_statuses(self, step_workspace: str, started_step_name: str) -> Dict[str, List[str]]: """ - Given a step workspace and the name of the step, read in all the statuses - for the step and return them in a dict. + Reads the statuses for a specified step from the given step workspace. + + This method traverses the specified step workspace directory to locate + `MERLIN_STATUS.json` files, reads their contents, and aggregates the statuses + into a dictionary. It also tracks the full names of the steps and counts + the number of statuses read. - :param step_workspace: The path to the step we're going to read statuses from - :param started_step_name: The name of the step that we're gathering statuses for - :returns: A dict of statuses for the given step + Args: + step_workspace: The path to the step directory from which to read statuses. + started_step_name: The name of the step for which statuses are being gathered. + + Returns: + A dictionary containing the statuses for the specified step, where each key is a full + step name and the value is a list of status information. """ step_statuses = {} num_statuses_read = 0 @@ -390,7 +496,13 @@ def get_step_statuses(self, step_workspace: str, started_step_name: str) -> Dict def load_requested_statuses(self): """ - Populate the requested_statuses dict with the statuses from the study. + Populates the `requested_statuses` dictionary with statuses from the study. + + This method iterates through the started steps in the step tracker, + retrieves their statuses using the + [`get_step_statuses`][study.status.Status.get_step_statuses] method, and merges + these statuses into the `requested_statuses` dictionary. It also calculates + the average and standard deviation of the run times for each step. """ LOG.info(f"Reading task statuses from {self.workspace}") @@ -406,14 +518,20 @@ def load_requested_statuses(self): # Count how many statuses in total that we just read in LOG.info(f"Read in {self.num_requested_statuses} statuses total.") - def get_runtime_avg_std_dev(self, step_statuses: Dict, step_name: str) -> Dict: + def get_runtime_avg_std_dev(self, step_statuses: Dict, step_name: str): """ - Calculate the mean and standard deviation for the runtime of each step. - Add this to the state information once calculated. - - :param `step_statuses`: A dict of step status information that we'll parse for run times - :param `step_name`: The name of the step - :returns: An updated dict of step status info with run time avg and std dev + Calculates the average and standard deviation of the runtime for a specified step. + + This method parses the provided step status information to extract runtime values, + computes the mean and standard deviation of these runtimes, and updates the state + information with the calculated values. The runtimes are expected to be in a specific + format (e.g., "1h30m15s") and are converted to seconds for the calculations. + + Args: + step_statuses: A dictionary containing step status information, where each + entry includes runtime data to be parsed. + step_name: The name of the step for which the average and standard deviation + of the runtime are being calculated. """ # Initialize a list to track all existing runtimes run_times_in_seconds = [] @@ -454,32 +572,53 @@ def get_runtime_avg_std_dev(self, step_statuses: Dict, step_name: str) -> Dict: self.run_time_info[step_name]["run_time_std_dev"] = f"±{pretty_format_hms(convert_timestring(run_time_std_dev))}" LOG.debug(f"Run time avg and std dev for step '{step_name}' calculated.") - def display(self, test_mode: Optional[bool] = False) -> Dict: + def display(self, test_mode: bool = False) -> Dict: """ - Displays the high level summary of the status. + Displays a high-level summary of the status. - :param `test_mode`: If true, run this in testing mode and don't print any output - :returns: A dict that will be empty if test_mode is False. Otherwise, the dict will - contain the status info that would be displayed. + This method provides an overview of the current status of the workflow. If + `test_mode` is enabled, it will not print any output but will return the + status information in a dictionary. + + Args: + test_mode: If true, run this in testing mode and don't print any output. + + Returns: + An empty dictionary if `test_mode` is False; otherwise, a dictionary containing + the status information that would be displayed. """ return display_status_summary(self, NON_WORKSPACE_KEYS, test_mode=test_mode) def format_json_dump(self, date: datetime) -> Dict: """ - Build the dict of statuses to dump to the json file. + Builds a dictionary of statuses to dump to a JSON file. + + This method prepares the status information for serialization by adding a timestamp + to the existing status data. - :param `date`: A timestamp for us to mark when this status occurred - :returns: A dictionary that's ready to dump to a json outfile + Args: + date: A timestamp marking when this status occurred. + + Returns: + A dictionary ready to be dumped to a JSON file, containing the timestamp + and the requested statuses. """ # Statuses are already in json format so we'll just add a timestamp for the dump here return {date: self.requested_statuses} def format_csv_dump(self, date: datetime) -> Dict: """ - Add the timestamp to the statuses to write. + Adds a timestamp to the statuses for CSV output. + + This method reformats the status information into a structure suitable for CSV + output, including a timestamp entry as the first column. - :param `date`: A timestamp for us to mark when this status occurred - :returns: A dict equivalent of formatted statuses with a timestamp entry at the start of the dict. + Args: + date: A timestamp marking when this status occurred. + + Returns: + A dictionary equivalent of formatted statuses with a timestamp entry + at the start of the dictionary. """ # Reformat the statuses to a new dict where the keys are the column labels and rows are the values LOG.debug("Formatting statuses for csv dump...") @@ -494,7 +633,11 @@ def format_csv_dump(self, date: datetime) -> Dict: def dump(self): """ - Dump the status information to a file. + Dumps the status information to a file. + + This method handles the creation of a timestamp and determines the appropriate + file format (CSV or JSON) for dumping the status information. It then calls + the appropriate formatting method and writes the data to the specified file. """ # Get a timestamp for this dump date = datetime.now().strftime("%Y-%m-%d %H:%M:%S") @@ -512,10 +655,16 @@ def dump(self): def format_status_for_csv(self) -> Dict: """ - Reformat our statuses to csv format so they can use Maestro's status renderer layouts. + Reformats statuses for CSV output to comply with + [Maestro's status renderer layouts](https://maestrowf.readthedocs.io/en/latest/Maestro/reference_guide/api_reference/index.html). - :returns: A formatted dictionary where each key is a column and the values are the rows - of information to display for that column. + This method transforms the status information into a dictionary format where each + key represents a column label and the corresponding values are the rows of information + to display for that column. + + Returns: + A formatted dictionary where each key is a column and the values are the + rows of information to display for that column. """ reformatted_statuses = { "step_name": [], @@ -591,10 +740,42 @@ def format_status_for_csv(self) -> Dict: class DetailedStatus(Status): """ This class handles obtaining and filtering requested statuses from the user. - This class shares similar methodology to the Status class it inherits from. + It inherits from the [`Status`][study.status.Status] class and provides + additional functionality for filtering and displaying task statuses based on + user-defined criteria. + + Attributes: + args (Namespace): A namespace containing user-defined arguments for filtering. + num_requested_statuses (int): The number of task statuses in the `requested_statuses` dictionary. + requested_statuses (Dict): A dictionary holding the statuses requested by the user. + spec (spec.specification.MerlinSpec): A [`MerlinSpec`][spec.specification.MerlinSpec] + object loaded from the workspace or spec file. + steps_filter_provided (bool): Indicates if a specific steps filter was provided. + + Methods: + apply_filters: Applies user-defined filters to the requested statuses. + apply_max_tasks_limit: Limits the number of tasks displayed based on the user-defined maximum. + display: Displays a task-by-task view of the status based on user filters. + filter_via_prompts: Interacts with the user to manage task display filters. + get_steps_to_display: Generates a list of steps to display the status for. + get_user_filters: Prompts the user for filters to apply to the statuses. + get_user_max_tasks: Prompts the user for a maximum task limit to display. + load_requested_statuses: Populates the requested statuses dictionary based on user-defined filters. """ def __init__(self, args: Namespace, spec_display: bool, file_or_ws: str): + """ + Initializes the `DetailedStatus` object, extending the functionality of the `Status` class + to include filtering and detailed task-by-task status handling. + + Args: + args: Command-line arguments provided by the user, including options + for filtering, displaying, or dumping detailed task statuses. + spec_display: A flag indicating whether the status should be loaded from a + specification file (`True`) or from a workspace (`False`). + file_or_ws: The path to the specification file or workspace, depending on the + value of `spec_display`. + """ args_copy = Namespace(**vars(args)) super().__init__(args, spec_display, file_or_ws) @@ -603,23 +784,30 @@ def __init__(self, args: Namespace, spec_display: bool, file_or_ws: str): os.environ["MANPAGER"] = "less -r" # Check if the steps filter was given - self.steps_filter_provided = "all" not in args_copy.steps + self.steps_filter_provided: bool = "all" not in args_copy.steps def _verify_filters( self, filters_to_check: List[str], valid_options: Union[List, Tuple], suppress_warnings: bool, - warning_msg: Optional[str] = "", + warning_msg: str = "", ): """ - Check each filter in a list of filters provided by the user against a list of valid options. - If the filter is invalid, remove it from the list of filters. - - :param `filters_to_check`: A list of filters provided by the user - :param `valid_options`: A list of valid options for this particular filter - :param `suppress_warnings`: If True, don't log warnings. Otherwise, log them - :param `warning_msg`: An optional warning message to attach to output + Verify and validate a list of user-provided filters against a set of valid options. + + This method checks each filter in the `filters_to_check` list to determine if it is present + in the `valid_options`. If a filter is found to be invalid (i.e., not in `valid_options`), + it is removed from the `filters_to_check` list. Depending on the value of `suppress_warnings`, + a warning message may be logged for each invalid filter. + + Args: + filters_to_check: A list of filters provided by the user that need to be validated. + valid_options: A list or tuple of valid options against which the filters will be checked. + suppress_warnings: A boolean flag indicating whether to suppress warning messages. + If True, no warnings will be logged for invalid filters. + warning_msg: An optional string that provides additional context for the warning message + logged when an invalid filter is detected. Default is an empty string. """ for filter_arg in filters_to_check[:]: if filter_arg not in valid_options: @@ -627,11 +815,17 @@ def _verify_filters( LOG.warning(f"The filter '{filter_arg}' is invalid. {warning_msg}") filters_to_check.remove(filter_arg) - def _verify_filter_args(self, suppress_warnings: Optional[bool] = False): + def _verify_filter_args(self, suppress_warnings: bool = False): """ - Verify that our filters are all valid and able to be used. + Verify the validity of filter arguments used in the current context. + + This method checks various filter arguments, including steps, max_tasks, task_status, + return_code, task_queues, and workers, to ensure they are valid and can be used. + Invalid filters are removed from their respective lists, and warnings may be logged + based on the `suppress_warnings` flag. - :param `suppress_warnings`: If True, don't log warnings. Otherwise, log them. + Args: + suppress_warnings: If True, suppress logging of warnings for invalid filters. """ # Ensure the steps are valid if "all" not in self.args.steps: @@ -717,8 +911,11 @@ def _verify_filter_args(self, suppress_warnings: Optional[bool] = False): def _process_task_queue(self): """ - Modifies the list of steps to display status for based on - the list of task queues provided by the user. + Modify the list of steps to display status for based on the provided task queues. + + This method processes the task queues specified by the user, removing any duplicates + and checking for their validity. It updates the list of steps to include those associated + with the valid task queues. If a provided task queue does not exist, a warning is logged. """ from merlin.config.configfile import CONFIG # pylint: disable=C0415 @@ -744,11 +941,16 @@ def _process_task_queue(self): def get_steps_to_display(self) -> Dict[str, List[str]]: """ - Generates a list of steps to display the status for based on information - provided to the merlin detailed-status command by the user. This function - will handle the --steps and --task-queues filter options. + Generate a dictionary of steps to display the status for based on user-provided filters. - :returns: A dictionary of started and unstarted steps for us to display the status of + This method processes the `--steps` and `--task-queues` options from the `merlin + detailed-status` command. It determines which steps should be included in the status + display based on the existing steps in the study and the specified filters. + + Returns: + A dictionary containing two lists:\n + - `started`: A list of steps that have been started. + - `unstarted`: A list of steps that have not yet been started. """ existing_steps = self.spec.get_study_step_names() @@ -781,9 +983,16 @@ def get_steps_to_display(self) -> Dict[str, List[str]]: def _remove_steps_without_statuses(self): """ - After applying filters, there's a chance that certain steps will still exist - in self.requested_statuses but won't have any tasks to view the status of so - we'll remove those here. + Remove steps from the requested statuses that do not have any associated tasks. + + This method iterates through the `requested_statuses` dictionary and checks each step + for associated sub-steps. If a step does not have any valid sub-step workspaces (i.e., + it has no tasks to view the status of), it is removed from the `requested_statuses`. + + Note: + After applying filters, there's a chance that certain steps will still exist + in self.requested_statuses but won't have any tasks to view the status of. That's + why this method is necessary. """ result = deepcopy(self.requested_statuses) @@ -797,11 +1006,16 @@ def _remove_steps_without_statuses(self): def _search_for_filter(self, filter_to_apply: List[str], entry_to_search: Union[List[str], str]) -> bool: """ - Search an entry to see if our filter(s) apply to this entry. If they do, return True. Otherwise, False. + Search an entry to see if the specified filters apply to it. - :param filter_to_apply: A list of filters to search for - :param entry_to_search: A list or string of entries to search for our filters in - :returns: True if a filter was found in the entry. False otherwise. + This method checks if any of the provided filters match the given entry or entries. + + Args: + filter_to_apply: A list of filters to search for. + entry_to_search: A list or string of entries to search for the filters in. + + Returns: + True if a filter was found in the entry; False otherwise. """ if not isinstance(entry_to_search, list): entry_to_search = [entry_to_search] @@ -814,10 +1028,12 @@ def _search_for_filter(self, filter_to_apply: List[str], entry_to_search: Union[ def apply_filters(self): """ - Apply any filters given by the --workers, --return-code, and/or --task-status arguments. - This function will also apply the --max-tasks limit if it was set by a user. We apply this - limit here so it can be done in-place; if we called apply_max_tasks_limit instead, this - would become a two-pass algorithm and can be really slow with lots of statuses. + Apply filters based on the provided command-line arguments for workers, return code, + and task status, as well as enforce a maximum task limit if specified. + + This method processes the `requested_statuses` to filter out entries that do not match + the specified criteria. It ensures that the filtering is done in-place to optimize performance + and avoid a two-pass algorithm, which can be inefficient with a large number of statuses. """ if self.args.max_tasks is not None: # Make sure the max_tasks variable is set to a reasonable number and store that value @@ -897,8 +1113,13 @@ def apply_filters(self): def apply_max_tasks_limit(self): """ - Given a number representing the maximum amount of tasks to display, filter the dict of statuses - so that there are at most a max_tasks amount of tasks. + Filter the dictionary of statuses to ensure that the number of displayed tasks does not exceed + the specified maximum limit. + + This method checks the current value of `max_tasks` and adjusts it if it exceeds the number + of available statuses. It then iterates through the `requested_statuses`, removing excess + entries to comply with the `max_tasks` limit. The method also merges the allowed task statuses + into a new dictionary and updates the `requested_statuses` accordingly. """ # Make sure the max_tasks variable is set to a reasonable number and store that value if self.args.max_tasks > self.num_requested_statuses: @@ -959,15 +1180,24 @@ def load_requested_statuses(self): def get_user_filters(self) -> bool: """ - Get a filter on the statuses to display from the user. Possible options - for filtering: - - A str MAX_TASKS -> will ask the user for another input that's equivalent to the --max-tasks flag - - A list of statuses -> equivalent to the --task-status flag - - A list of return codes -> equivalent to the --return-code flag - - A list of workers -> equivalent to the --workers flag - - An exit keyword to leave the filter prompt without filtering - - :returns: True if we need to exit without filtering. False otherwise. + Prompt the user to specify filters for the statuses to display. The user can choose from + several filtering options, including setting a maximum number of tasks, filtering by status, + return code, or worker, or exiting the filter prompt without applying any filters. + + The method displays available filter options and their descriptions, then collects and + validates the user's input. If the user provides valid filters, they are stored in the + corresponding attributes. If the user opts to exit, the method returns True; otherwise, + it returns False. + + Possible filtering options include:\n + - A string "MAX_TASKS" to request a limit on the number of tasks. + - A list of statuses to filter by, corresponding to the `--task-status` flag. + - A list of return codes to filter by, corresponding to the `--return-code` flag. + - A list of workers to filter by, corresponding to the `--workers` flag. + - An exit keyword to leave the filter prompt without applying any filters. + + Returns: + True if the user chooses to exit without filtering; False otherwise. """ valid_workers = tuple(self.spec.get_worker_names()) @@ -1050,7 +1280,17 @@ def get_user_filters(self) -> bool: def get_user_max_tasks(self): """ - Get a limit for the amount of tasks to display from the user. + Prompt the user to specify a maximum limit for the number of tasks to display. + + The method repeatedly requests input from the user until a valid integer greater than 0 + is provided. Once a valid input is received, it sets the `max_tasks` attribute in the + `args` object to the specified limit. + + This method ensures that the user input is validated and handles any exceptions + related to invalid input types or values. + + Raises: + ValueError: If the input is not a valid integer greater than 0. """ invalid_input = True @@ -1069,8 +1309,16 @@ def get_user_max_tasks(self): def filter_via_prompts(self): """ - Interact with the user to manage how many/which tasks are displayed. This helps to - prevent us from overloading the terminal by displaying a bazillion tasks at once. + Interact with the user to determine how many and which tasks should be displayed, + preventing terminal overload by limiting the output to a manageable number of tasks. + + This method prompts the user for filtering options, including task statuses, return codes, + and worker specifications. It also handles the case where the user opts to exit without + applying any filters. If filters are provided, it applies them accordingly. + + Warning: + The method includes specific handling for the "RESTART" and "RETRY" return codes, + which are currently not implemented, and issues warnings if these filters are selected. """ # Get the filters from the user exit_without_filtering = self.get_user_filters() @@ -1095,11 +1343,18 @@ def filter_via_prompts(self): elif self.args.max_tasks is not None: self.apply_max_task_limit() - def display(self, test_mode: Optional[bool] = False): + def display(self, test_mode: bool = False): """ - Displays a task-by-task view of the status based on user filter(s). + Displays a task-by-task view of the statuses based on the user-defined filters. - :param `test_mode`: If true, run this in testing mode and don't print any output + This method checks for any requested statuses and, if found, invokes the + `display_status_task_by_task` function to present the tasks accordingly. + If no statuses are available to display, it logs a warning message. + + Args: + test_mode: If set to True, the method runs in testing mode, suppressing + any output to the terminal. This is useful for unit testing or debugging + without cluttering the output. """ # Check that there's statuses found and display them if self.requested_statuses: @@ -1111,26 +1366,31 @@ def display(self, test_mode: Optional[bool] = False): # Pylint complains that args is unused but we can ignore that def status_conflict_handler(*args, **kwargs) -> Any: # pylint: disable=W0613 """ - The conflict handler function to apply to any status entries that have conflicting - values while merging two status files together. - - kwargs should include: - - dict_a_val: The conflicting value from the dictionary that we're merging into - - dict_b_val: The conflicting value from the dictionary that we're pulling from - - key: The key into each dictionary that has a conflict - - path: The path down the dictionary tree that `dict_deep_merge` is currently at - - When we're reading in status files, we're merging all of the statuses into one dictionary. - This function defines the merge rules in case there is a merge conflict. We ignore the list - and dictionary entries since `dict_deep_merge` from `utils.py` handles these scenarios already. - - There are currently 4 rules: - - string-concatenate: take the two conflicting values and concatenate them in a string - - use-dict_b-and-log-debug: use the value from dict_b and log a debug message - - use-longest-time: use the longest time between the two conflicting values - - use-max: use the larger integer between the two conflicting values - - :returns: The value to merge into dict_a at `key` + Handles conflicts that arise when merging two status files by applying specific merge rules + to conflicting values. + + This function is designed to be used during the merging process of status entries, where + conflicting values may exist. It defines how to resolve these conflicts based on predefined + rules, ensuring that the merged dictionary maintains integrity and clarity. + + The merge rules currently implemented are:\n + - **string-concatenate**: Concatenates the two conflicting string values. + - **use-dict_b-and-log-debug**: Uses the value from dict_b and logs a debug message indicating + the conflict. + - **use-longest-time**: Chooses the longest time value between the two conflicting entries, + converting them to a timedelta for comparison. + - **use-max**: Selects the maximum integer value from the two conflicting entries. + + If a key does not have a defined merge rule, a warning is logged, and the function returns None. + + The function expects the following keyword arguments:\n + - `dict_a_val`: The conflicting value from the dictionary that we are merging into (dict_a). + - `dict_b_val`: The conflicting value from the dictionary that we are merging from (dict_b). + - `key`: The key in each dictionary that has a conflict. + - `path`: The current path in the dictionary tree during the merge process. + + Returns: + The resolved value to merge into dict_a at the specified key. """ # Grab the arguments passed into this function dict_a_val = kwargs.get("dict_a_val", None) @@ -1203,12 +1463,25 @@ def read_status( """ Locks the status file for reading and returns its contents. - :param status_filepath: The path to the status file that we'll read from. - :param lock_file: The path to the lock file that we'll use to create a FileLock. - :param display_fnf_message: If True, display the file not found warning. Otherwise don't. - :param raise_errors: A boolean indicating whether to ignore errors or raise them. - :param timeout: An integer representing how long to hold a lock for before timing out. - :returns: A dict of the contents in the status file + This function attempts to read the contents of a status file while ensuring that the file is + locked to prevent race conditions. It handles various exceptions that may occur during the + reading process, including file not found errors and JSON decoding errors. + + Args: + status_filepath: The path to the status file that will be read. + lock_file: The path to the lock file used to create a FileLock. + display_fnf_message: If True, displays a warning message if the file is not found. + raise_errors: If True, raises exceptions when errors occur. + timeout: The maximum time (in seconds) to hold the lock before timing out. + + Returns: + A dictionary containing the contents of the status file. + + Raises: + Timeout: If the lock acquisition times out. + FileNotFoundError: If the status file does not exist and `raise_errors` is True. + json.decoder.JSONDecodeError: If the status file is empty or contains invalid JSON and `raise_errors` is True. + Exception: Any other exceptions that occur during the reading process if `raise_errors` is True. """ statuses_read = {} @@ -1249,13 +1522,20 @@ def read_status( def write_status(status_to_write: Dict, status_filepath: str, lock_file: str, timeout: int = 10): """ - Locks the status file for writing. We're not catching any errors here since we likely want to - know if something went wrong in this process. + Locks the status file for writing and writes the provided status to the file. + + This function ensures that the status file is locked during the write operation to prevent + race conditions. It does not catch errors during the writing process, as it is important to + be aware of any issues that may arise. + + Args: + status_to_write: The status data to write to the status file. + status_filepath: The path to the status file where the status will be written. + lock_file: The path to the lock file used to create a FileLock for the write operation. + timeout: The maximum time (in seconds) to hold the lock before timing out. - :param status_to_write: The status to write to the status file - :param status_filepath: The path to the status file that we'll write the status to - :param lock_file: The path to the lock file we'll use for this status write - :param timeout: A timeout value for the lock so it's always released eventually + Raises: + Exception: Any exceptions that occur during the writing process will be logged, but not caught. """ # Pylint complains that we're instantiating an abstract class but this is correct usage try: diff --git a/merlin/study/status_constants.py b/merlin/study/status_constants.py index 78ce331a6..072702e39 100644 --- a/merlin/study/status_constants.py +++ b/merlin/study/status_constants.py @@ -1,36 +1,13 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2 -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ This file contains all of the constants used for the status command. -Separating this from status.py and status_renderers.py helps with circular -import issues. +Separating this from [`status.py`][study.status] and [`status_renderers.py`][study.status_renderers] +helps with circular import issues. """ VALID_STATUS_FILTERS = ("INITIALIZED", "RUNNING", "FINISHED", "FAILED", "CANCELLED", "DRY_RUN", "UNKNOWN") diff --git a/merlin/study/status_renderers.py b/merlin/study/status_renderers.py index 2100c2bda..49d99a0f8 100644 --- a/merlin/study/status_renderers.py +++ b/merlin/study/status_renderers.py @@ -1,35 +1,12 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2 -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """This module handles creating a formatted task-by-task status display""" import logging -from typing import Dict, List, Optional, Union +from typing import Dict, List, Union from maestrowf import BaseStatusRenderer, FlatStatusRenderer, StatusRendererFactory from rich import box @@ -45,14 +22,20 @@ LOG = logging.getLogger(__name__) -def format_label(label_to_format: str, delimiter: Optional[str] = "_") -> str: +def format_label(label_to_format: str, delimiter: str = "_") -> str: """ - Take a string of the format 'word1_word2_...' and format it so it's prettier. - This would turn the string above to 'Word1 Word2 ...'. + Format a string by replacing a specified delimiter with spaces and capitalizing each word. + + This function takes a string that uses a specific delimiter to separate words and returns a + more readable version of that string, where the words are separated by spaces and each word + is capitalized. + + Args: + label_to_format: The string to format. + delimiter: The character that separates words in `label_to_format`. - :param `label_to_format`: The string we want to format - :param `delimiter`: The character separating words in `label_to_format` - :returns: A formatted string based on `label_to_format` + Returns: + A formatted string where the delimiter is replaced with spaces and each word is capitalized. """ return label_to_format.replace(delimiter, " ").title() @@ -60,20 +43,41 @@ def format_label(label_to_format: str, delimiter: Optional[str] = "_") -> str: class MerlinDefaultRenderer(BaseStatusRenderer): """ This class handles the default status formatting for task-by-task display. - It will separate the display on a step-by-step basis. - - Similar to Maestro's 'narrow' status display. + It will separate the display on a step-by-step basis, similar to Maestro's 'narrow' status display. + + Attributes: + disable_theme (bool): Flag to disable theming for the display. + disable_pager (bool): Flag to disable pager functionality for the display. + _theme_dict (Dict[str, str]): A dictionary containing the theme settings for various status types. + _status_table (Table): A Table object that contains the formatted status information. + + Methods: + create_param_table: Creates the parameter section of the display. + create_step_table: Creates each step entry in the display. + create_task_details_table: Creates the task details section of the display. + layout: Sets up the overall layout of the display. + render: Performs the actual printing of the status table with optional theme customization. """ - def __init__(self, *args, **kwargs): + def __init__(self, *args: List, **kwargs: Dict): + """ + Initializes the `MerlinDefaultRenderer` instance, which handles the default status formatting + for task-by-task display, with optional theming and pager functionality. + + Args: + *args: Positional arguments passed to the superclass (`BaseStatusRenderer`). + **kwargs: Keyword arguments used to configure the renderer. Supported keys include:\n + - disable_theme (bool, optional): If `True`, disables theming for the display. Defaults to `False`. + - disable_pager (bool, optional): If `True`, disables pager functionality for the display. Defaults to `False`. + """ super().__init__(*args, **kwargs) - self.disable_theme = kwargs.pop("disable_theme", False) - self.disable_pager = kwargs.pop("disable_pager", False) + self.disable_theme: bool = kwargs.pop("disable_theme", False) + self.disable_pager: bool = kwargs.pop("disable_pager", False) # Setup default theme # TODO modify this theme to add more colors - self._theme_dict = { + self._theme_dict: Dict[str, str] = { "INITIALIZED": "blue", "RUNNING": "blue", "DRY_RUN": "green", @@ -92,14 +96,23 @@ def __init__(self, *args, **kwargs): } # Setup the status table that will contain our formatted status - self._status_table = Table.grid(padding=0) + self._status_table: Table = Table.grid(padding=0) def create_param_table(self, parameters: Dict[str, Dict[str, str]]) -> Columns: """ - Create the parameter section of the display + Create the parameter section of the display. + + This method generates a formatted table for the parameters associated with each command type. + Each command type (e.g., "cmd", "restart") will have its own sub-table displaying the tokens + and their corresponding values. + + Args: + parameters: A dictionary where each key is a command type (e.g., "cmd", "restart") + and each value is another dictionary containing token-value pairs. + Example format: `{"cmd": {"TOKEN1": "value1"}, "restart": {"TOKEN2": "value1"}}`. - :param `parameters`: A dict of the form {"cmd": {"TOKEN1": "value1"}, "restart": {"TOKEN2": "value1"}} - :returns: A rich Columns object with the parameter info formatted appropriately + Returns: + A rich Columns object containing the formatted parameter tables, arranged side-by-side. """ param_table = [] # Loop through cmd and restart entries @@ -135,18 +148,26 @@ def create_step_table( self, step_name: str, parameters: Dict[str, Dict[str, str]], - task_queue: Optional[str] = None, - workers: Optional[str] = None, + task_queue: str = None, + workers: str = None, ) -> Table: """ - Create each step entry in the display - - :param `step_name`: The name of the step that we're setting the layout for - :param `parameters`: The parameters dict for this step - :param `task_queue`: The name of the task queue associated with this step if one was provided - :param `workers`: The name of the worker(s) that ran this step if one was provided - :returns: A rich Table object with info for one sub step (here a 'sub step' is referencing a step - with multiple parameters; each parameter set will have it's own entry in the output) + Create each step entry in the display. + + This method constructs a formatted table entry for a specific step in the process, including + relevant details such as the step name, associated task queue, worker(s), and any parameters + related to the step. Each parameter set will be displayed in a sub-table format. + + Args: + step_name: The name of the step for which the layout is being created. + parameters: A dictionary of parameters associated with the step, where each key is a + parameter type and each value is a dictionary of token-value pairs. + task_queue: The name of the task queue associated with this step, if provided. + workers: The name(s) of the worker(s) that executed this step, if provided. + + Returns: + A rich Table object containing the formatted information for the specified step, + including its parameters and any associated task queue or worker details. """ # Initialize the table that will have our step entry information step_table = Table(box=box.SIMPLE_HEAVY, show_header=False) @@ -172,10 +193,20 @@ def create_step_table( def create_task_details_table(self, task_statuses: Dict) -> Table: """ - Create the task details section of the display + Create the task details section of the display. + + This method constructs a formatted table that displays detailed information about various tasks, + including their statuses, return codes, elapsed times, run times, restarts, and associated workers. + Each task is represented as a row in the table, with specific styling applied based on the task's status. + + Args: + task_statuses: A dictionary containing task statuses, where each key represents a step workspace + and each value is another dictionary with details such as status, return code, elapsed time, + run time, restarts, and workers. - :param `task_statuses`: A dict of task statuses to format into our layout - :returns: A rich Table with the formatted task info for a sub step + Returns: + A rich Table object containing the formatted task details, structured for easy readability + and visual distinction based on task status. """ # Initialize the task details table task_details = Table(title="Task Details") @@ -222,15 +253,23 @@ def create_task_details_table(self, task_statuses: Dict) -> Table: return task_details - def layout( - self, status_data, study_title: Optional[str] = None, status_time: Optional[str] = None - ): # pylint: disable=W0237 + def layout(self, status_data: Dict, study_title: str = None, status_time: str = None): # pylint: disable=W0237 """ - Setup the overall layout of the display + Setup the overall layout of the display. - :param `status_data`: A dict of status data to display - :param `study_title`: A title for the study to display at the top of the output - :param `status_time`: A timestamp to add to the title + This method configures the main display layout for the status data, including setting up + the title with optional study information and timestamp. It organizes the status data into + a structured table format, displaying each step's details along with associated task information. + + Args: + status_data: A dictionary containing status data to be displayed, where each key + represents a step and its associated information. + study_title: A title for the study to be displayed at the top of the output. + status_time: A timestamp to be included in the title, indicating when the status + data was captured. + + Raises: + ValueError: If `status_data` is not a dictionary or is empty. """ if isinstance(status_data, dict) and status_data: self._status_data = status_data @@ -280,11 +319,17 @@ def layout( # Add this step to the full status table self._status_table.add_row(step_table, end_section=True) - def render(self, theme: Optional[Dict[str, str]] = None): + def render(self, theme: Dict[str, str] = None): """ - Do the actual printing + Do the actual printing of the status table. + + This method is responsible for rendering the status table to the console, applying any specified + theme settings for visual customization. It handles the enabling or disabling of themes and + manages the output display, either using a pager for long outputs or printing directly to the console. - :param `theme`: A dict of theme settings (see self._theme_dict for the appropriate layout) + Args: + theme: A dictionary of theme settings that define the appearance of the output. The keys and + values should correspond to the layout defined in `self._theme_dict`. """ # Apply any theme customization if theme: @@ -314,24 +359,37 @@ class MerlinFlatRenderer(FlatStatusRenderer): """ This class handles the flat status formatting for task-by-task display. It will not separate the display on a step-by-step basis and instead group - all statuses together in a single table. + all statuses together in a single table, similar to Maestro's 'flat' status display. - Similar to Maestro's 'flat' status display. + Attributes: + disable_theme (bool): A flag indicating whether to disable theme customization for the output. + disable_pager (bool): A flag indicating whether to disable the use of a pager for long outputs. + + Methods: + layout: Sets up the layout of the display, formatting the status data and study title. + render: Renders the status table to the console, applying any specified theme settings and + managing the output display. """ def __init__(self, *args, **kwargs): super().__init__(args, kwargs) - self.disable_theme = kwargs.pop("disable_theme", False) - self.disable_pager = kwargs.pop("disable_pager", False) + self.disable_theme: bool = kwargs.pop("disable_theme", False) + self.disable_pager: bool = kwargs.pop("disable_pager", False) - def layout( - self, status_data: Dict[str, List[Union[str, int]]], study_title: Optional[str] = None - ): # pylint: disable=W0221 + def layout(self, status_data: Dict[str, List[Union[str, int]]], study_title: str = None): # pylint: disable=W0221 """ - Setup the layout of the display - - :param `status_data`: A dict of status information that we'll display - :param `study_title`: The title of the study to display at the top of the output + Set up the layout of the display for the status information. + + This method processes the provided status data by removing unnecessary parameters, + capitalizing the column labels, and preparing the data for display. It also allows + for an optional study title to be displayed at the top of the output. + + Args: + status_data: A dictionary containing status information to be displayed. The + keys represent the status categories, and the values are lists of + corresponding status values. + study_title: The title of the study to display at the top of the output. + If provided, it will be included in the layout. """ if "cmd_parameters" in status_data: del status_data["cmd_parameters"] @@ -344,11 +402,22 @@ def layout( super().layout(status_data, study_title=study_title) - def render(self, theme: Optional[Dict[str, str]] = None): + def render(self, theme: Dict[str, str] = None): """ - Do the actual printing - - :param `theme`: A dict of theme settings (see self._theme_dict for the appropriate layout) + Render the status table to the console. + + This method is responsible for displaying the formatted status information + in the console. It applies any specified theme settings to customize the + appearance of the output. If the theme is disabled, it sets all theme + attributes to 'none'. The method also handles the output display, either + printing directly to the console or using a pager for long outputs based + on the `disable_pager` attribute. + + Args: + theme (Dict[str, str], optional): A dictionary of theme settings that + customize the appearance of the output. The keys represent the + theme attributes, and the values are the corresponding settings. + If not provided, the default theme settings will be used. """ # Apply any theme customization if theme: @@ -377,6 +446,21 @@ def render(self, theme: Optional[Dict[str, str]] = None): class MerlinStatusRendererFactory(StatusRendererFactory): """ This class keeps track of all available status layouts for Merlin. + + The `MerlinStatusRendererFactory` is responsible for managing different + status layout renderers used in the Merlin application. It provides a + method to retrieve the appropriate renderer based on the specified layout + type and user preferences regarding theme and pager usage. + + Attributes: + _layouts (Dict[str, BaseStatusRenderer]): A dictionary mapping layout names to their corresponding renderer + classes. Currently includes "table" for + [`MerlinFlatRenderer`][study.status_renderers.MerlinFlatRenderer] and + "default" for [`MerlinDefaultRenderer`][study.status_renderers.MerlinDefaultRenderer]. + + Methods: + get_renderer: Retrieves an instance of the specified layout renderer, applying + user preferences for theme and pager settings. """ # TODO: when maestro releases the pager changes: @@ -387,21 +471,29 @@ class MerlinStatusRendererFactory(StatusRendererFactory): # - remove render method in MerlinDefaultRenderer # - this will also be in BaseStatusRenderer in Maestro def __init__(self): # pylint: disable=W0231 - self._layouts = { + self._layouts: Dict[str, BaseStatusRenderer] = { "table": MerlinFlatRenderer, "default": MerlinDefaultRenderer, } - def get_renderer(self, layout: str, disable_theme: bool, disable_pager: bool): # pylint: disable=W0221 - """Get handle for specific layout renderer to instantiate + def get_renderer( + self, layout: str, disable_theme: bool, disable_pager: bool + ) -> BaseStatusRenderer: # pylint: disable=W0221 + """ + Get handle for specific layout renderer to instantiate. + + Args: + layout: A string denoting the name of the layout renderer to use. + disable_theme: True if the user wants to disable themes when displaying + status; False otherwise. + disable_pager: True if the user wants to disable the pager when displaying + status; False otherwise. - :param `layout`: A string denoting the name of the layout renderer to use - :param `disable_theme`: True if the user wants to disable themes when displaying status. - False otherwise. - :param `disable_pager`: True if the user wants to disable the pager when displaying status. - False otherwise. + Returns: + The status renderer class to use for displaying the output. - :returns: The status renderer class to use for displaying the output + Raises: + ValueError: If the specified layout is not found in the available layouts. """ renderer = self._layouts.get(layout) diff --git a/merlin/study/step.py b/merlin/study/step.py index 8e0356b68..ac13ac061 100644 --- a/merlin/study/step.py +++ b/merlin/study/step.py @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """This module represents all of the logic that goes into a step""" import logging @@ -34,14 +11,16 @@ import re from contextlib import suppress from copy import deepcopy -from typing import Dict, Optional, Tuple +from typing import Dict, List, Tuple from celery import current_task from maestrowf.abstracts.enums import State +from maestrowf.abstracts.interfaces.scriptadapter import ScriptAdapter from maestrowf.datastructures.core.executiongraph import _StepRecord from maestrowf.datastructures.core.study import StudyStep +from maestrowf.interfaces.script import SubmissionRecord -from merlin.common.abstracts.enums import ReturnCode +from merlin.common.enums import ReturnCode from merlin.study.script_adapter import MerlinScriptAdapter from merlin.study.status import read_status, write_status from merlin.utils import needs_merlin_expansion @@ -50,15 +29,34 @@ LOG = logging.getLogger(__name__) -def get_current_worker(): - """Get the worker on the current running task from celery""" +def get_current_worker() -> str: + """ + Get the worker on the current running task from Celery. + + This function retrieves the name of the worker that is currently + executing the task. It extracts the worker's name from the task's + request hostname. + + Returns: + The name of the current worker. + """ worker = re.search(r"@.+\.", current_task.request.hostname).group() worker = worker[1 : len(worker) - 1] return worker -def get_current_queue(): - """Get the queue on the current running task from celery""" +def get_current_queue() -> str: + """ + Get the queue on the current running task from Celery. + + This function retrieves the name of the queue that the current + task is associated with. It extracts the routing key from the + task's delivery information and removes the queue tag defined + in the configuration. + + Returns: + The name of the current queue. + """ from merlin.config.configfile import CONFIG # pylint: disable=C0415 queue = current_task.request.delivery_info["routing_key"] @@ -68,24 +66,55 @@ def get_current_queue(): class MerlinStepRecord(_StepRecord): """ - This class is a wrapper for the Maestro _StepRecord to remove + This class is a wrapper for the Maestro `_StepRecord` to remove a re-submit message and handle status updates. + + Attributes: + condensed_workspace (str): A condensed version of the workspace path. + elapsed_time (str): The total elapsed time for the step execution. + jobid (List[int]): A list of job identifiers assigned by the scheduler. + maestro_step (StudyStep): The StudyStep object associated with this step. + merlin_step (Step): The Step object associated with this step. + restart_limit (int): Upper limit on the number of restart attempts. + restart_script (str): Script to resume record execution (if applicable). + run_time (str): The run time for the step execution. + status (State): The current status of the step. + to_be_scheduled (bool): Indicates if the record needs scheduling. + workspace (Variable): The output workspace for this step, represented as a Variable. + + Methods: + mark_end: Marks the end of the step with the given state. + mark_restart: Increments the restart count for the step. + mark_running: Marks the step as running and updates the status file. + setup_workspace: Initializes the workspace and status file for the step. """ def __init__(self, workspace: str, maestro_step: StudyStep, merlin_step: "Step", **kwargs): """ - :param `workspace`: The output workspace for this step - :param `maestro_step`: The StudyStep object associated with this step - :param `merlin_step`: The Step object associated with this step + Initializes the `MerlinStepRecord` class which helps track the status of a step. + + Args: + workspace: The output workspace for this step. + maestro_step: The + [StudyStep](https://maestrowf.readthedocs.io/en/latest/Maestro/reference_guide/api_reference/datastructures/core/index.html#maestrowf.datastructures.core.StudyStep) + object associated with this step. + merlin_step: The [Step][study.step.Step] object associated with this step. """ _StepRecord.__init__(self, workspace, maestro_step, status=State.INITIALIZED, **kwargs) - self.merlin_step = merlin_step + self.merlin_step: Step = merlin_step @property def condensed_workspace(self) -> str: """ - Put together a smaller version of the workspace path to display. - :returns: A condensed workspace name + Generate a condensed version of the workspace path for display purposes. + + This property constructs a shorter representation of the workspace path by extracting relevant + components based on the study name and a timestamp pattern. If a match is found using a regular + expression, the workspace path is split to isolate the condensed portion. If no match is found, + a fallback method is used to manually create a condensed path based on the step name. + + Returns: + A string representing the condensed workspace path, which is easier to read and display. """ timestamp_regex = r"\d{8}-\d{6}/" match = re.search(rf"{self.merlin_step.study_name}_{timestamp_regex}", self.workspace.value) @@ -102,15 +131,21 @@ def condensed_workspace(self) -> str: LOG.debug(f"Condense workspace '{condensed_workspace}'") return condensed_workspace - def _execute(self, adapter: "ScriptAdapter", script: str) -> Tuple["SubmissionRecord", int]: # noqa: F821 + def _execute(self, adapter: ScriptAdapter, script: str) -> Tuple[SubmissionRecord, int]: """ - Overwrites _StepRecord's _execute method from Maestro since self.to_be_scheduled is - always true here. Also, if we didn't overwrite this we wouldn't be able to call - self.mark_running() for status updates. + Executes the script using the provided adapter, overriding the default behavior to ensure + that the step is marked as running and to facilitate job submission. - :param `adapter`: The script adapter to submit jobs to - :param `script`: The script to send to the script adapter - :returns: A tuple of a return code and the jobid from the execution of `script` + This method overrides the `_execute` method from the base class `_StepRecord` in Maestro. + It ensures that `self.to_be_scheduled` is always true, allowing for the invocation of + `self.mark_running()` to update the status of the step. + + Args: + adapter: The script adapter used to submit jobs. + script: The script to be submitted to the script adapter. + + Returns: + A tuple containing the return code and the job identifier from the execution of the script. """ self.mark_running() @@ -129,11 +164,17 @@ def mark_running(self): def mark_end(self, state: ReturnCode, max_retries: bool = False): """ - Mark the end time of the record with associated termination state - and update the status file. + Marks the end time of the record with the associated termination state + and updates the status file. + + This method logs the action of marking the end of the step, maps the provided + termination state to a corresponding Maestro state and result, and updates + the status file accordingly. If the maximum number of retries has been reached + for a soft failure, it appends a message to the result. - :param `state`: A merlin ReturnCode object representing the end state of a task - :param `max_retries`: A bool representing whether we hit the max number of retries or not + Args: + state: A ReturnCode object representing the end state of the task. + max_retries: A flag indicating whether the maximum number of retries has been reached. """ LOG.debug(f"Marking end for {self.name}") @@ -189,7 +230,7 @@ def mark_end(self, state: ReturnCode, max_retries: bool = False): self._update_status_file(result=step_result) def mark_restart(self): - """Increment the number of restarts we've had for this step and update the status file""" + """Increment the number of restarts we've had for this step and update the status file.""" LOG.debug(f"Marking restart for {self.name}") if self.restart_limit == 0 or self._num_restarts < self.restart_limit: self._num_restarts += 1 @@ -203,16 +244,22 @@ def setup_workspace(self): def _update_status_file( self, - result: Optional[str] = None, - task_server: Optional[str] = "celery", + result: str = None, + task_server: str = "celery", ): """ - Puts together a dictionary full of status info and creates a signature - for the update_status celery task. This signature is ran here as well. + Constructs a dictionary containing status information and creates a signature + for the update_status Celery task. This signature is executed within the method. - :param `result`: Optional parameter only applied when we've finished running - this step. String representation of a ReturnCode value. - :param `task_server`: Optional parameter to define the task server we're using. + This method checks if a status file already exists; if it does, it updates the + existing file with the current status information. If not, it initializes a new + status dictionary. The method also includes optional parameters for the result + of the task and the task server being used. + + Args: + result: An optional string representation of a ReturnCode value, applied + when the step has finished running. + task_server: An optional parameter to specify the task server being used. """ # This dict is used for converting an enum value to a string for readability @@ -294,43 +341,91 @@ def _update_status_file( class Step: """ This class provides an abstraction for an execution step, which can be - executed by calling execute. + executed by calling the [`execute`][study.step.Step.execute] method. + + Attributes: + max_retries (int): Returns the maximum number of retries for this step. + mstep (_StepRecord): The Maestro StepRecord object associated with this step. + parameter_info (dict): A dictionary containing information about parameters in the study. + params (Dict): A dictionary containing command parameters for the step, including 'cmd' and 'restart_cmd'. + restart (bool): Property to get or set the restart status of the step. + retry_delay (int): Returns the retry delay for the step (default is 1). + study_name (str): The name of the study this step belongs to. + + Methods: + check_if_expansion_needed: Checks if command expansion is needed based on specified labels. + clone_changing_workspace_and_cmd: Produces a deep copy of the current step, with optional command + and workspace modifications. + establish_params: Pulls parameters from the step parameter map if applicable. + execute: Executes the step using the provided adapter configuration. + get_cmd: Retrieves the run command text body. + get_restart_cmd: Retrieves the restart command text body, or None if not available. + get_task_queue: Retrieves the task queue for the step. + get_task_queue_from_dict: Static method to get the task queue from a step dictionary. + get_workspace: Retrieves the workspace where this step is to be executed. + name: Retrieves the name of the step. + name_no_params: Gets the original name of the step without parameters or sample labels. """ - def __init__(self, maestro_step_record, study_name, parameter_info): + def __init__(self, maestro_step_record: _StepRecord, study_name: str, parameter_info: Dict): """ - :param maestro_step_record: The StepRecord object. - :param `study_name`: The name of the study - :param `parameter_info`: A dict containing information about parameters in the study + Initializes the `Step` object which acts as a way to track everything about a step. + + Args: + maestro_step_record: The `StepRecord` object. + study_name: The name of the study + parameter_info: A dict containing information about parameters in the study """ - self.mstep = maestro_step_record - self.study_name = study_name - self.parameter_info = parameter_info - self.__restart = False - self.params = {"cmd": {}, "restart_cmd": {}} + self.mstep: _StepRecord = maestro_step_record + self.study_name: str = study_name + self.parameter_info: Dict = parameter_info + self.__restart: bool = False + self.params: Dict = {"cmd": {}, "restart_cmd": {}} self.establish_params() - def get_cmd(self): + def get_cmd(self) -> str: """ - get the run command text body" + Retrieve the run command text body for the step. + + Returns: + The run command text body for the step. """ return self.mstep.step.__dict__["run"]["cmd"] - def get_restart_cmd(self): + def get_restart_cmd(self) -> str: """ - get the restart command text body, else return None" - """ - return self.mstep.step.__dict__["run"]["restart"] + Retrieve the restart command text body for the step. - def clone_changing_workspace_and_cmd(self, new_cmd=None, cmd_replacement_pairs=None, new_workspace=None): + Returns: + The restart command text body for the step, or None if no restart command is available. """ - Produces a deep copy of the current step, performing variable - substitutions as we go + return self.mstep.step.__dict__["run"]["restart"] - :param new_cmd : (Optional) replace the existing cmd with the new_cmd. - :param cmd_replacement_pairs : (Optional) replaces strings in the cmd - according to the list of pairs in cmd_replacement_pairs - :param new_workspace : (Optional) the workspace for the new step. + def clone_changing_workspace_and_cmd( + self, + new_cmd: str = None, + cmd_replacement_pairs: List[Tuple[str]] = None, + new_workspace: str = None, + ) -> "Step": + """ + Produces a deep copy of the current step, with optional modifications to + the command and workspace, performing variable substitutions as we go. + + This method creates a new instance of the Step class by cloning the + current step and allowing for modifications to the command text and + workspace. It performs variable substitutions in the command based on + the provided replacement pairs. + + Args: + new_cmd: If provided, replaces the existing command with this new command. + cmd_replacement_pairs: A list of pairs where each pair contains a string to + be replaced and its replacement. The method will perform replacements in + both the run command and the restart command. + new_workspace: If provided, sets this as the workspace for the new step. If + not specified, the current workspace will be used. + + Returns: + A new Step instance with the modified command and workspace. """ LOG.debug(f"clone called with new_workspace {new_workspace}") step_dict = deepcopy(self.mstep.step.__dict__) @@ -356,13 +451,35 @@ def clone_changing_workspace_and_cmd(self, new_cmd=None, cmd_replacement_pairs=N study_step.run = step_dict["run"] return Step(MerlinStepRecord(new_workspace, study_step, self), self.study_name, self.parameter_info) - def get_task_queue(self): - """Retrieve the task queue for the Step.""" + def get_task_queue(self) -> str: + """ + Retrieve the task queue for the current Step. + + Returns: + The name of the task queue for the Step, which may be influenced + by the configuration settings. + """ return self.get_task_queue_from_dict(self.mstep.step.__dict__) @staticmethod - def get_task_queue_from_dict(step_dict): - """given a maestro step dict, get the task queue""" + def get_task_queue_from_dict(step_dict: Dict) -> str: + """ + Get the task queue from a given Maestro step dictionary. + + This static method extracts the task queue information from the + provided step dictionary. It considers the configuration settings + to determine the appropriate queue name, including handling cases + where the task queue may be omitted. + + Args: + step_dict: A dictionary representation of a Maestro step, expected + to contain a "run" key with a "task_queue" entry. + + Returns: + The name of the task queue. If the task queue is not specified + or is set to "none", it returns the default queue name based + on the configuration. + """ from merlin.config.configfile import CONFIG # pylint: disable=C0415 queue_tag = CONFIG.celery.queue_tag @@ -382,34 +499,56 @@ def get_task_queue_from_dict(step_dict): return queue @property - def retry_delay(self): - """Returns the retry delay (default 1)""" + def retry_delay(self) -> int: + """ + Get the retry delay for the step. + + Returns: + The retry delay in seconds. Defaults to 1 if not specified. + """ default_retry_delay = 1 return self.mstep.step.__dict__["run"].get("retry_delay", default_retry_delay) @property - def max_retries(self): + def max_retries(self) -> int: """ - Returns the max number of retries for this step. + Get the maximum number of retries for this step. + + Returns: + The maximum number of retries for the step. """ return self.mstep.step.__dict__["run"]["max_retries"] @property - def restart(self): + def restart(self) -> bool: """ - Get the restart property + Get the restart property. + + Returns: + True if the step is set to restart, False otherwise. """ return self.__restart @restart.setter - def restart(self, val): + def restart(self, val: bool): """ - Set the restart property ensuring that restart is false + Set the restart property. + + Args: + val: The new value for the restart property. It should be + a boolean value indicating whether the step should restart. """ self.__restart = val def establish_params(self): - """If this step uses parameters, pull them from the step param map.""" + """ + Establish parameters for the step from the parameter map. + + This method checks if the current step uses parameters by accessing + the `step_param_map` from `parameter_info`. If parameters are found + for the current step, it updates the `params` dictionary with the + corresponding values. + """ try: step_params = self.parameter_info["step_param_map"][self.name()] for cmd_type in step_params: @@ -417,29 +556,52 @@ def establish_params(self): except KeyError: pass - def check_if_expansion_needed(self, labels): + def check_if_expansion_needed(self, labels: List[str]) -> bool: """ - :return : True if the cmd has any of the default keywords or spec - specified sample column labels. + Check if expansion is needed based on commands and labels. + + This method determines whether the command associated with the + current step requires expansion. It checks for the presence of + default keywords or specified sample column labels. + + Args: + labels: A list of labels to check against the commands. + + Returns: + True if the command requires expansion, False otherwise. """ return needs_merlin_expansion(self.get_cmd(), self.get_restart_cmd(), labels) - def get_workspace(self): + def get_workspace(self) -> str: """ - :return : The workspace this step is to be executed in. + Get the workspace for the current step. + + Returns: + The workspace associated with this step. """ return self.mstep.workspace.value - def name(self): + def name(self) -> str: """ - :return : The step name. + Get the name of the current step. + + Returns: + The name of the step. """ return self.mstep.step.__dict__["_name"] - def name_no_params(self): + def name_no_params(self) -> str: """ - Get the original name of the step without any parameters/samples in the name. - :returns: A string representing the name of the step + Get the original name of the step without parameters or sample labels. + + This method retrieves the name of the step and removes any + parameter labels or sample identifiers that may be included + in the name. It ensures that the returned name is clean and + free from extraneous characters, such as trailing periods or + underscores. + + Returns: + The cleaned name of the step, free from parameters and sample labels. """ # Get the name with everything still in it name = self.name() @@ -459,13 +621,28 @@ def name_no_params(self): return name - def execute(self, adapter_config): + def execute(self, adapter_config: Dict) -> ReturnCode: """ - Execute the step. + Execute the step with the provided adapter configuration. + + This method performs the execution of the step by configuring + the necessary parameters and invoking the appropriate adapter. + It updates the adapter configuration based on the step's + requirements, sets up the workspace, and generates the script + for execution. If a dry run is specified, it prepares the + workspace without executing any tasks. + + Args: + adapter_config (dict): A dictionary containing configuration + for the maestro script adapter, including:\n + - `shell`: The shell to use for execution. + - `batch_type`: The type of batch processing to use. + - `dry_run`: A boolean indicating whether to perform a + dry run (setup only, no execution). - :param adapter_config : A dictionary containing configuration for - the maestro script adapter, as well as which sort of adapter - to use. + Returns: + (common.enums.ReturnCode): A [`ReturnCode`][common.enums.ReturnCode] object representing + the result of the execution. """ # Update shell if the task overrides the default value from the batch section default_shell = adapter_config.get("shell") diff --git a/merlin/study/study.py b/merlin/study/study.py index 1d7b9d96d..442f38d3b 100644 --- a/merlin/study/study.py +++ b/merlin/study/study.py @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """This module represents all of the logic for a study""" import logging @@ -37,13 +14,16 @@ from contextlib import suppress from copy import deepcopy from pathlib import Path +from typing import Dict, List, Union +import numpy as np from cached_property import cached_property from maestrowf.datastructures.core import Study +from maestrowf.datastructures.core.studyenvironment import StudyEnvironment from maestrowf.maestro import load_parameter_generator from maestrowf.utils import create_dictionary -from merlin.common.abstracts.enums import ReturnCode +from merlin.common.enums import ReturnCode from merlin.spec import defaults from merlin.spec.expansion import determine_user_variables, expand_by_line, expand_env_vars, expand_line from merlin.spec.override import error_override_vars, replace_override_vars @@ -63,37 +43,83 @@ class MerlinStudy: # pylint: disable=R0902,R0904 """ Represents a Merlin study run on a specification. Used for 'merlin run'. - :param `filepath`: path to the desired specification file. - :param `override_vars`: Dictionary (keyword-variable name, value-variable - value) to override in the spec. - :param `restart_dir`: Filepath to restart study. If None, study runs - normally. - :param `samples_file`: File to load samples from. Ignores sample lookup - and generation in the spec if set. - :param `dry_run`: Flag to dry-run a workflow, which sets up the workspace but does not launch tasks. - :param `no_errors`: Flag to ignore some errors for testing. + This class manages the execution of a study based on a provided specification file, + handling sample data, output paths, workspace management, and the generation of a + Directed Acyclic Graph (DAG) for execution. + + Attributes: + dag (study.dag.DAG): Directed acyclic graph representing the execution flow of the study. + dry_run (bool): Flag indicating whether to perform a dry run of the workflow. + expanded_spec (spec.specification.MerlinSpec): The expanded specification after applying overrides. + filepath (str): Path to the desired specification file. + flux_command (str): Command for running flux jobs, if applicable. + info (str): Path to the 'merlin_info' directory within the workspace. + level_max_dirs (int): The number of directories at each level of the sample hierarchy. + no_errors (bool): Flag to ignore some errors for testing purposes. + original_spec (spec.specification.MerlinSpec): The original specification loaded + from the filepath. + output_path (str): Path to the output directory for the study. + override_vars (Dict[str, Union[str, int]]): Dictionary of variables to override in the specification. + parameter_labels (List[str]): List of parameter labels used in the study. + pargs (List[str]): Arguments for the parameter generator. + pgen_file (str): Filepath for the parameter generator, if applicable. + restart_dir (str): Filepath to restart the study, if applicable. + sample_labels (List[str]): The column labels of the samples. + samples (np.ndarray): The samples in the study. + samples_file (str): File to load samples from, if specified. + special_vars (Dict[str, str]): Dictionary of special variables used in the study. + timestamp (str): Timestamp representing the start time of the study. + user_vars (Dict[str, str]): The user-defined variables in the study. + workspace (str): Path to the workspace directory for the study. + + Methods: + generate_samples: Executes a command to generate sample data if the sample file is missing. + get_adapter_config: Builds and returns the adapter configuration dictionary. + get_expanded_spec: Returns a new YAML spec file with defaults, CLI overrides, and variable expansions. + get_sample_labels: Retrieves the column labels for the samples. + get_user_vars: Returns a dictionary of expanded user-defined variables from the specification. + label_clash_error: Checks for clashes between sample and parameter names. + load_dag: Generates a Directed Acyclic Graph (DAG) for the study's execution. + load_pgen: Executes a parameter generator script. + load_samples: Loads samples from disk or generates them if the file does not exist. + write_original_spec: Copies the original specification to the 'merlin_info' directory. """ def __init__( # pylint: disable=R0913 self, - filepath, - override_vars=None, - restart_dir=None, - samples_file=None, - dry_run=False, - no_errors=False, - pgen_file=None, - pargs=None, + filepath: str, + override_vars: Dict[str, Union[str, int]] = None, + restart_dir: str = None, + samples_file: str = None, + dry_run: bool = False, + no_errors: bool = False, + pgen_file: str = None, + pargs: List[str] = None, ): - self.filepath = filepath - self.original_spec = MerlinSpec.load_specification(filepath) - self.override_vars = override_vars + """ + Initializes a MerlinStudy object, which represents a study run based on a specification file. + + Args: + filepath: Path to the specification file for the study. + override_vars: Dictionary of variables to override in the specification. + restart_dir: Path to the directory for restarting the study. + samples_file: Path to a file containing sample data. If specified, the samples + will be loaded from this file. + dry_run: Flag indicating whether to perform a dry run of the workflow + without executing tasks. + no_errors: Flag to suppress certain errors for testing purposes. + pgen_file: Path to a parameter generator file. + pargs: Arguments for the parameter generator. + """ + self.filepath: str = filepath + self.original_spec: MerlinSpec = MerlinSpec.load_specification(filepath) + self.override_vars: Dict = override_vars error_override_vars(self.override_vars, self.original_spec.path) - self.samples_file = samples_file + self.samples_file: str = samples_file self.label_clash_error() - self.dry_run = dry_run - self.no_errors = no_errors + self.dry_run: bool = dry_run + self.no_errors: bool = no_errors # If we load from a file, record that in the object for provenance # downstream @@ -101,9 +127,9 @@ def __init__( # pylint: disable=R0913 self.original_spec.merlin["samples"]["file"] = self.samples_file self.original_spec.merlin["samples"]["generate"]["cmd"] = "" - self.restart_dir = restart_dir + self.restart_dir: str = restart_dir - self.special_vars = { + self.special_vars: Dict[str, str] = { "SPECROOT": self.original_spec.specroot, "MERLIN_TIMESTAMP": self.timestamp, "MERLIN_INFO": self.info, @@ -120,14 +146,21 @@ def __init__( # pylint: disable=R0913 } self._set_special_file_vars() - self.pgen_file = pgen_file - self.pargs = pargs + self.pgen_file: str = pgen_file + self.pargs: List[str] = pargs - self.dag = None + self.dag: DAG = None self.load_dag() def _set_special_file_vars(self): - """Setter for the orig, partial, and expanded file paths of a study.""" + """ + Sets the original, partial, and expanded file paths for a study. + + This method constructs file paths for three special variables + related to the study's specifications. It generates paths for + the original template, the executed run, and the archived copy + of the specification files based on the study's base file name. + """ shortened_filepath = self.filepath.replace(".out", "").replace(".partial", "").replace(".expanded", "") base_name = Path(shortened_filepath).stem self.special_vars["MERLIN_SPEC_ORIGINAL_TEMPLATE"] = os.path.join( @@ -145,16 +178,29 @@ def _set_special_file_vars(self): def write_original_spec(self): """ - Copy the original spec into merlin_info/ as '.orig.yaml'. + Copies the original specification file to the designated directory. + + This method copies the original specification file from its + current location to the `merlin_info/` directory, renaming it + to '.orig.yaml'. The base file name is derived + from the original specification's path. """ shutil.copyfile(self.original_spec.path, self.special_vars["MERLIN_SPEC_ORIGINAL_TEMPLATE"]) def label_clash_error(self): """ - Detect any illegal clashes between merlin's - merlin -> samples -> column_labels and Maestro's - global.parameters. Raises an error if any such - clash exists. + Detects illegal clashes between Merlin's sample column labels and + [Maestro's global parameters](https://maestrowf.readthedocs.io/en/latest/Maestro/specification.html#parameters-globalparameters). + + This method checks for any conflicts between the column labels + defined in the `merlin` section of the original specification and + the global parameters defined in the same specification. If a + column label is found to also exist in the global parameters, + a ValueError is raised to indicate the clash. + + Raises: + ValueError: If any column label in `merlin.samples.column_labels` + is also found in `merlin.globals`, indicating an illegal clash. """ if self.original_spec.merlin["samples"]: for label in self.original_spec.merlin["samples"]["column_labels"]: @@ -165,10 +211,24 @@ def label_clash_error(self): # to not use the MerlinStudy object so we disable this pylint error # pylint: disable=duplicate-code @staticmethod - def get_user_vars(spec): + def get_user_vars(spec: MerlinSpec) -> Dict[str, str]: """ - Using the spec environment, return a dictionary - of expanded user-defined variables. + Retrieves and expands user-defined variables from the specification environment. + + This static method examines the provided specification's environment + to collect user-defined variables and labels. It constructs a list + of these variables and passes them to the `determine_user_variables` + function to obtain a dictionary of expanded variables. + + Args: + spec (spec.specification.MerlinSpec): The specification object containing the environment from which + to extract user-defined variables. The environment should have keys + "variables" and/or "labels" that contain the relevant data. + + Returns: + A dictionary of expanded user-defined variables, where the keys + are variable names and the values are their corresponding + expanded values. """ uvars = [] if "variables" in spec.environment: @@ -180,14 +240,35 @@ def get_user_vars(spec): # pylint: enable=duplicate-code @property - def user_vars(self): - """Get the user defined variables""" + def user_vars(self) -> Dict[str, str]: + """ + Retrieves the user-defined variables for the study. + + This property accesses the original specification of the study and + retrieves the user-defined variables using the `get_user_vars` + method from this class. + + Returns: + A dictionary containing the user-defined variables + associated with the study. + """ return MerlinStudy.get_user_vars(self.original_spec) - def get_expanded_spec(self): + def get_expanded_spec(self) -> MerlinSpec: """ - Get a new yaml spec file with defaults, cli overrides, and variable expansions. - Useful for provenance. + Generates a new YAML specification file with applied defaults, + command-line interface (CLI) overrides, and variable expansions. + + This method creates a modified version of the original specification + by incorporating default values and user-defined overrides from the + command line. It also expands user-defined variables and reserved + words to produce a fully resolved specification. This is particularly + useful for tracking provenance and ensuring that the specification + accurately reflects all applied configurations. + + Returns: + (spec.specification.MerlinSpec): A new instance of the [`MerlinSpec`][spec.specification.MerlinSpec] + class that contains the fully expanded specification. """ # get specification including defaults and cli-overridden user variables new_env = replace_override_vars(self.original_spec.environment, self.override_vars) @@ -204,63 +285,98 @@ def get_expanded_spec(self): return expand_env_vars(result) @property - def samples(self): + def samples(self) -> np.ndarray: """ - Return this study's corresponding samples. + Retrieves the samples associated with this study. - :return: list of samples + This property checks if there are any samples defined in the + expanded specification of the study. If samples are present, + it loads and returns them; otherwise, it returns an empty list. + + Returns: + A numpy array of samples corresponding to the study. + If no samples are defined, an empty list is returned. """ if self.expanded_spec.merlin["samples"]: return self.load_samples() return [] - def get_sample_labels(self, from_spec): - """Return the column labels of the samples (if any)""" + def get_sample_labels(self, from_spec: MerlinSpec) -> List[str]: + """ + Retrieves the column labels of the samples from the provided specification. + + This method checks the specified [`MerlinSpec`][spec.specification.MerlinSpec] + object for sample information and returns the associated column labels if they + exist. If no sample labels are found, an empty list is returned. + + Args: + from_spec (spec.specification.MerlinSpec): The specification object + from which to extract sample column labels. It is expected to contain + a "samples" key within its "merlin" dictionary. + + Returns: + A list of column labels for the samples. If no sample labels are + present, an empty list is returned. + """ if from_spec.merlin["samples"]: return from_spec.merlin["samples"]["column_labels"] return [] @property - def sample_labels(self): + def sample_labels(self) -> List[str]: """ - Return this study's corresponding sample labels + Retrieves the labels of the samples associated with this study. - Example spec_file contents: + This property extracts the sample labels from the study's + expanded specification. It returns a list of labels that + correspond to the samples defined in the specification. - --spec_file.yaml-- - ... - merlin: - samples: - column_labels: [X0, X1] + Returns: + A list of sample labels. If no labels are defined, an empty list is returned. - :return: list of labels (e.g. ["X0", "X1"] ) + Example: + Given the following contents in a specification file: + + ```yaml + merlin: + samples: + column_labels: [X0, X1] + ``` + + This property would return: `["X0", "X1"]` """ return self.get_sample_labels(from_spec=self.expanded_spec) - def load_samples(self): + def load_samples(self) -> np.ndarray: """ - load this study's samples from disk, generating if the file does - not yet exist and the file is defined in the YAML file. - (no generation will occur if file is defined via __init__) + Loads the study's samples from disk, generating them if the file + does not exist and is defined in the YAML specification. - Runs the function defined in 'generate' and then loads up - the sample files defined in 'file', assigning them to the - variables in 'column_labels' + This method checks if a sample file is specified in the expanded + specification. If the file does not exist, it will invoke the + generation command defined in the 'generate' section of the + specification to create the sample file. Once the file is available, + it loads the samples into a NumPy array and assigns them to the + variables specified in 'column_labels'. - Example spec_file contents: + Returns: + A NumPy array containing the loaded samples. The shape of the + array will be (n_samples, n_features), where n_samples is + the number of samples loaded and n_features is the number + of features corresponding to the column labels. - --spec_file.yaml-- - ... - merlin: - samples: - generate: - cmd: python make_samples.py -outfile=samples.npy - file: samples.npy - column_labels: [X0, X1] + Example: + The spec file contents will look something like: - :return: numpy samples - :return: the samples loaded + ```yaml + merlin: + samples: + generate: + cmd: python make_samples.py -outfile=samples.npy + file: samples.npy + column_labels: [X0, X1] + ``` """ if self.samples_file is None: if self.expanded_spec.merlin["samples"]: @@ -287,18 +403,41 @@ def load_samples(self): return samples @property - def level_max_dirs(self): + def level_max_dirs(self) -> int: """ - Returns the maximum number of directory levels. + Retrieves the maximum number of directory levels for sample organization. + + This property checks the expanded specification for the maximum + number of directory levels defined under the 'merlin' section. + If the value is not found, it falls back to a default value + specified in the `defaults.SAMPLES` dictionary. + + Returns: + The maximum number of directory levels. If the value is + not specified in the expanded specification, the default + value from `defaults.SAMPLES["level_max_dirs"]` is returned. """ with suppress(TypeError, KeyError): return self.expanded_spec.merlin["samples"]["level_max_dirs"] return defaults.SAMPLES["level_max_dirs"] @cached_property - def output_path(self): + def output_path(self) -> str: """ Determines and creates an output directory for this study. + + This property checks if a restart directory is specified. If so, it validates + the existence of the directory and returns its absolute path. If no restart + directory is provided, it constructs the output path based on the original + specification and any override variables. The output path is expanded to + include user-defined variables and environment variables. If the directory + does not exist, it is created. + + Returns: + The absolute path to the output directory for the study. + + Raises: + ValueError: If the specified restart directory does not exist. """ if self.restart_dir is not None: output_path = self.restart_dir @@ -331,10 +470,18 @@ def output_path(self): return output_path @cached_property - def timestamp(self): + def timestamp(self) -> str: """ - Returns a timestamp string, representing the time this - study began. May be used as an id or unique identifier. + Returns a timestamp string representing the time this study began. + + This property generates a unique identifier based on the current time + when the study is initiated. If a restart directory is specified, it + extracts a substring from the directory name as the timestamp. Otherwise, + it formats the current time in the 'YYYYMMDD-HHMMSS' format. + + Returns: + A string representing the timestamp of the study's initiation, + which can be used as an identifier or unique key. """ if self.restart_dir is not None: return self.restart_dir.strip("/")[-15:] @@ -343,12 +490,22 @@ def timestamp(self): # TODO look into why pylint complains that this method is hidden # - might be because we reset self.workspace's value in the expanded_spec method @cached_property - def workspace(self): # pylint: disable=E0202 + def workspace(self) -> str: # pylint: disable=E0202 """ - Determines, makes, and returns the path to this study's - workspace directory. This directory holds workspace directories - for each step in the study, as well as 'merlin_info/'. The - name of this directory ends in a timestamp. + Determines, creates, and returns the path to this study's workspace directory. + + This property generates a unique workspace directory for the study, which + contains subdirectories for each step of the study and a 'merlin_info/' + directory. The name of the workspace directory is derived from the original + specification name and includes a timestamp to ensure uniqueness. If a + restart directory is specified, it validates the existence of the directory + and returns its absolute path. + + Returns: + The absolute path to the workspace directory for the study. + + Raises: + ValueError: If the specified restart directory does not exist. """ if self.restart_dir is not None: if not os.path.isdir(self.restart_dir): @@ -366,9 +523,17 @@ def workspace(self): # pylint: disable=E0202 # TODO look into why pylint complains that this method is hidden # - might be because we reset self.info's value in the expanded_spec method @cached_property - def info(self): # pylint: disable=E0202 + def info(self) -> str: # pylint: disable=E0202 """ - Creates the 'merlin_info' directory inside this study's workspace directory. + Creates and returns the path to the 'merlin_info' directory within the study's workspace. + + This property checks if a restart directory is specified. If not, it creates + the 'merlin_info' directory inside the study's workspace directory. This + directory is intended to store metadata and other relevant information related + to the study. + + Returns: + The absolute path to the 'merlin_info' directory. """ info_name = os.path.join(self.workspace, "merlin_info") if self.restart_dir is None: @@ -376,10 +541,22 @@ def info(self): # pylint: disable=E0202 return info_name @cached_property - def expanded_spec(self): + def expanded_spec(self) -> MerlinSpec: """ - Determines, writes to yaml, and loads into memory an expanded - specification. + Determines, writes to YAML, and loads into memory an expanded specification. + + This property handles the expansion of the study's specification based on + the original specification and any provided environment variables. If the + study is being restarted, it retrieves the previously expanded specification + without re-expanding it. Otherwise, it processes the original specification, + expands any tokens or shell references, and updates paths accordingly. + + Returns: + (spec.specification.MerlinSpec): The expanded specification object. + + Raises: + ValueError: If the expanded name for the workspace contains invalid + characters for a filename. """ # If we are restarting, we don't need to re-expand, just need to read # in the previously expanded spec @@ -448,9 +625,17 @@ def expanded_spec(self): return result @cached_property - def flux_command(self): + def flux_command(self) -> str: """ - Returns the flux command, this will include the full path, if flux_path given in the workflow. + Returns the full path to the flux command based on the specified workflow configuration. + + This property constructs the command to execute the flux binary. If a + `flux_path` is provided in the expanded specification's batch configuration, + it will use that path to create the full command. Otherwise, it defaults + to the standard 'flux' command. + + Returns: + The complete command string for executing flux. """ flux_bin = "flux" if "flux_path" in self.expanded_spec.batch.keys(): @@ -459,18 +644,24 @@ def flux_command(self): def generate_samples(self): """ - Runs the function defined in 'generate' if self.samples_file is not - yet a file. + Generates sample data by executing the command defined in the + 'generate' section of the specification if the sample file does + not already exist. - Example spec_file contents: + This method checks if the specified sample file exists. If it + does not, it retrieves the command from the YAML specification + and executes it using a subprocess. The output and error logs + from the command execution are saved to files for later review. - --spec_file.yaml-- - ... - merlin: - samples: - generate: - cmd: python make_samples.py -outfile=samples.npy + Example: + Here's an example sample generation command: + ```yaml + merlin: + samples: + generate: + cmd: python make_samples.py -outfile=samples.npy + ``` """ try: if not os.path.exists(self.samples_file): @@ -495,8 +686,33 @@ def generate_samples(self): LOG.error(f"Could not generate samples:\n{e}") return - def load_pgen(self, filepath, pargs, env): - """Creates a dict of variable names and values defined in a pgen script""" + def load_pgen(self, filepath: str, pargs: List[str], env: StudyEnvironment) -> Dict[str, Dict[str, str]]: + """ + Loads a parameter generator script and creates a dictionary of + variable names and their corresponding values. + + This method reads a parameter generator script from the specified + file path and extracts variable names and values defined within + the script. It constructs a dictionary where each key is a + variable name, and the value is another dictionary containing + the variable's label and its associated values. + + Args: + filepath: The path to the parameter generator script to be loaded. + pargs: A list of additional arguments to be passed to the parameter + generator. If None, an empty list will be used. + env: A Maestro + [`StudyEnvironment`](https://maestrowf.readthedocs.io/en/latest/Maestro/reference_guide/api_reference/datastructures/core/studyenvironment.html) + object containing custom information. + + Returns: + A dictionary where each key is a variable name and each value + is a dictionary containing:\n + - `values`: The values associated with the variable, + or None if not defined. + - `label`: The label of the variable as defined in the + parameter generator script. + """ if filepath: if pargs is None: pargs = [] @@ -512,8 +728,20 @@ def load_pgen(self, filepath, pargs, env): def load_dag(self): """ - Generates a dag (a directed acyclic execution graph). - Assigns it to `self.dag`. + Generates a Directed Acyclic Graph (DAG) for the execution of + the study and assigns it to the `self.dag` attribute. + + This method constructs a DAG based on the specifications defined + in the expanded study specification. It retrieves the study + environment, steps, and parameters, and initializes a Maestro + [`Study`](https://maestrowf.readthedocs.io/en/latest/Maestro/reference_guide/api_reference/datastructures/core/study.html) + object. The method then sets up the workspace and environment + for the study, configures it, and generates the DAG using the + Maestro framework. + + The generated DAG contains the execution flow of the study, + ensuring that all steps are executed in the correct order + without cycles. """ environment = self.expanded_spec.get_study_environment() steps = self.expanded_spec.get_study_steps() @@ -555,8 +783,25 @@ def load_dag(self): # To avoid pickling issues with _pass_detect_cycle from maestro, we unpack the dag here self.dag = DAG(maestro_dag.adjacency_table, maestro_dag.values, column_labels, study.name, parameter_info) - def get_adapter_config(self, override_type=None): - """Builds and returns the adapter configuration dictionary""" + def get_adapter_config(self, override_type: str = None) -> Dict[str, str]: + """ + Builds and returns the adapter configuration dictionary. + + This method constructs a configuration dictionary for the adapter + based on the specifications defined in `self.expanded_spec.batch`. + It ensures that the configuration includes a type, which can be + overridden if specified. The method also checks for a dry run + flag and adds relevant commands if the batch type is set to + "flux". + + Args: + override_type: An optional string to override the default adapter + type. If not provided, the type from the expanded specification + will be used. + + Returns: + A dictionary containing the adapter configuration. + """ adapter_config = dict(self.expanded_spec.batch) if "type" not in adapter_config.keys(): @@ -579,10 +824,16 @@ def get_adapter_config(self, override_type=None): return adapter_config @property - def parameter_labels(self): + def parameter_labels(self) -> List[str]: """ - Get the parameter labels for this study. - :returns: A list of parameter labels used in this study + Retrieves the parameter labels associated with this study. + + This property extracts parameter labels from the expanded specification + of the study. It accesses the parameters and their associated metadata, + collecting all labels defined for each parameter. + + Returns: + A list of parameter labels used in this study. """ parameters = self.expanded_spec.get_parameters() metadata = parameters.get_metadata() diff --git a/merlin/utils.py b/merlin/utils.py index 50b3474e2..b8ad8c071 100644 --- a/merlin/utils.py +++ b/merlin/utils.py @@ -1,32 +1,8 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ Module for project-wide utility functions. @@ -34,6 +10,7 @@ import getpass import logging import os +import pickle import re import socket import subprocess @@ -42,7 +19,7 @@ from copy import deepcopy from datetime import datetime, timedelta from types import SimpleNamespace -from typing import Callable, List, Optional, Union +from typing import Any, Callable, Dict, Generator, List, Tuple, Union import numpy as np import pkg_resources @@ -51,24 +28,30 @@ from tabulate import tabulate -try: - import cPickle as pickle -except ImportError: - import pickle - - LOG = logging.getLogger(__name__) ARRAY_FILE_FORMATS = ".npy, .csv, .tab" DEFAULT_FLUX_VERSION = "0.48.0" -def get_user_process_info(user=None, attrs=None): +def get_user_process_info(user: str = None, attrs: List[str] = None) -> List[Dict]: """ - Return a list of process info for all of the user's running processes. + Return a list of process information for all of the user's running processes. + + This function retrieves and returns details about the currently running processes + for a specified user. If no user is specified, it defaults to the current user. + It can also return information for all users if specified. + + Args: + user: The username for which to retrieve process information. If set to + 'all_users', retrieves processes for all users. Defaults to the current + user's username if not provided. + attrs: A list of attributes to include in the process information. Defaults + to ["pid", "name", "username", "cmdline"] if None. If "username" is not + included in the list, it will be added. - :param `user`: user name (default from getpass). Option: 'all_users': get - all processes - :param `atts`: the attributes to include + Returns: + A list of dictionaries containing the specified attributes for each process + belonging to the specified user or all users if 'all_users' is specified. """ if attrs is None: attrs = ["pid", "name", "username", "cmdline"] @@ -84,13 +67,23 @@ def get_user_process_info(user=None, attrs=None): return [p.info for p in psutil.process_iter(attrs=attrs) if user in p.info["username"]] -def check_pid(pid, user=None): +def check_pid(pid: int, user: str = None) -> bool: """ - Check if pid is in process list. + Check if a given process ID (PID) is in the process list for a specified user. - :param `pid`: process id - :param `user`: user name (default from getpass). Option: 'all_users': get - all processes + This function determines whether a specific PID is currently running + for the specified user. If no user is specified, it defaults to the + current user. It can also check for all users if specified. + + Args: + pid: The process ID to check for in the process list. + user: The username for which to check the process. If set to 'all_users', + checks processes for all users. Defaults to the current user's username + if not provided. + + Returns: + True if the specified PID is found in the process list for the given user, + False otherwise. """ user_processes = get_user_process_info(user=user) for process in user_processes: @@ -99,13 +92,23 @@ def check_pid(pid, user=None): return False -def get_pid(name, user=None): +def get_pid(name: str, user: str = None) -> List[int]: """ - Return pid of process with name. + Return the process ID(s) (PID) of processes with the specified name. + + This function retrieves the PID(s) of all running processes that match + the given name for a specified user. If no user is specified, it defaults + to the current user. It can also retrieve PIDs for all users if specified. - :param `name`: process name - :param `user`: user name (default from getpass). Option: 'all_users': get - all processes + Args: + name: The name of the process to search for. + user: The username for which to retrieve the process IDs. If set to + 'all_users', retrieves processes for all users. Defaults to the + current user's username if not provided. + + Returns: + A list of PIDs for processes matching the specified name. Returns None + if no matching processes are found. """ user_processes = get_user_process_info(user=user) name_list = [p["pid"] for p in user_processes if name in p["name"]] @@ -114,37 +117,68 @@ def get_pid(name, user=None): return None -def get_procs(name, user=None): +def get_procs(name: str, user: str = None) -> List[Tuple[int, str]]: """ - Return a list of (pid, cmdline) tuples of process with name. + Return a list of tuples containing the process ID (PID) and command line + of processes with the specified name. + + This function retrieves all running processes that match the given name + for a specified user. If no user is specified, it defaults to the current + user. It can also retrieve processes for all users if specified. - :param `name`: process name - :param `user`: user name (default from getpass). Option: 'all_users': get - all processes + Args: + name: The name of the process to search for. + user: The username for which to retrieve the process information. + If set to 'all_users', retrieves processes for all users. + Defaults to the current user's username if not provided. + + Returns: + A list of tuples, each containing the PID and command line of processes + matching the specified name. Returns an empty list if no matching + processes are found. """ user_processes = get_user_process_info(user=user) procs = [(p["pid"], p["cmdline"]) for p in user_processes if name in p["name"]] return procs -def is_running_psutil(cmd, user=None): +def is_running_psutil(cmd: str, user: str = None) -> bool: """ - Determine if process with given command is running. - Uses psutil command instead of call to 'ps' + Determine if a process with the given command is currently running. + + This function checks for the existence of any running processes that + match the specified command. It uses the `psutil` library to gather + process information instead of making a call to the 'ps' command. - :param `cmd`: process cmd - :param `user`: user name (default from getpass). Option: 'all_users': get - all processes + Args: + cmd: The command or command line snippet to search for in running + processes. + user: The username for which to check running processes. If set to + 'all_users', checks processes for all users. Defaults to the + current user's username if not provided. + + Returns: + True if at least one matching process is found; otherwise, False. """ user_processes = get_user_process_info(user=user) return any(cmd in " ".join(p["cmdline"]) for p in user_processes) -def is_running(name, all_users=False): +def is_running(name: str, all_users: bool = False) -> bool: """ - Determine if process with name is running. + Determine if a process with the specified name is currently running. + + This function checks for the existence of a running process with the + provided name by executing the 'ps' command. It can be configured to + check processes for all users or just the current user. + + Args: + name: The name of the process to search for. + all_users: If True, checks for processes across all users. Defaults + to False, which checks only the current user's processes. - :param `name`: process name + Returns: + True if a process with the specified name is found; otherwise, False. """ cmd = ["ps", "ux"] @@ -164,23 +198,42 @@ def is_running(name, all_users=False): return False -def expandvars2(path): +def expandvars2(path: str) -> str: """ - Replace shell strings from the current environment variables + Replace shell variables in the given path with their corresponding + environment variable values. - :param `path`: a path + This function expands shell-style variable references (e.g., $VAR) + in the input path using the current environment variables. It also + ensures that any escaped dollar signs (e.g., \\$) are not expanded. + + Args: + path: The input path containing shell variable references to be expanded. + + Returns: + The path with shell variables replaced by their corresponding values + from the environment, with unescaped variables expanded. """ return re.sub(r"(? List[str]: """ - Apply a regex filter to a list + Apply a regex filter to a list. - :param `regex` : the regular expression - :param `list_to_filter` : the list to filter + This function filters a given list based on a specified regular expression. + Depending on the `match` parameter, it can either match the entire string + or search for the regex pattern within the strings of the list. - :return `new_list` + Args: + regex: The regular expression to use for filtering the list. + list_to_filter: The list of strings to be filtered based on the regex. + match: If True, uses re.match to filter items that match the regex from + the start. If False, uses re.search to filter items that contain the + regex pattern. + + Returns: + A new list containing the filtered items that match the regex. """ r = re.compile(regex) # pylint: disable=C0103 if match: @@ -188,16 +241,27 @@ def regex_list_filter(regex, list_to_filter, match=True): return list(filter(r.search, list_to_filter)) -def apply_list_of_regex(regex_list, list_to_filter, result_list, match=False, display_warning: bool = True): +def apply_list_of_regex( + regex_list: List[str], list_to_filter: List[str], result_list: List[str], match: bool = False, display_warning: bool = True +): """ - Take a list of regex's, apply each regex to a list we're searching through, - and append each result to a result list. + Apply a list of regex patterns to a list and accumulate the results. + + This function takes each regex from the provided list of regex patterns + and applies it to the specified list. The results of each successful + match or search are appended to a result list. Optionally, it can display + a warning if a regex does not match any item in the list. - :param `regex_list`: A list of regular expressions to apply to the list_to_filter - :param `list_to_filter`: A list that we'll apply regexs to - :param `result_list`: A list that we'll append results of the regex filters to - :param `match`: A bool where when true we use re.match for applying the regex, - when false we use re.search for applying the regex. + Args: + regex_list: A list of regular expressions to apply to the `list_to_filter`. + list_to_filter: The list of strings that the regex patterns will be applied to. + result_list: The list where results of the regex filters will be appended. + match: If True, uses re.match for applying the regex. If False, uses re.search. + display_warning: If True, displays a warning message when no matches are + found for a regex. + + Side Effect: + This function modifies the `result_list` in place. """ for regex in regex_list: filter_results = set(regex_list_filter(regex, list_to_filter, match)) @@ -209,28 +273,37 @@ def apply_list_of_regex(regex_list, list_to_filter, result_list, match=False, di result_list += filter_results -def load_yaml(filepath): +def load_yaml(filepath: str) -> Dict: """ - Safely read a yaml file. + Safely read a YAML file and return its contents. - :param `filepath`: a filepath to a yaml file - :type filepath: str + Args: + filepath: The file path to the YAML file to be read. - :returns: Python objects holding the contents of the yaml file + Returns: + A dict representing the contents of the YAML file. """ with open(filepath, "r") as _file: return yaml.safe_load(_file) -def get_yaml_var(entry, var, default): +def get_yaml_var(entry: Dict[str, Any], var: str, default: Any) -> Any: """ - Return entry[var], else return default + Retrieve the value associated with a specified key from a YAML dictionary. - :param `entry`: a yaml dict - :param `var`: a yaml key - :param `default`: default value in the absence of data - """ + This function attempts to return the value of `var` from the provided `entry` + dictionary. If the key does not exist, it will try to access it as an attribute + of the entry object. If neither is found, the function returns the specified + `default` value. + + Args: + entry: A dictionary representing the contents of a YAML file. + var: The key or attribute name to retrieve from the entry. + default: The default value to return if the key or attribute is not found. + Returns: + The value associated with `var` in the entry, or `default` if not found. + """ try: return entry[var] except (TypeError, KeyError): @@ -240,19 +313,31 @@ def get_yaml_var(entry, var, default): return default -def load_array_file(filename, ndmin=2): +def load_array_file(filename: str, ndmin: int = 2) -> np.ndarray: """ - Loads up an array stored in filename, based on extension. + Load an array from a file based on its extension. - Valid filename extensions: - '.npy' : numpy binary file - '.csv' : comma separated text file - '.tab' : whitespace (or tab) separated text file + This function reads an array stored in the specified `filename`. + It supports three file types based on their extensions: - :param `filename` : The file to load - :param `ndmin` : The minimum number of dimensions to load - """ + - `.npy` for NumPy binary files + - `.csv` for comma-separated values + - `.tab` for whitespace (or tab) separated values + + The function ensures that the loaded array has at least `ndmin` dimensions. + If the array is in binary format, it checks the dimensions without altering the data. + + Args: + filename: The path to the file to load. + ndmin: The minimum number of dimensions the array should have. + Returns: + The loaded array. + + Raises: + TypeError: If the file extension is not one of the supported types + (`.npy`, `.csv`, `.tab`). + """ protocol = determine_protocol(filename) # Don't change binary-stored numpy arrays; just check dimensions @@ -277,9 +362,18 @@ def load_array_file(filename, ndmin=2): return array -def determine_protocol(fname): +def determine_protocol(fname: str) -> str: """ - Determines a file protocol based on file name extension. + Determine the file protocol based on the file name extension. + + Args: + fname: The name of the file whose protocol is to be determined. + + Returns: + The protocol corresponding to the file extension (e.g., 'hdf5'). + + Raises: + ValueError: If the provided file name does not have a valid extension. """ _, ext = os.path.splitext(fname) if ext.startswith("."): @@ -294,13 +388,21 @@ def determine_protocol(fname): def verify_filepath(filepath: str) -> str: """ - Verify that the filepath argument is a valid - file. + Verify that the given file path is valid and return its absolute form. + + This function checks if the specified `filepath` points to an existing file. + It expands any user directory shortcuts (e.g., `~`) and environment variables + in the provided path before verifying its existence. If the file does not exist, + a ValueError is raised. + + Args: + filepath: The path of the file to verify. - :param [str] `filepath`: the path of a file + Returns: + The verified absolute file path with expanded environment variables. - :return: the verified absolute filepath with expanded environment variables. - :rtype: str + Raises: + ValueError: If the provided file path does not point to a valid file. """ filepath = os.path.abspath(os.path.expandvars(os.path.expanduser(filepath))) if not os.path.isfile(filepath): @@ -310,13 +412,21 @@ def verify_filepath(filepath: str) -> str: def verify_dirpath(dirpath: str) -> str: """ - Verify that the dirpath argument is a valid - directory. + Verify that the given directory path is valid and return its absolute form. - :param [str] `dirpath`: the path of a directory + This function checks if the specified `dirpath` points to an existing directory. + It expands any user directory shortcuts (e.g., `~`) and environment variables + in the provided path before verifying its existence. If the directory does not exist, + a ValueError is raised. - :return: returns the absolute path with expanded environment vars for a given dirpath. - :rtype: str + Args: + dirpath: The path of the directory to verify. + + Returns: + The verified absolute directory path with expanded environment variables. + + Raises: + ValueError: If the provided directory path does not point to a valid directory. """ dirpath: str = os.path.abspath(os.path.expandvars(os.path.expanduser(dirpath))) if not os.path.isdir(dirpath): @@ -325,9 +435,19 @@ def verify_dirpath(dirpath: str) -> str: @contextmanager -def cd(path): # pylint: disable=C0103 +def cd(path: str) -> Generator[None, None, None]: # pylint: disable=C0103 """ - TODO + Context manager for changing the current working directory. + + This context manager changes the current working directory to the specified `path` + while executing the block of code within the context. Once the block is exited, + it restores the original working directory. + + Args: + path: The path to the directory to change to. + + Yields: + Control is yielded back to the block of code within the context. """ old_dir = os.getcwd() os.chdir(path) @@ -337,15 +457,37 @@ def cd(path): # pylint: disable=C0103 os.chdir(old_dir) -def pickle_data(filepath, content): - """Dump content to a pickle file""" +def pickle_data(filepath: str, content: Any): + """ + Dump content to a pickle file. + + This function serializes the given `content` and writes it to a specified file + in pickle format. The file is opened in write mode, which will overwrite any + existing content in the file. + + Args: + filepath: The path to the file where the content will be saved. + content: The data to be serialized and saved to the pickle file. + """ with open(filepath, "w") as f: # pylint: disable=C0103 pickle.dump(content, f) -def get_source_root(filepath): - """Used to find the absolute project path given a sample file path from - within the project. +def get_source_root(filepath: str) -> str: + """ + Find the absolute project path given a file path from within the project. + + This function determines the root directory of a project by analyzing the given + file path. It works by traversing the directory structure upwards until it + encounters a directory name that is not an integer, which is assumed to be the + project root. + + Args: + filepath: The file path from within the project for which to find the root. + + Returns: + The absolute path to the root directory of the project. Returns None if + the path corresponds to the root directory itself. """ filepath = os.path.abspath(filepath) sep = os.path.sep @@ -367,9 +509,19 @@ def get_source_root(filepath): return root -def ensure_directory_exists(**kwargs): +def ensure_directory_exists(**kwargs: Dict[Any, Any]) -> bool: """ - TODO + Ensure that the directory for the specified aggregate file exists. + + This function checks if the directory for the given `aggregate_file` exists. + If it does not exist, the function creates the necessary directories. + + Args: + **kwargs: Keyword arguments that must include:\n + - `aggregate_file` (str): The file path for which the directory needs to be ensured. + + Returns: + True if the directory already existed. False otherwise. """ aggregate_bundle = kwargs["aggregate_file"] dirname = os.path.dirname(aggregate_bundle) @@ -381,9 +533,23 @@ def ensure_directory_exists(**kwargs): return True -def nested_dict_to_namespaces(dic): - """Code for recursively converting dictionaries of dictionaries - into SimpleNamespaces instead. +def nested_dict_to_namespaces(dic: Dict) -> SimpleNamespace: + """ + Convert a nested dictionary into a nested SimpleNamespace structure. + + This function recursively transforms a dictionary (which may contain other + dictionaries) into a structure of SimpleNamespace objects. Each key in the + dictionary becomes an attribute of a SimpleNamespace, allowing for attribute-style + access to the data. + + Args: + dic: The nested dictionary to be converted. + + Returns: + A SimpleNamespace object representing the nested structure of the input dictionary. + + Raises: + TypeError: If the input is not a dictionary. """ def recurse(dic): @@ -400,9 +566,23 @@ def recurse(dic): return recurse(new_dic) -def nested_namespace_to_dicts(namespaces): - """Code for recursively converting namespaces of namespaces - into dictionaries instead. +def nested_namespace_to_dicts(namespaces: SimpleNamespace) -> Dict: + """ + Convert a nested SimpleNamespace structure into a nested dictionary. + + This function recursively transforms a SimpleNamespace (which may contain + other SimpleNamespaces) into a dictionary structure. Each attribute of the + SimpleNamespace becomes a key in the resulting dictionary. + + Args: + namespaces: The nested SimpleNamespace to be converted. + + Returns: + A dictionary representing the nested structure of the input + SimpleNamespace. + + Raises: + TypeError: If the input is not a SimpleNamespace. """ def recurse(namespaces): @@ -419,12 +599,28 @@ def recurse(namespaces): return recurse(new_ns) -def get_flux_version(flux_path, no_errors=False): +def get_flux_version(flux_path: str, no_errors: bool = False) -> str: """ - Return the flux version as a string + Retrieve the version of Flux as a string. + + This function executes the Flux binary located at `flux_path` with the + "version" command and parses the output to return the version number. + If the command fails or the Flux binary cannot be found, it can either + raise an error or return a default version based on the `no_errors` flag. - :param `flux_path`: the full path to the flux bin - :param `no_errors`: a flag to determine if this a test run to ignore errors + Args: + flux_path: The full path to the Flux binary. + no_errors: A flag to suppress error messages and exceptions. If set to + True, errors will be logged but not raised. + + Returns: + The version of Flux as a string. + + Raises: + FileNotFoundError: If the Flux binary cannot be found and `no_errors` + is set to False. + ValueError: If the version cannot be determined from the output and + `no_errors` is set to False. """ cmd = [flux_path, "version"] @@ -451,12 +647,23 @@ def get_flux_version(flux_path, no_errors=False): return flux_ver -def get_flux_cmd(flux_path, no_errors=False): +def get_flux_cmd(flux_path: str, no_errors: bool = False) -> str: """ - Return the flux run command as string + Generate the Flux run command based on the installed version. + + This function determines the appropriate Flux command to use for + running jobs, depending on the version of Flux installed at the + specified `flux_path`. It defaults to "flux run" for versions + greater than or equal to 0.48.x. For older versions, it adjusts + the command accordingly. - :param `flux_path`: the full path to the flux bin - :param `no_errors`: a flag to determine if this a test run to ignore errors + Args: + flux_path: The full path to the Flux binary. + no_errors: A flag to suppress error messages and exceptions + if set to True. + + Returns: + The appropriate Flux run command as a string. """ # The default is for flux version >= 0.48.x # this may change in the future. @@ -474,12 +681,23 @@ def get_flux_cmd(flux_path, no_errors=False): return flux_cmd -def get_flux_alloc(flux_path, no_errors=False): +def get_flux_alloc(flux_path: str, no_errors: bool = False) -> str: """ - Return the flux alloc command as string + Generate the `flux alloc` command based on the installed version. + + This function constructs the appropriate command for allocating + resources with Flux, depending on the version of Flux installed + at the specified `flux_path`. It defaults to "{flux_path} alloc" + for versions greater than or equal to 0.48.x. For older versions, + it adjusts the command accordingly. - :param `flux_path`: the full path to the flux bin - :param `no_errors`: a flag to determine if this a test run to ignore errors + Args: + flux_path: The full path to the Flux binary. + no_errors: A flag to suppress error messages and exceptions + if set to True. + + Returns: + The appropriate Flux allocation command as a string. """ # The default is for flux version >= 0.48.x # this may change in the future. @@ -495,12 +713,21 @@ def get_flux_alloc(flux_path, no_errors=False): return flux_alloc -def check_machines(machines): +def check_machines(machines: Union[str, List[str], Tuple[str]]) -> bool: """ - Return a True if the current machine is in the list of machines. + Check if the current machine is in the list of specified machines. + + This function determines whether the hostname of the current + machine matches any entry in a provided list of machine names. + It returns True if a match is found, otherwise it returns False. - :param `machines`: A single machine or list of machines to compare - with the current machine. + Args: + machines: A single machine name or a list/tuple of machine + names to compare with the current machine's hostname. + + Returns: + True if the current machine's hostname matches any of the + specified machines; False otherwise. """ local_hostname = socket.gethostname() @@ -514,36 +741,67 @@ def check_machines(machines): return False -def contains_token(string): +def contains_token(string: str) -> bool: """ - Return True if given string contains a token of the form $(STR). + Check if the given string contains a token of the form $(STR). + + This function uses a regular expression to search for tokens + that match the pattern $(), where consists of + alphanumeric characters and underscores. It returns True if + such a token is found; otherwise, it returns False. + + Args: + string: The input string to be checked for tokens. + + Returns: + True if the input string contains a token of the form + $(STR); False otherwise. """ if re.search(r"\$\(\w+\)", string): return True return False -def contains_shell_ref(string): +def contains_shell_ref(string: str) -> bool: """ - Return True if given string contains a shell variable reference - of the form $STR or ${STR}. + Check if the given string contains a shell variable reference. + + This function searches for shell variable references in the + format of $ or ${}, where + consists of alphanumeric characters and underscores. It returns + True if a match is found; otherwise, it returns False. + + Args: + string: The input string to be checked for shell + variable references. + + Returns: + True if the input string contains a shell variable + reference of the form $STR or ${STR}; False otherwise. """ if re.search(r"\$\w+", string) or re.search(r"\$\{\w+\}", string): return True return False -def needs_merlin_expansion( - cmd: str, restart_cmd: str, labels: List[str], include_sample_keywords: Optional[bool] = True -) -> bool: +def needs_merlin_expansion(cmd: str, restart_cmd: str, labels: List[str], include_sample_keywords: bool = True) -> bool: """ - Check if the cmd or restart cmd provided have variables that need expansion. + Check if the provided command or restart command contains variables that require expansion. - :param `cmd`: The command inside a study step to check for expansion - :param `restart_cmd`: The restart command inside a study step to check for expansion - :param `labels`: A list of labels to check for inside `cmd` and `restart_cmd` - :return : True if the cmd has any of the default keywords or spec - specified sample column labels. False otherwise. + This function checks both the command (`cmd`) and the restart command (`restart_cmd`) + for the presence of specified labels or sample keywords that indicate a need for variable + expansion. + + Args: + cmd: The command inside a study step to check for variable expansion. + restart_cmd: The restart command inside a study step to check for variable expansion. + labels: A list of labels to check for inside `cmd` and `restart_cmd`. + include_sample_keywords: Flag to indicate whether to include default sample keywords + in the label check. + + Returns: + True if either `cmd` or `restart_cmd` contains any of the specified labels + or default sample keywords, indicating a need for expansion. False otherwise. """ sample_keywords = ["MERLIN_SAMPLE_ID", "MERLIN_SAMPLE_PATH", "merlin_sample_id", "merlin_sample_path"] if include_sample_keywords: @@ -560,20 +818,28 @@ def needs_merlin_expansion( return False -def dict_deep_merge(dict_a: dict, dict_b: dict, path: str = None, conflict_handler: Callable = None): +def dict_deep_merge(dict_a: Dict, dict_b: Dict, path: str = None, conflict_handler: Callable = None): """ - This function recursively merges dict_b into dict_a. The built-in - merge of dictionaries in python (dict(dict_a) | dict(dict_b)) does not do a - deep merge so this function is necessary. This will only merge in new keys, - it will NOT update existing ones, unless you specify a conflict handler function. - Credit to this stack overflow post: https://stackoverflow.com/a/7205107. + Recursively merges `dict_b` into `dict_a`, performing a deep merge. + + This function combines two dictionaries by recursively merging + the contents of `dict_b` into `dict_a`. Unlike Python's built-in + dictionary merge, this function performs a deep merge, meaning + it will merge nested dictionaries instead of just updating top-level keys. + Existing keys in `dict_a` will not be updated unless a conflict handler + is provided to resolve key conflicts. - :param `dict_a`: A dict that we'll merge dict_b into - :param `dict_b`: A dict that we want to merge into dict_a - :param `path`: The path down the dictionary tree that we're currently at - :param `conflict_handler`: An optional function to handle conflicts between values at the same key. - The function should return the value to be used in the merged dictionary. - The default behavior without this argument is to log a warning. + Credit to [this stack overflow post](https://stackoverflow.com/a/7205107). + + Args: + dict_a: The dictionary that will be merged into. + dict_b: The dictionary to merge into `dict_a`. + path: The current path in the dictionary tree. This is used for logging + purposes during recursion. + conflict_handler: A function to handle conflicts when both dictionaries + have the same key with different values. The function should return + the value to be used in the merged dictionary. If not provided, a + warning will be logged for conflicts. """ # Check to make sure we have valid dict_a and dict_b input @@ -609,15 +875,28 @@ def dict_deep_merge(dict_a: dict, dict_b: dict, path: str = None, conflict_handl dict_a[key] = dict_b[key] -def find_vlaunch_var(vlaunch_var: str, step_cmd: str, accept_no_matches=False) -> str: +def find_vlaunch_var(vlaunch_var: str, step_cmd: str, accept_no_matches: bool = False) -> str: """ - Given a variable used for VLAUNCHER and the step cmd value, find - the variable. + Find and return the specified VLAUNCHER variable from the step command. + + This function searches for a variable defined in the VLAUNCHER context + within the provided step command string. It looks for the variable in + the format `MERLIN_=`. If the variable is found, + it returns the variable in a format suitable for use in a command string. + If the variable is not found, the behavior depends on the `accept_no_matches` flag. + + Args: + vlaunch_var: The name of the VLAUNCHER variable (without the prefix 'MERLIN_'). + step_cmd: The command string of a step where the variable may be defined. + accept_no_matches: If True, returns None if the variable is not found. + If False, raises a ValueError. Defaults to False. - :param `vlaunch_var`: The name of the VLAUNCHER variable (without MERLIN_) - :param `step_cmd`: The string for the cmd of a step - :param `accept_no_matches`: If True, return None if we couldn't find the variable. Otherwise, raise an error. - :returns: the `vlaunch_var` variable or None + Returns: + The variable in the format '${MERLIN_}' if found, otherwise None + (if `accept_no_matches` is True) or raises a ValueError (if False). + + Raises: + ValueError: If the variable is not found and `accept_no_matches` is False. """ matches = list(re.findall(rf"^(?!#).*MERLIN_{vlaunch_var}=\d+", step_cmd, re.MULTILINE)) @@ -631,10 +910,25 @@ def find_vlaunch_var(vlaunch_var: str, step_cmd: str, accept_no_matches=False) - # Time utilities def convert_to_timedelta(timestr: Union[str, int]) -> timedelta: - """Convert a timestring to a timedelta object. - Timestring is given in in the format '[days]:[hours]:[minutes]:seconds' - with days, hours, minutes all optional add ons. - If passed as an int, will convert to a string first and interpreted as seconds. + """ + Convert a time string or integer to a timedelta object. + + The function takes a time string formatted as + '[days]:[hours]:[minutes]:seconds', where days, hours, and minutes + are optional. If an integer is provided, it is interpreted as the + total number of seconds. + + Args: + timestr: The time string in the specified format or an integer + representing seconds. + + Returns: + A timedelta object representing the duration specified by the input + string or integer. + + Raises: + ValueError: If the input string does not conform to the expected + format or contains more than four time fields. """ # make sure it's a string in case we get an int timestr = str(timestr) @@ -652,7 +946,20 @@ def convert_to_timedelta(timestr: Union[str, int]) -> timedelta: def _repr_timedelta_HMS(time_delta: timedelta) -> str: # pylint: disable=C0103 - """Represent a timedelta object as a string in hours:minutes:seconds""" + """ + Represent a timedelta object as a string in 'HH:MM:SS' format. + + This function converts a given timedelta object into a string that + represents the duration in hours, minutes, and seconds. The output + is formatted as 'HH:MM:SS', with leading zeros for single-digit + hours, minutes, or seconds. + + Args: + time_delta: The timedelta object to be converted. + + Returns: + A string representation of the timedelta in the format 'HH:MM:SS'. + """ hours, remainder = divmod(time_delta.total_seconds(), 3600) minutes, seconds = divmod(remainder, 60) hours, minutes, seconds = int(hours), int(minutes), int(seconds) @@ -660,20 +967,46 @@ def _repr_timedelta_HMS(time_delta: timedelta) -> str: # pylint: disable=C0103 def _repr_timedelta_FSD(time_delta: timedelta) -> str: # pylint: disable=C0103 - """Represent a timedelta as a flux standard duration string, using seconds. + """ + Represent a timedelta as a Flux Standard Duration (FSD) string in seconds. + + The FSD format represents a duration as a floating-point number followed + by a suffix indicating the time unit. This function simplifies the + representation by using seconds and appending an 's' suffix. + + Args: + time_delta: The timedelta object to be converted. - flux standard duration (FSD) is a floating point number with a single character suffix: s,m,h or d. - This uses seconds for simplicity. + Returns: + A string representation of the timedelta in FSD format, expressed + in seconds (e.g., '123.45s'). """ fsd = f"{time_delta.total_seconds()}s" return fsd def repr_timedelta(time_delta: timedelta, method: str = "HMS") -> str: - """Represent a timedelta object as a string using a particular method. + """ + Represent a timedelta object as a string using a specified format method. + + This function formats a given timedelta object according to the chosen + method. The available methods are: + + - HMS: Represents the duration in 'hours:minutes:seconds' format. + - FSD: Represents the duration in Flux Standard Duration (FSD), + expressed as a floating-point number of seconds with an 's' suffix. - method - HMS: 'hours:minutes:seconds' - method - FSD: flux standard duration: 'seconds.s'""" + Args: + time_delta: The timedelta object to be formatted. + method: The method to use for formatting. Must be either 'HMS' or 'FSD'. + + Returns: + A string representation of the timedelta formatted according + to the specified method. + + Raises: + ValueError: If an invalid method is provided. + """ if method == "HMS": return _repr_timedelta_HMS(time_delta) if method == "FSD": @@ -682,16 +1015,27 @@ def repr_timedelta(time_delta: timedelta, method: str = "HMS") -> str: def convert_timestring(timestring: Union[str, int], format_method: str = "HMS") -> str: - """Converts a timestring to a different format. + """ + Converts a timestring to a specified format. - timestring: -either- - a timestring in in the format '[days]:[hours]:[minutes]:seconds' - days, hours, minutes are all optional add ons - -or- - an integer representing seconds - format_method: HMS - 'hours:minutes:seconds' - FSD - 'seconds.s' (flux standard duration) + This function accepts a timestring in a specific format or an integer + representing seconds, and converts it to a formatted string based on + the chosen format method. The available format methods are: + - HMS: Represents the duration in 'hours:minutes:seconds' format. + - FSD: Represents the duration in Flux Standard Duration (FSD), + expressed as a floating-point number of seconds with an 's' suffix. + + Args: + timestring: A string representing time in the format + '[days]:[hours]:[minutes]:seconds' (where days, hours, and + minutes are optional), or an integer representing time in seconds. + format_method: The method to use for formatting. Must be either + 'HMS' or 'FSD'. + + Returns: + A string representation of the converted timestring formatted + according to the specified method. """ LOG.debug(f"Timestring is: {timestring}") tdelta = convert_to_timedelta(timestring) @@ -701,16 +1045,37 @@ def convert_timestring(timestring: Union[str, int], format_method: str = "HMS") def pretty_format_hms(timestring: str) -> str: """ - Given an HMS timestring, format it so it removes blank entries and adds - labels. + Format an HMS timestring to remove blank entries and add appropriate labels. + + This function takes a timestring in the 'HH:MM:SS' format and formats + it by removing any components that are zero and appending the relevant + labels (days, hours, minutes, seconds). The output is a cleaner string + representation of the time. - :param `timestring`: the HMS timestring we'll format - :returns: a formatted timestring + Args: + timestring: A timestring formatted as 'DD:HH:MM:SS'. Each component + represents days, hours, minutes, and seconds, respectively. + Only the last four components are relevant and may include + leading zeros. + + Returns: + A formatted timestring with non-zero components labeled appropriately. + + Raises: + ValueError: If the input timestring contains more than four components + or is not in the expected format. Examples: - - "00:00:34:00" -> "34m" - - "01:00:00:25" -> "01d:25s" - - "00:19:44:28" -> "19h:44m:28s" + ```python + >>> pretty_format_hms("00:00:34:00") + '34m' + >>> pretty_format_hms("01:00:00:25") + '01d:25s' + >>> pretty_format_hms("00:19:44:28") + '19h:44m:28s' + >>> pretty_format_hms("00:00:00:00") + '00s' + ``` """ # Create labels and split the timestring labels = ["d", "h", "m", "s"] @@ -735,10 +1100,23 @@ def pretty_format_hms(timestring: str) -> str: def ws_time_to_dt(ws_time: str) -> datetime: """ - Converts a workspace timestring to a datetime object. + Convert a workspace timestring to a datetime object. + + This function takes a workspace timestring formatted as 'YYYYMMDD-HHMMSS' + and converts it into a corresponding datetime object. The input string + must adhere to the specified format to ensure accurate conversion. - :param `ws_time`: A workspace timestring in the format YYYYMMDD-HHMMSS - :returns: A datetime object created from the workspace timestring + Args: + ws_time: A workspace timestring in the format 'YYYYMMDD-HHMMSS', where:\n + - YYYY is the four-digit year, + - MM is the two-digit month (01 to 12), + - DD is the two-digit day (01 to 31), + - HH is the two-digit hour (00 to 23), + - MM is the two-digit minute (00 to 59), + - SS is the two-digit second (00 to 59). + + Returns: + A datetime object constructed from the provided workspace timestring. """ year = int(ws_time[:4]) month = int(ws_time[4:6]) @@ -751,11 +1129,19 @@ def ws_time_to_dt(ws_time: str) -> datetime: def get_package_versions(package_list: List[str]) -> str: """ - Return a table of the versions and locations of installed packages, including python. - If the package is not installed says "Not installed" + Generate a formatted table of installed package versions and their locations. + + This function takes a list of package names and checks for their installed + versions and locations. If a package is not installed, it indicates that + the package is "Not installed". The output includes the Python version + and its executable location at the top of the table. + + Args: + package_list: A list of package names to check for installed versions. - :param `package_list`: A list of packages. - :returns: A string that's a formatted table. + Returns: + A formatted string representing a table of package names, their versions, + and installation locations. """ table = [] for package in package_list: diff --git a/mkdocs.yml b/mkdocs.yml index 76c123bd3..ea52f4e30 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -24,6 +24,10 @@ nav: - Variables: "user_guide/variables.md" - Running Studies: "user_guide/running_studies.md" - Interpreting Study Output: "user_guide/interpreting_output.md" + - Merlin Database: + - Database Overview: "user_guide/database/index.md" + - The Database Command: "user_guide/database/database_cmd.md" + - Database Entities: "user_guide/database/entities.md" - Monitoring Studies: - Monitoring Overview: "user_guide/monitoring/index.md" - The Status Commands: "user_guide/monitoring/status_cmds.md" @@ -48,6 +52,9 @@ nav: - API Reference: "api_reference/" - Contact Us: "contact.md" +exclude_docs: | + /README.md + theme: name: material language: en @@ -98,23 +105,25 @@ markdown_extensions: - pymdownx.tabbed: alternate_style: true - markdown_grid_tables + - toc: + toc_depth: 4 plugins: - glightbox - search - codeinclude: title_mode: pymdownx.tabbed - # - gen-files: - # scripts: - # - docs/gen_ref_pages.py - # - mkdocstrings: - # handlers: - # python: - # paths: [merlin] - # options: - # docstring_style: sphinx - # - literate-nav: - # nav_file: SUMMARY.md + - gen-files: + scripts: + - docs/gen_ref_pages.py + - mkdocstrings: + handlers: + python: + paths: [merlin] + options: + docstring_style: google + - literate-nav: + nav_file: SUMMARY.md extra: social: diff --git a/requirements/dev.txt b/requirements/dev.txt index ab5962119..097e419da 100644 --- a/requirements/dev.txt +++ b/requirements/dev.txt @@ -12,5 +12,6 @@ sphinx>=2.0.0 alabaster johnnydep deepdiff +orderly-set==5.3.0; python_version == '3.8' pytest-order pytest-mock diff --git a/requirements/release.txt b/requirements/release.txt index dcdb9b81b..b2b0309ce 100644 --- a/requirements/release.txt +++ b/requirements/release.txt @@ -2,8 +2,6 @@ cached_property celery[redis,sqlalchemy]>=5.0.3 coloredlogs cryptography -importlib_metadata<5.0.0; python_version == '3.7' -importlib_resources; python_version < '3.7' maestrowf>=1.1.9dev1 numpy parse diff --git a/setup.py b/setup.py index 48def98f6..491592aad 100644 --- a/setup.py +++ b/setup.py @@ -1,32 +1,8 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## import os @@ -96,11 +72,12 @@ def extras_require(): long_description=readme(), long_description_content_type="text/markdown", classifiers=[ - "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", ], keywords="machine learning workflow", url="https://github.com/LLNL/merlin", @@ -111,7 +88,6 @@ def extras_require(): entry_points={ "console_scripts": [ "merlin=merlin.main:main", - "merlin-templates=merlin.merlin_templates:main", ] }, include_package_data=True, diff --git a/tests/README.md b/tests/README.md index 9b2f7ba1f..e2fa22cbf 100644 --- a/tests/README.md +++ b/tests/README.md @@ -4,16 +4,20 @@ This directory utilizes pytest to create and run our test suite. This directory is organized like so: - `conftest.py` - The script containing common fixtures for our tests +- `constants.py` - Constant values to be used throughout the test suite. +- `fixture_data_classes.py` - Dataclasses to help group pytest fixtures together, reducing the required number of imports. +- `fixture_types.py` - Aliases for type hinting fixtures. - `context_managers/` - The directory containing context managers used for testing - `celery_workers_manager.py` - A context manager used to manage celery workers for integration testing - `server_manager.py` - A context manager used to manage the redis server used for integration testing - `fixtures/` - The directory containing specific test module fixtures - `.py` - Fixtures for specific test modules - `integration/` - The directory containing integration tests - - `definitions.py` - The test definitions - `run_tests.py` - The script to run the tests defined in `definitions.py` - `conditions.py` - The conditions to test against + - `commands/` - The directory containing tests for commands of the Merlin library. + - `workflows/` The directory containing tests for entire workflow runs. - `unit/` - The directory containing unit tests - `test_*.py` - The actual test scripts to run diff --git a/tests/__init__.py b/tests/__init__.py index e69de29bb..3232b50b9 100644 --- a/tests/__init__.py +++ b/tests/__init__.py @@ -0,0 +1,5 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## diff --git a/tests/conftest.py b/tests/conftest.py index eb68542bf..e0f25b328 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ This module contains pytest fixtures to be used throughout the entire test suite. """ @@ -35,18 +12,28 @@ from copy import copy from glob import glob from time import sleep -from typing import Dict import pytest import yaml from _pytest.tmpdir import TempPathFactory from celery import Celery -from celery.canvas import Signature +from redis import Redis from merlin.config.configfile import CONFIG from tests.constants import CERT_FILES, SERVER_PASS from tests.context_managers.celery_workers_manager import CeleryWorkersManager from tests.context_managers.server_manager import RedisServerManager +from tests.fixture_data_classes import RedisBrokerAndBackend +from tests.fixture_types import ( + FixtureBytes, + FixtureCallable, + FixtureCelery, + FixtureDict, + FixtureModification, + FixtureRedis, + FixtureSignature, + FixtureStr, +) from tests.utils import create_cert_files, create_pass_file @@ -57,9 +44,11 @@ # Loading in Module Specific Fixtures # ####################################### - +fixture_glob = os.path.join("tests", "fixtures", "**", "*.py") pytest_plugins = [ - fixture_file.replace("/", ".").replace(".py", "") for fixture_file in glob("tests/fixtures/[!__]*.py", recursive=True) + fixture_file.replace(os.sep, ".").replace(".py", "") + for fixture_file in glob(fixture_glob, recursive=True) + if not fixture_file.endswith("__init__.py") ] @@ -94,13 +83,98 @@ def create_encryption_file(key_filepath: str, encryption_key: bytes, app_yaml_fi yaml.dump(app_yaml, app_yaml_file) +def setup_redis_config(config_type: str, merlin_server_dir: str): + """ + Sets up the Redis configuration for either broker or results backend. + + Args: + config_type: The type of configuration to set up ('broker' or 'results_backend'). + merlin_server_dir: The directory to the merlin test server configuration. + """ + port = 6379 + name = "redis" + pass_file = os.path.join(merlin_server_dir, "redis.pass") + create_pass_file(pass_file) + + if config_type == "broker": + CONFIG.broker.password = pass_file + CONFIG.broker.port = port + CONFIG.broker.name = name + elif config_type == "results_backend": + CONFIG.results_backend.password = pass_file + CONFIG.results_backend.port = port + CONFIG.results_backend.name = name + else: + raise ValueError("Invalid config_type. Must be 'broker' or 'results_backend'.") + + ####################################### ######### Fixture Definitions ######### ####################################### @pytest.fixture(scope="session") -def temp_output_dir(tmp_path_factory: TempPathFactory) -> str: +def path_to_test_specs() -> FixtureStr: + """ + Fixture to provide the path to the directory containing test specifications. + + This fixture returns the absolute path to the 'test_specs' directory + within the 'integration' folder of the test directory. It expands + environment variables and user home directory as necessary. + + Returns: + The absolute path to the 'test_specs' directory. + """ + path_to_test_dir = os.path.abspath(os.path.expandvars(os.path.expanduser(os.path.dirname(__file__)))) + return os.path.join(path_to_test_dir, "integration", "test_specs") + + +@pytest.fixture(scope="session") +def path_to_merlin_codebase() -> FixtureStr: + """ + Fixture to provide the path to the directory containing the Merlin code. + + This fixture returns the absolute path to the 'merlin' directory at the + top level of this repository. It expands environment variables and user + home directory as necessary. + + Returns: + The absolute path to the 'merlin' directory. + """ + path_to_test_dir = os.path.abspath(os.path.expandvars(os.path.expanduser(os.path.dirname(__file__)))) + return os.path.join(path_to_test_dir, "..", "merlin") + + +@pytest.fixture(scope="session") +def create_testing_dir() -> FixtureCallable: + """ + Fixture to create a temporary testing directory. + + Returns: + A function that creates the testing directory. + """ + + def _create_testing_dir(base_dir: str, sub_dir: str) -> str: + """ + Helper function to create a temporary testing directory. + + Args: + base_dir: The base directory where the testing directory will be created. + sub_dir: The name of the subdirectory to create. + + Returns: + The path to the created testing directory. + """ + testing_dir = os.path.join(base_dir, sub_dir) + if not os.path.exists(testing_dir): + os.makedirs(testing_dir) # Use makedirs to create intermediate directories if needed + return testing_dir + + return _create_testing_dir + + +@pytest.fixture(scope="session") +def temp_output_dir(tmp_path_factory: TempPathFactory) -> FixtureStr: """ This fixture will create a temporary directory to store output files of integration tests. The temporary directory will be stored at /tmp/`whoami`/pytest-of-`whoami`/. There can be at most @@ -123,21 +197,21 @@ def temp_output_dir(tmp_path_factory: TempPathFactory) -> str: @pytest.fixture(scope="session") -def merlin_server_dir(temp_output_dir: str) -> str: +def merlin_server_dir(temp_output_dir: FixtureStr) -> FixtureStr: """ The path to the merlin_server directory that will be created by the `redis_server` fixture. :param temp_output_dir: The path to the temporary output directory we'll be using for this test run :returns: The path to the merlin_server directory that will be created by the `redis_server` fixture """ - server_dir = f"{temp_output_dir}/merlin_server" + server_dir = os.path.join(temp_output_dir, "merlin_server") if not os.path.exists(server_dir): os.mkdir(server_dir) return server_dir @pytest.fixture(scope="session") -def redis_server(merlin_server_dir: str, test_encryption_key: bytes) -> str: +def redis_server(merlin_server_dir: FixtureStr, test_encryption_key: FixtureBytes) -> FixtureStr: """ Start a redis server instance that runs on localhost:6379. This will yield the redis server uri that can be used to create a connection with celery. @@ -146,11 +220,14 @@ def redis_server(merlin_server_dir: str, test_encryption_key: bytes) -> str: :param test_encryption_key: An encryption key to be used for testing :yields: The local redis server uri """ + os.environ["CELERY_ENV"] = "test" with RedisServerManager(merlin_server_dir, SERVER_PASS) as redis_server_manager: redis_server_manager.initialize_server() redis_server_manager.start_server() create_encryption_file( - f"{merlin_server_dir}/encrypt_data_key", test_encryption_key, app_yaml_filepath=f"{merlin_server_dir}/app.yaml" + os.path.join(merlin_server_dir, "encrypt_data_key"), + test_encryption_key, + app_yaml_filepath=os.path.join(merlin_server_dir, "app.yaml"), ) # Yield the redis_server uri to any fixtures/tests that may need it yield redis_server_manager.redis_server_uri @@ -158,18 +235,36 @@ def redis_server(merlin_server_dir: str, test_encryption_key: bytes) -> str: @pytest.fixture(scope="session") -def celery_app(redis_server: str) -> Celery: +def redis_client(redis_server: FixtureStr) -> FixtureRedis: + """ + Fixture that provides a Redis client instance for the test session. + It connects to this client using the url created from the `redis_server` + fixture. + + Args: + redis_server: The redis server uri we'll use to connect to redis + + Returns: + An instance of the Redis client that can be used to interact + with the Redis server. + """ + return Redis.from_url(url=redis_server) + + +@pytest.fixture(scope="session") +def celery_app(redis_server: FixtureStr) -> FixtureCelery: """ Create the celery app to be used throughout our integration tests. :param redis_server: The redis server uri we'll use to connect to redis :returns: The celery app object we'll use for testing """ + os.environ["CELERY_ENV"] = "test" return Celery("merlin_test_app", broker=redis_server, backend=redis_server) @pytest.fixture(scope="session") -def sleep_sig(celery_app: Celery) -> Signature: +def sleep_sig(celery_app: FixtureCelery) -> FixtureSignature: """ Create a task registered to our celery app and return a signature for it. Once requested by a test, you can set the queue you'd like to send this to @@ -191,7 +286,7 @@ def sleep_task(): @pytest.fixture(scope="session") -def worker_queue_map() -> Dict[str, str]: +def worker_queue_map() -> FixtureDict[str, str]: """ Worker and queue names to be used throughout tests @@ -201,7 +296,7 @@ def worker_queue_map() -> Dict[str, str]: @pytest.fixture(scope="class") -def launch_workers(celery_app: Celery, worker_queue_map: Dict[str, str]): +def launch_workers(celery_app: FixtureCelery, worker_queue_map: FixtureDict[str, str]): """ Launch the workers on the celery app fixture using the worker and queue names defined in the worker_queue_map fixture. @@ -219,7 +314,7 @@ def launch_workers(celery_app: Celery, worker_queue_map: Dict[str, str]): @pytest.fixture(scope="session") -def test_encryption_key() -> bytes: +def test_encryption_key() -> FixtureBytes: """ An encryption key to be used for tests that need it. @@ -243,28 +338,33 @@ def test_encryption_key() -> bytes: ####################################### -@pytest.fixture(scope="function") -def config(merlin_server_dir: str, test_encryption_key: bytes): +def _config(merlin_server_dir: FixtureStr, test_encryption_key: FixtureBytes): """ - DO NOT USE THIS FIXTURE IN A TEST, USE `redis_config` OR `rabbit_config` INSTEAD. - This fixture is intended to be used strictly by the `redis_config` and `rabbit_config` - fixtures. It sets up the CONFIG object but leaves certain broker settings unset. + Sets up the configuration for testing purposes by modifying the global CONFIG object. - :param merlin_server_dir: The directory to the merlin test server configuration - :param test_encryption_key: An encryption key to be used for testing - """ + This helper function prepares the broker and results backend configurations for testing + by creating necessary encryption key files and resetting the CONFIG object to its + original state after the tests are executed. + Args: + merlin_server_dir: The directory to the merlin test server configuration + test_encryption_key: An encryption key to be used for testing + + Yields: + This function yields control back to the test function, allowing tests to run + with the modified CONFIG settings. + """ # Create a copy of the CONFIG option so we can reset it after the test orig_config = copy(CONFIG) # Create an encryption key file (if it doesn't already exist) - key_file = f"{merlin_server_dir}/encrypt_data_key" + key_file = os.path.join(merlin_server_dir, "encrypt_data_key") create_encryption_file(key_file, test_encryption_key) # Set the broker configuration for testing - CONFIG.broker.password = None # This will be updated in `redis_broker_config` or `rabbit_broker_config` - CONFIG.broker.port = None # This will be updated in `redis_broker_config` or `rabbit_broker_config` - CONFIG.broker.name = None # This will be updated in `redis_broker_config` or `rabbit_broker_config` + CONFIG.broker.password = None # This will be updated in `redis_broker_config_*` or `rabbit_broker_config` + CONFIG.broker.port = None # This will be updated in `redis_broker_config_*` or `rabbit_broker_config` + CONFIG.broker.name = None # This will be updated in `redis_broker_config_*` or `rabbit_broker_config` CONFIG.broker.server = "127.0.0.1" CONFIG.broker.username = "default" CONFIG.broker.vhost = "host4testing" @@ -272,11 +372,11 @@ def config(merlin_server_dir: str, test_encryption_key: bytes): # Set the results_backend configuration for testing CONFIG.results_backend.password = ( - None # This will be updated in `redis_results_backend_config` or `mysql_results_backend_config` + None # This will be updated in `redis_results_backend_config_function` or `mysql_results_backend_config` ) - CONFIG.results_backend.port = None # This will be updated in `redis_results_backend_config` + CONFIG.results_backend.port = None # This will be updated in `redis_results_backend_config_function` CONFIG.results_backend.name = ( - None # This will be updated in `redis_results_backend_config` or `mysql_results_backend_config` + None # This will be updated in `redis_results_backend_config_function` or `mysql_results_backend_config` ) CONFIG.results_backend.dbname = None # This will be updated in `mysql_results_backend_config` CONFIG.results_backend.server = "127.0.0.1" @@ -295,51 +395,149 @@ def config(merlin_server_dir: str, test_encryption_key: bytes): @pytest.fixture(scope="function") -def redis_broker_config( - merlin_server_dir: str, config: "fixture" # noqa: F821 pylint: disable=redefined-outer-name,unused-argument -): +def config_function(merlin_server_dir: FixtureStr, test_encryption_key: FixtureBytes) -> FixtureModification: """ - This fixture is intended to be used for testing any functionality in the codebase - that uses the CONFIG object with a Redis broker and results_backend. + Sets up the configuration for testing with a function scope. - :param merlin_server_dir: The directory to the merlin test server configuration - :param config: The fixture that sets up most of the CONFIG object for testing + Warning: + DO NOT USE THIS FIXTURE IN A TEST, USE ONE OF THE SERVER SPECIFIC CONFIGURATIONS + (LIKE `redis_broker_config_function`, `rabbit_broker_config`, etc.) INSTEAD. + + This fixture modifies the global CONFIG object to prepare the broker and results backend + configurations for testing. It creates necessary encryption key files and ensures that + the original configuration is restored after the tests are executed. + + Args: + merlin_server_dir: The directory to the merlin test server configuration + test_encryption_key: An encryption key to be used for testing + + Yields: + This function yields control back to the test function, allowing tests to run + with the modified CONFIG settings. """ - pass_file = f"{merlin_server_dir}/redis.pass" - create_pass_file(pass_file) + yield from _config(merlin_server_dir, test_encryption_key) - CONFIG.broker.password = pass_file - CONFIG.broker.port = 6379 - CONFIG.broker.name = "redis" +@pytest.fixture(scope="class") +def config_class(merlin_server_dir: FixtureStr, test_encryption_key: FixtureBytes) -> FixtureModification: + """ + Sets up the configuration for testing with a class scope. + + Warning: + DO NOT USE THIS FIXTURE IN A TEST, USE ONE OF THE SERVER SPECIFIC CONFIGURATIONS + (LIKE `redis_broker_config_class`, `rabbit_broker_config`, etc.) INSTEAD. + + This fixture modifies the global CONFIG object to prepare the broker and results backend + configurations for testing. It creates necessary encryption key files and ensures that + the original configuration is restored after the tests are executed. + + Args: + merlin_server_dir: The directory to the merlin test server configuration + test_encryption_key: An encryption key to be used for testing + + Yields: + This function yields control back to the test function, allowing tests to run + with the modified CONFIG settings. + """ + yield from _config(merlin_server_dir, test_encryption_key) + + +@pytest.fixture(scope="function") +def redis_broker_config_function( + merlin_server_dir: FixtureStr, config_function: FixtureModification # pylint: disable=redefined-outer-name,unused-argument +) -> FixtureModification: + """ + Fixture for configuring the Redis broker for testing with a function scope. + + This fixture sets up the CONFIG object to use a Redis broker for testing any functionality + in the codebase that interacts with the broker. It modifies the configuration to point + to the specified Redis broker settings. + + Args: + merlin_server_dir: The directory to the merlin test server configuration. + config_function: The fixture that sets up most of the CONFIG object for testing. + + Yields: + This function yields control back to the test function, allowing tests to run + with the modified CONFIG settings. + """ + setup_redis_config("broker", merlin_server_dir) + yield + + +@pytest.fixture(scope="class") +def redis_broker_config_class( + merlin_server_dir: FixtureStr, config_class: FixtureModification # pylint: disable=redefined-outer-name,unused-argument +) -> FixtureModification: + """ + Fixture for configuring the Redis broker for testing with a class scope. + + This fixture sets up the CONFIG object to use a Redis broker for testing any functionality + in the codebase that interacts with the broker. It modifies the configuration to point + to the specified Redis broker settings. + + Args: + merlin_server_dir: The directory to the merlin test server configuration. + config_function: The fixture that sets up most of the CONFIG object for testing. + + Yields: + This function yields control back to the test function, allowing tests to run + with the modified CONFIG settings. + """ + setup_redis_config("broker", merlin_server_dir) yield @pytest.fixture(scope="function") -def redis_results_backend_config( - merlin_server_dir: str, config: "fixture" # noqa: F821 pylint: disable=redefined-outer-name,unused-argument -): +def redis_results_backend_config_function( + merlin_server_dir: FixtureStr, config_function: FixtureModification # pylint: disable=redefined-outer-name,unused-argument +) -> FixtureModification: """ - This fixture is intended to be used for testing any functionality in the codebase - that uses the CONFIG object with a Redis results_backend. + Fixture for configuring the Redis results backend for testing with a function scope. - :param merlin_server_dir: The directory to the merlin test server configuration - :param config: The fixture that sets up most of the CONFIG object for testing + This fixture sets up the CONFIG object to use a Redis results backend for testing any + functionality in the codebase that interacts with the results backend. It modifies the + configuration to point to the specified Redis results backend settings. + + Args: + merlin_server_dir: The directory to the merlin test server configuration. + config_function: The fixture that sets up most of the CONFIG object for testing. + + Yields: + This function yields control back to the test function, allowing tests to run + with the modified CONFIG settings. """ - pass_file = f"{merlin_server_dir}/redis.pass" - create_pass_file(pass_file) + setup_redis_config("results_backend", merlin_server_dir) + yield - CONFIG.results_backend.password = pass_file - CONFIG.results_backend.port = 6379 - CONFIG.results_backend.name = "redis" +@pytest.fixture(scope="class") +def redis_results_backend_config_class( + merlin_server_dir: FixtureStr, config_class: FixtureModification # pylint: disable=redefined-outer-name,unused-argument +) -> FixtureModification: + """ + Fixture for configuring the Redis results backend for testing with a class scope. + + This fixture sets up the CONFIG object to use a Redis results backend for testing any + functionality in the codebase that interacts with the results backend. It modifies the + configuration to point to the specified Redis results backend settings. + + Args: + merlin_server_dir: The directory to the merlin test server configuration. + config_function: The fixture that sets up most of the CONFIG object for testing. + + Yields: + This function yields control back to the test function, allowing tests to run + with the modified CONFIG settings. + """ + setup_redis_config("results_backend", merlin_server_dir) yield @pytest.fixture(scope="function") def rabbit_broker_config( - merlin_server_dir: str, config: "fixture" # noqa: F821 pylint: disable=redefined-outer-name,unused-argument -): + merlin_server_dir: FixtureStr, config_function: FixtureModification # pylint: disable=redefined-outer-name,unused-argument +) -> FixtureModification: """ This fixture is intended to be used for testing any functionality in the codebase that uses the CONFIG object with a RabbitMQ broker. @@ -347,7 +545,7 @@ def rabbit_broker_config( :param merlin_server_dir: The directory to the merlin test server configuration :param config: The fixture that sets up most of the CONFIG object for testing """ - pass_file = f"{merlin_server_dir}/rabbit.pass" + pass_file = os.path.join(merlin_server_dir, "rabbit.pass") create_pass_file(pass_file) CONFIG.broker.password = pass_file @@ -359,8 +557,8 @@ def rabbit_broker_config( @pytest.fixture(scope="function") def mysql_results_backend_config( - merlin_server_dir: str, config: "fixture" # noqa: F821 pylint: disable=redefined-outer-name,unused-argument -): + merlin_server_dir: FixtureStr, config_function: FixtureModification # pylint: disable=redefined-outer-name,unused-argument +) -> FixtureModification: """ This fixture is intended to be used for testing any functionality in the codebase that uses the CONFIG object with a MySQL results_backend. @@ -368,7 +566,7 @@ def mysql_results_backend_config( :param merlin_server_dir: The directory to the merlin test server configuration :param config: The fixture that sets up most of the CONFIG object for testing """ - pass_file = f"{merlin_server_dir}/mysql.pass" + pass_file = os.path.join(merlin_server_dir, "mysql.pass") create_pass_file(pass_file) create_cert_files(merlin_server_dir, CERT_FILES) @@ -381,3 +579,81 @@ def mysql_results_backend_config( CONFIG.results_backend.ca_certs = CERT_FILES["ssl_ca"] yield + + +@pytest.fixture(scope="function") +def redis_broker_and_backend_function( + redis_client: FixtureRedis, + redis_server: FixtureStr, + redis_broker_config_function: FixtureModification, + redis_results_backend_config_function: FixtureModification, +): + """ + Fixture for setting up Redis broker and backend for function-scoped tests. + + This fixture creates an instance of `RedisBrokerAndBackend`, which + encapsulates all necessary Redis-related fixtures required for + establishing connections to Redis as both a broker and a backend + during function-scoped tests. + + Args: + redis_client: A fixture that provides a client for interacting with the + Redis server. + redis_server: A fixture providing the connection string to the Redis + server instance. + redis_broker_config_function: A fixture that modifies the configuration + to point to the Redis server used as the message broker for + function-scoped tests. + redis_results_backend_config_function: A fixture that modifies the + configuration to point to the Redis server used for storing results + in function-scoped tests. + + Returns: + An instance containing the Redis client, server connection string, and + configuration modifications for both the broker and backend. + """ + return RedisBrokerAndBackend( + client=redis_client, + server=redis_server, + broker_config=redis_broker_config_function, + results_backend_config=redis_results_backend_config_function, + ) + + +@pytest.fixture(scope="class") +def redis_broker_and_backend_class( + redis_client: FixtureRedis, + redis_server: FixtureStr, + redis_broker_config_class: FixtureModification, + redis_results_backend_config_class: FixtureModification, +) -> RedisBrokerAndBackend: + """ + Fixture for setting up Redis broker and backend for class-scoped tests. + + This fixture creates an instance of `RedisBrokerAndBackend`, which + encapsulates all necessary Redis-related fixtures required for + establishing connections to Redis as both a broker and a backend + during class-scoped tests. + + Args: + redis_client: A fixture that provides a client for interacting with the + Redis server. + redis_server: A fixture providing the connection string to the Redis + server instance. + redis_broker_config_function: A fixture that modifies the configuration + to point to the Redis server used as the message broker for + class-scoped tests. + redis_results_backend_config_function: A fixture that modifies the + configuration to point to the Redis server used for storing results + in class-scoped tests. + + Returns: + An instance containing the Redis client, server connection string, and + configuration modifications for both the broker and backend. + """ + return RedisBrokerAndBackend( + client=redis_client, + server=redis_server, + broker_config=redis_broker_config_class, + results_backend_config=redis_results_backend_config_class, + ) diff --git a/tests/constants.py b/tests/constants.py index 26cfe4c0a..9dcdba5b4 100644 --- a/tests/constants.py +++ b/tests/constants.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ This module will store constants that will be used throughout our test suite. """ diff --git a/tests/context_managers/__init__.py b/tests/context_managers/__init__.py index e69de29bb..3232b50b9 100644 --- a/tests/context_managers/__init__.py +++ b/tests/context_managers/__init__.py @@ -0,0 +1,5 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## diff --git a/tests/context_managers/celery_task_manager.py b/tests/context_managers/celery_task_manager.py new file mode 100644 index 000000000..f21febb21 --- /dev/null +++ b/tests/context_managers/celery_task_manager.py @@ -0,0 +1,161 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Module to define functionality for sending tasks to the server +and ensuring they're cleared from the server when the test finishes. +""" + +from types import TracebackType +from typing import List, Type + +from celery import Celery +from celery.result import AsyncResult +from redis import Redis + + +class CeleryTaskManager: + """ + A context manager for managing Celery tasks. + + This class provides a way to send tasks to a Celery server and clean up + any tasks that were sent during its lifetime. It is designed to be used + as a context manager, ensuring that tasks are properly removed from the + server when the context is exited. + + Attributes: + celery_app: The Celery application instance. + redis_server: The Redis server connection string. + """ + + def __init__(self, app: Celery, redis_client: Redis): + self.celery_app: Celery = app + self.redis_client = redis_client + + def __enter__(self) -> "CeleryTaskManager": + """ + Enters the runtime context related to this object. + + Returns: + The current instance of the manager. + """ + return self + + def __exit__(self, exc_type: Type[Exception], exc_value: Exception, traceback: TracebackType): + """ + Exits the runtime context and performs cleanup. + + This method removes any tasks currently in the server. + + Args: + exc_type: The exception type raised, if any. + exc_value: The exception instance raised, if any. + traceback: The traceback object, if an exception was raised. + """ + self.remove_tasks() + + def send_task(self, task_name: str, *args, **kwargs) -> AsyncResult: + """ + Sends a task to the Celery server. + + This method will be used for tests that don't call + `merlin run`, allowing for isolated test functionality. + + Args: + task_name: The name of the task to send to the server. + *args: Additional positional arguments to pass to the task. + **kwargs: Additional keyword arguments to pass to the task. + + Returns: + A Celery AsyncResult object containing information about the + task that was sent to the server. + """ + valid_kwargs = [ + "add_to_parent", + "chain", + "chord", + "compression", + "connection", + "countdown", + "eta", + "exchange", + "expires", + "group_id", + "group_index", + "headers", + "ignore_result", + "link", + "link_error", + "parent_id", + "priority", + "producer", + "publisher", + "queue", + "replaced_task_nesting", + "reply_to", + "result_cls", + "retries", + "retry", + "retry_policy", + "root_id", + "route_name", + "router", + "routing_key", + "serializer", + "shadow", + "soft_time_limit", + "task_id", + "task_type", + "time_limit", + ] + send_task_kwargs = {key: kwargs.pop(key) for key in valid_kwargs if key in kwargs} + + return self.celery_app.send_task(task_name, args=args, kwargs=kwargs, **send_task_kwargs) + + def remove_tasks(self): + """ + Removes tasks from the Celery server. + + Tasks are removed in two ways: + 1. By purging the Celery app queues, which will only purge tasks + sent with `send_task`. + 2. By deleting the remaining queues in the Redis server, which will + purge any tasks that weren't sent with `send_task` (e.g., tasks + sent with `merlin run`). + """ + # Purge the tasks + self.celery_app.control.purge() + + # Purge any remaining tasks directly through redis that may have been missed + queues = self.get_queue_list() + for queue in queues: + self.redis_client.delete(queue) + + def get_queue_list(self) -> List[str]: + """ + Builds a list of Celery queues that exist on the Redis server. + + Queries the Redis server for its keys and returns the keys + that represent the Celery queues. + + Returns: + A list of Celery queue names. + """ + cursor = 0 + queues = [] + while True: + # Get the 'merlin' queue if it exists + cursor, matching_queues = self.redis_client.scan(cursor=cursor, match="merlin") + queues.extend(matching_queues) + + # Get any queues that start with '[merlin]' + cursor, matching_queues = self.redis_client.scan(cursor=cursor, match="\\[merlin\\]*") + queues.extend(matching_queues) + + if cursor == 0: + break + + return queues diff --git a/tests/context_managers/celery_workers_manager.py b/tests/context_managers/celery_workers_manager.py index 5e750b6ef..93423dfbe 100644 --- a/tests/context_managers/celery_workers_manager.py +++ b/tests/context_managers/celery_workers_manager.py @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Module to define functionality for test workers and how to start/stop them in their own processes. @@ -52,7 +29,8 @@ class CeleryWorkersManager: def __init__(self, app: Celery): self.app = app - self.running_workers = [] + self.running_workers = set() + self.run_worker_processes = set() self.worker_processes = {} self.echo_processes = {} @@ -80,8 +58,9 @@ def __exit__(self, exc_type: Type[Exception], exc_value: Exception, traceback: T try: if str(pid) in ps_proc.stdout: os.kill(pid, signal.SIGKILL) - except ProcessLookupError as exc: - raise ProcessLookupError(f"PID {pid} not found. Output of 'ps ux':\n{ps_proc.stdout}") from exc + # If the process can't be found then it doesn't exist anymore + except ProcessLookupError: + pass def _is_worker_ready(self, worker_name: str, verbose: bool = False) -> bool: """ @@ -144,7 +123,7 @@ def start_worker(self, worker_launch_cmd: List[str]): """ self.app.worker_main(worker_launch_cmd) - def launch_worker(self, worker_name: str, queues: List[str], concurrency: int = 1): + def launch_worker(self, worker_name: str, queues: List[str], concurrency: int = 1, prefetch: int = 1): """ Launch a single worker. We'll add the process that the worker is running in to the list of worker processes. We'll also create an echo process to simulate a celery worker command that will show up with 'ps ux'. @@ -158,6 +137,8 @@ def launch_worker(self, worker_name: str, queues: List[str], concurrency: int = self.stop_all_workers() raise ValueError(f"The worker {worker_name} is already running. Choose a different name.") + queues = [f"[merlin]_{queue}" for queue in queues] + # Create the launch command for this worker worker_launch_cmd = [ "worker", @@ -167,6 +148,8 @@ def launch_worker(self, worker_name: str, queues: List[str], concurrency: int = ",".join(queues), "--concurrency", str(concurrency), + "--prefetch-multiplier", + str(prefetch), f"--logfile={worker_name}.log", "--loglevel=DEBUG", ] @@ -174,9 +157,9 @@ def launch_worker(self, worker_name: str, queues: List[str], concurrency: int = # Create an echo command to simulate a running celery worker since our celery worker will be spun up in # a different process and we won't be able to see it with 'ps ux' like we normally would echo_process = subprocess.Popen( # pylint: disable=consider-using-with - f"echo 'celery merlin_test_app {' '.join(worker_launch_cmd)}'; sleep inf", + f"echo 'celery -A merlin_test_app {' '.join(worker_launch_cmd)}'; sleep inf", shell=True, - preexec_fn=os.setpgrp, # Make this the parent of the group so we can kill the 'sleep inf' that's spun up + start_new_session=True, # Make this the parent of the group so we can kill the 'sleep inf' that's spun up ) self.echo_processes[worker_name] = echo_process.pid @@ -184,7 +167,7 @@ def launch_worker(self, worker_name: str, queues: List[str], concurrency: int = worker_process = multiprocessing.Process(target=self.start_worker, args=(worker_launch_cmd,)) worker_process.start() self.worker_processes[worker_name] = worker_process - self.running_workers.append(worker_name) + self.running_workers.add(worker_name) # Wait for the worker to launch properly try: @@ -204,6 +187,24 @@ def launch_workers(self, worker_info: Dict[str, Dict]): for worker_name, worker_settings in worker_info.items(): self.launch_worker(worker_name, worker_settings["queues"], worker_settings["concurrency"]) + def add_run_workers_process(self, pid: int): + """ + Add a process ID for a `merlin run-workers` process to the + set that tracks all `merlin run-workers` processes that are + currently running. + + Warning: + The process that's added here must utilize the + `start_new_session=True` setting of subprocess.Popen. This + is necessary for us to be able to terminate all the workers + that are started with it safely since they will be seen as + child processes of the `merlin run-workers` process. + + Args: + pid: The process ID running `merlin run-workers`. + """ + self.run_worker_processes.add(pid) + def stop_worker(self, worker_name: str): """ Stop a single running worker and its associated processes. @@ -223,12 +224,16 @@ def stop_worker(self, worker_name: str): self.worker_processes[worker_name].kill() # Terminate the echo process and its sleep inf subprocess - os.killpg(os.getpgid(self.echo_processes[worker_name]), signal.SIGTERM) - sleep(2) + if self.echo_processes[worker_name] is not None: + os.killpg(os.getpgid(self.echo_processes[worker_name]), signal.SIGKILL) + sleep(2) def stop_all_workers(self): """ Stop all of the running workers and the processes associated with them. """ + for run_worker_pid in self.run_worker_processes: + os.killpg(os.getpgid(run_worker_pid), signal.SIGKILL) + for worker_name in self.running_workers: self.stop_worker(worker_name) diff --git a/tests/context_managers/server_manager.py b/tests/context_managers/server_manager.py index b99afb2c6..79a133cba 100644 --- a/tests/context_managers/server_manager.py +++ b/tests/context_managers/server_manager.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Module to define functionality for managing the containerized server used for testing. @@ -91,7 +97,7 @@ def stop_server(self): if "Merlin server terminated." not in kill_process.stderr: # If it wasn't, try to kill the process by using the pid stored in a file created by `merlin server` try: - with open(f"{self.server_dir}/merlin_server.pf", "r") as process_file: + with open(os.path.join(self.server_dir, "merlin_server.pf"), "r") as process_file: server_process_info = yaml.load(process_file, yaml.Loader) os.kill(int(server_process_info["image_pid"]), signal.SIGKILL) # If the file can't be found then let's make sure there's even a redis-server process running diff --git a/tests/fixture_data_classes.py b/tests/fixture_data_classes.py new file mode 100644 index 000000000..89c71f876 --- /dev/null +++ b/tests/fixture_data_classes.py @@ -0,0 +1,105 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module houses dataclasses to be used with pytest fixtures. +""" + +from dataclasses import dataclass + +from tests.fixture_types import FixtureInt, FixtureModification, FixtureRedis, FixtureStr + + +@dataclass +class RedisBrokerAndBackend: + """ + Data class to encapsulate all Redis-related fixtures required for + establishing connections to Redis for both the broker and backend. + + This class simplifies the management of Redis fixtures by grouping + them into a single object, reducing the number of individual fixture + imports needed in tests that require Redis functionality. + + Attributes: + client: A fixture that provides a client for interacting + with the Redis server. + server: A fixture providing the connection string to the + Redis server instance. + broker_config: A fixture that modifies the configuration + to point to the Redis server used as the message broker. + results_backend_config: A fixture that modifies the + configuration to point to the Redis server used for storing + results. + """ + + client: FixtureRedis + server: FixtureStr + results_backend_config: FixtureModification + broker_config: FixtureModification + + +@dataclass +class FeatureDemoSetup: + """ + Data class to encapsulate all feature-demo-related fixtures required + for testing the feature demo workflow. + + This class simplifies the management of feature demo setup fixtures + by grouping them into a single object, reducing the number of individual + fixture imports needed in tests that require feature demo setup. + + Attributes: + testing_dir: The path to the temp output directory for feature_demo workflow tests. + num_samples: An integer representing the number of samples to use in the feature_demo + workflow. + name: A string representing the name to use for the feature_demo workflow. + path: The path to the feature demo YAML file. + """ + + testing_dir: FixtureStr + num_samples: FixtureInt + name: FixtureStr + path: FixtureStr + + +@dataclass +class ChordErrorSetup: + """ + Data class to encapsulate all chord-error-related fixtures required + for testing the chord error workflow. + + This class simplifies the management of chord error setup fixtures + by grouping them into a single object, reducing the number of individual + fixture imports needed in tests that require chord error setup. + + Attributes: + testing_dir: The path to the temp output directory for chord_err workflow tests. + name: A string representing the name to use for the chord_err workflow. + path: The path to the chord error YAML file. + """ + + testing_dir: FixtureStr + name: FixtureStr + path: FixtureStr + + +@dataclass +class MonitorSetup: + """ + Data class to encapsulate all monitor-related fixtures required + for testing the monitor command. + + This class simplifies the management of monitor setup fixtures + by grouping them into a single object, reducing the number of individual + fixture imports needed in tests that require monitor setup. + + Attributes: + testing_dir: The path to the temp output directory for monitor tests. + auto_restart_yaml: The path to the monitor auto restart YAML file. + """ + + testing_dir: FixtureStr + auto_restart_yaml: FixtureStr diff --git a/tests/fixture_types.py b/tests/fixture_types.py new file mode 100644 index 000000000..579de7669 --- /dev/null +++ b/tests/fixture_types.py @@ -0,0 +1,87 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +It's hard to type hint pytest fixtures in a way that makes it clear +that the variable being used is a fixture. This module will created +aliases for these fixtures in order to make it easier to track what's +happening. + +The types here will be defined as such: +- `FixtureBytes`: A fixture that returns bytes +- `FixtureCelery`: A fixture that returns a Celery app object +- `FixtureDict`: A fixture that returns a dictionary +- `FixtureInt`: A fixture that returns an integer +- `FixtureModification`: A fixture that modifies something but never actually + returns/yields a value to be used in the test. +- `FixtureRedis`: A fixture that returns a Redis client +- `FixtureSignature`: A fixture that returns a Celery Signature object +- `FixtureStr`: A fixture that returns a string +""" + +import sys +from argparse import Namespace +from collections.abc import Callable +from typing import Any, Dict, Generic, List, Tuple, TypeVar + +import pytest +from celery import Celery +from celery.canvas import Signature +from redis import Redis + + +# TODO convert unit test type hinting to use these +# - likely will do this when I work on API docs for test library + +K = TypeVar("K") +V = TypeVar("V") + +# TODO when we drop support for Python 3.8, remove this if/else statement +# Check Python version +if sys.version_info >= (3, 9): + from typing import Annotated + + FixtureBytes = Annotated[bytes, pytest.fixture] + FixtureCallable = Annotated[Callable, pytest.fixture] + FixtureCelery = Annotated[Celery, pytest.fixture] + FixtureDict = Annotated[Dict[K, V], pytest.fixture] + FixtureInt = Annotated[int, pytest.fixture] + FixtureModification = Annotated[Any, pytest.fixture] + FixtureNamespace = Annotated[Namespace, pytest.fixture] + FixtureRedis = Annotated[Redis, pytest.fixture] + FixtureSignature = Annotated[Signature, pytest.fixture] + FixtureStr = Annotated[str, pytest.fixture] + FixtureTuple = Annotated[Tuple[K], pytest.fixture] + FixtureList = Annotated[List[K], pytest.fixture] +else: + # Fallback for Python 3.8 + class FixtureDict(Generic[K, V], Dict[K, V]): + """ + This class is necessary to allow FixtureDict to be subscriptable + when using it to type hint. + """ + + class FixtureTuple(Generic[K], Tuple[K]): + """ + This class is necessary to allow FixtureTuple to be subscriptable + when using it to type hint. + """ + + class FixtureList(Generic[K], List[K]): + """ + This class is necessary to allow FixtureList to be subscriptable + when using it to type hint. + """ + + FixtureBytes = pytest.fixture + FixtureCallable = pytest.fixture + FixtureCelery = pytest.fixture + FixtureInt = pytest.fixture + FixtureModification = pytest.fixture + FixtureNamespace = pytest.fixture + FixtureRedis = pytest.fixture + FixtureSignature = pytest.fixture + FixtureStr = pytest.fixture diff --git a/tests/fixtures/__init__.py b/tests/fixtures/__init__.py index ab3e56590..84687dc03 100644 --- a/tests/fixtures/__init__.py +++ b/tests/fixtures/__init__.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ This directory is for help modularizing fixture definitions so that we don't have to store every single fixture in the `conftest.py` file. diff --git a/tests/fixtures/chord_err.py b/tests/fixtures/chord_err.py new file mode 100644 index 000000000..88930a62f --- /dev/null +++ b/tests/fixtures/chord_err.py @@ -0,0 +1,120 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Fixtures specifically for help testing the chord_err workflow. +""" + +import os +import subprocess + +import pytest + +from tests.fixture_data_classes import ChordErrorSetup, RedisBrokerAndBackend +from tests.fixture_types import FixtureCallable, FixtureStr +from tests.integration.helper_funcs import copy_app_yaml_to_cwd, run_workflow + + +# pylint: disable=redefined-outer-name + + +@pytest.fixture(scope="session") +def chord_err_testing_dir(create_testing_dir: FixtureCallable, temp_output_dir: FixtureStr) -> FixtureStr: + """ + Fixture to create a temporary output directory for tests related to testing the + chord_err workflow. + + Args: + create_testing_dir: A fixture which returns a function that creates the testing directory. + temp_output_dir: The path to the temporary ouptut directory we'll be using for this test run. + + Returns: + The path to the temporary testing directory for chord_err workflow tests. + """ + return create_testing_dir(temp_output_dir, "chord_err_testing") + + +@pytest.fixture(scope="session") +def chord_err_name() -> FixtureStr: + """ + Defines a specific name to use for the chord_err workflow. This helps ensure + that even if changes were made to the chord_err workflow, tests using this fixture + should still run the same thing. + + Returns: + A string representing the name to use for the chord_err workflow. + """ + return "chord_err_test" + + +@pytest.fixture(scope="session") +def chord_err_setup( + chord_err_testing_dir: FixtureStr, + chord_err_name: FixtureStr, + path_to_test_specs: FixtureStr, +) -> ChordErrorSetup: + """ + Fixture for setting up the environment required for testing the chord error workflow. + + This fixture prepares the necessary configuration and paths for executing tests related + to the chord error workflow. It aggregates the required parameters into a single + [`ChordErrorSetup`][fixture_data_classes.ChordErrorSetup] data class instance, which + simplifies the management of these parameters in tests. + + Args: + chord_err_testing_dir: The path to the temporary output directory where chord error + workflow tests will store their results. + chord_err_name: A string representing the name to use for the chord error workflow. + path_to_test_specs: The base path to the Merlin test specs directory, which is + used to locate the chord error YAML file. + + Returns: + A [`ChordErrorSetup`][fixture_data_classes.ChordErrorSetup] instance containing + the testing directory, name, and path to the chord error YAML file, which can + be used in tests that require this setup. + """ + chord_err_path = os.path.join(path_to_test_specs, "chord_err.yaml") + return ChordErrorSetup( + testing_dir=chord_err_testing_dir, + name=chord_err_name, + path=chord_err_path, + ) + + +@pytest.fixture(scope="class") +def chord_err_run_workflow( + redis_broker_and_backend_class: RedisBrokerAndBackend, + chord_err_setup: ChordErrorSetup, + merlin_server_dir: FixtureStr, +) -> subprocess.CompletedProcess: + """ + Run the chord error workflow. + + This fixture sets up and executes the chord error workflow using the specified configurations + and parameters. It prepares the environment by modifying the CONFIG object to connect to a + Redis server and runs the workflow with the provided name and output path. + + Args: + redis_broker_and_backend_class: Fixture for setting up Redis broker and + backend for class-scoped tests. + chord_err_setup: A fixture that returns a [`ChordErrorSetup`][fixture_data_classes.ChordErrorSetup] + instance. + merlin_server_dir: A fixture to provide the path to the merlin_server directory that will be + created by the [`redis_server`][conftest.redis_server] fixture. + + Returns: + The completed process object containing information about the execution of the workflow, including + return code, stdout, and stderr. + """ + # Setup the test + copy_app_yaml_to_cwd(merlin_server_dir) + # chord_err_path = os.path.join(path_to_test_specs, "chord_err.yaml") + + # Create the variables to pass in to the workflow + vars_to_substitute = [f"NAME={chord_err_setup.name}", f"OUTPUT_PATH={chord_err_setup.testing_dir}"] + + # Run the workflow + return run_workflow(redis_broker_and_backend_class.client, chord_err_setup.path, vars_to_substitute) diff --git a/tests/fixtures/config.py b/tests/fixtures/config.py new file mode 100644 index 000000000..0a2855db2 --- /dev/null +++ b/tests/fixtures/config.py @@ -0,0 +1,29 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Fixtures for the modules in the `config` folder. +""" + +import pytest + +from tests.fixture_types import FixtureCallable, FixtureStr + + +@pytest.fixture(scope="session") +def config_testing_dir(create_testing_dir: FixtureCallable, temp_output_dir: FixtureStr) -> FixtureStr: + """ + Fixture to create a temporary output directory for tests related to testing the + `config` directory. + + Args: + create_testing_dir: A fixture which returns a function that creates the testing directory. + temp_output_dir: The path to the temporary ouptut directory we'll be using for this test run. + + Returns: + The path to the temporary testing directory for tests of files in the `config` directory. + """ + return create_testing_dir(temp_output_dir, "config_testing") diff --git a/tests/fixtures/examples.py b/tests/fixtures/examples.py index 7c4626e3e..f2b1f17e2 100644 --- a/tests/fixtures/examples.py +++ b/tests/fixtures/examples.py @@ -1,22 +1,28 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Fixtures specifically for help testing the modules in the examples/ directory. """ -import os - import pytest +from tests.fixture_types import FixtureCallable, FixtureStr + @pytest.fixture(scope="session") -def examples_testing_dir(temp_output_dir: str) -> str: +def examples_testing_dir(create_testing_dir: FixtureCallable, temp_output_dir: FixtureStr) -> FixtureStr: """ Fixture to create a temporary output directory for tests related to the examples functionality. - :param temp_output_dir: The path to the temporary output directory we'll be using for this test run - :returns: The path to the temporary testing directory for examples tests - """ - testing_dir = f"{temp_output_dir}/examples_testing" - if not os.path.exists(testing_dir): - os.mkdir(testing_dir) + Args: + create_testing_dir: A fixture which returns a function that creates the testing directory. + temp_output_dir: The path to the temporary output directory we'll be using for this test run. - return testing_dir + Returns: + The path to the temporary testing directory for examples tests. + """ + return create_testing_dir(temp_output_dir, "examples_testing") diff --git a/tests/fixtures/feature_demo.py b/tests/fixtures/feature_demo.py new file mode 100644 index 000000000..77a5e1c38 --- /dev/null +++ b/tests/fixtures/feature_demo.py @@ -0,0 +1,142 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Fixtures specifically for help testing the feature_demo workflow. +""" + +import os +import subprocess + +import pytest + +from tests.fixture_data_classes import FeatureDemoSetup, RedisBrokerAndBackend +from tests.fixture_types import FixtureCallable, FixtureInt, FixtureStr +from tests.integration.helper_funcs import copy_app_yaml_to_cwd, run_workflow + + +# pylint: disable=redefined-outer-name + + +@pytest.fixture(scope="session") +def feature_demo_testing_dir(create_testing_dir: FixtureCallable, temp_output_dir: FixtureStr) -> FixtureStr: + """ + Fixture to create a temporary output directory for tests related to testing the + feature_demo workflow. + + Args: + create_testing_dir: A fixture which returns a function that creates the testing directory. + temp_output_dir: The path to the temporary ouptut directory we'll be using for this test run. + + Returns: + The path to the temporary testing directory for feature_demo workflow tests. + """ + return create_testing_dir(temp_output_dir, "feature_demo_testing") + + +@pytest.fixture(scope="session") +def feature_demo_num_samples() -> FixtureInt: + """ + Defines a specific number of samples to use for the feature_demo workflow. + This helps ensure that even if changes were made to the feature_demo workflow, + tests using this fixture should still run the same thing. + + Returns: + An integer representing the number of samples to use in the feature_demo workflow. + """ + return 8 + + +@pytest.fixture(scope="session") +def feature_demo_name() -> FixtureStr: + """ + Defines a specific name to use for the feature_demo workflow. This helps ensure + that even if changes were made to the feature_demo workflow, tests using this fixture + should still run the same thing. + + Returns: + A string representing the name to use for the feature_demo workflow. + """ + return "feature_demo_test" + + +@pytest.fixture(scope="session") +def feature_demo_setup( + feature_demo_testing_dir: FixtureStr, + feature_demo_num_samples: FixtureInt, + feature_demo_name: FixtureStr, + path_to_merlin_codebase: FixtureStr, +) -> FeatureDemoSetup: + """ + Fixture for setting up the environment required for testing the feature demo workflow. + + This fixture prepares the necessary configuration and paths for executing tests related + to the feature demo workflow. It aggregates the required parameters into a single + [`FeatureDemoSetup`][fixture_data_classes.FeatureDemoSetup] data class instance, which + simplifies the management of these parameters in tests. + + Args: + feature_demo_testing_dir: The path to the temporary output directory where + feature demo workflow tests will store their results. + feature_demo_num_samples: An integer representing the number of samples + to use in the feature demo workflow. + feature_demo_name: A string representing the name to use for the feature + demo workflow. + path_to_merlin_codebase: The base path to the Merlin codebase, which is + used to locate the feature demo YAML file. + + Returns: + A [`FeatureDemoSetup`][fixture_data_classes.FeatureDemoSetup] instance containing + the testing directory, number of samples, name, and path to the feature demo + YAML file, which can be used in tests that require this setup. + """ + demo_workflow = os.path.join("examples", "workflows", "feature_demo", "feature_demo.yaml") + feature_demo_path = os.path.join(path_to_merlin_codebase, demo_workflow) + return FeatureDemoSetup( + testing_dir=feature_demo_testing_dir, + num_samples=feature_demo_num_samples, + name=feature_demo_name, + path=feature_demo_path, + ) + + +@pytest.fixture(scope="class") +def feature_demo_run_workflow( + redis_broker_and_backend_class: RedisBrokerAndBackend, + feature_demo_setup: FeatureDemoSetup, + merlin_server_dir: FixtureStr, +) -> subprocess.CompletedProcess: + """ + Run the feature demo workflow. + + This fixture sets up and executes the feature demo workflow using the specified configurations + and parameters. It prepares the environment by modifying the CONFIG object to connect to a + Redis server and runs the demo workflow with the provided sample size and name. + + Args: + redis_broker_and_backend_class: Fixture for setting up Redis broker and + backend for class-scoped tests. + feature_demo_setup: A fixture that returns a [`FeatureDemoSetup`][fixture_data_classes.FeatureDemoSetup] + instance. + merlin_server_dir: A fixture to provide the path to the merlin_server directory that will be + created by the [`redis_server`][conftest.redis_server] fixture. + + Returns: + The completed process object containing information about the execution of the workflow, including + return code, stdout, and stderr. + """ + # Setup the test + copy_app_yaml_to_cwd(merlin_server_dir) + + # Create the variables to pass in to the workflow + vars_to_substitute = [ + f"N_SAMPLES={feature_demo_setup.num_samples}", + f"NAME={feature_demo_setup.name}", + f"OUTPUT_PATH={feature_demo_setup.testing_dir}", + ] + + # Run the workflow + return run_workflow(redis_broker_and_backend_class.client, feature_demo_setup.path, vars_to_substitute) diff --git a/tests/fixtures/monitor.py b/tests/fixtures/monitor.py new file mode 100644 index 000000000..fffc166d3 --- /dev/null +++ b/tests/fixtures/monitor.py @@ -0,0 +1,64 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Fixtures specifically for help testing the monitor command. +""" + +import os + +import pytest + +from tests.fixture_data_classes import MonitorSetup +from tests.fixture_types import FixtureCallable, FixtureStr + + +# pylint: disable=redefined-outer-name + + +@pytest.fixture(scope="session") +def monitor_testing_dir(create_testing_dir: FixtureCallable, temp_output_dir: FixtureStr) -> FixtureStr: + """ + Fixture to create a temporary output directory for tests related to testing the + monitor workflow. + + Args: + create_testing_dir: A fixture which returns a function that creates the testing directory. + temp_output_dir: The path to the temporary ouptut directory we'll be using for this test run. + + Returns: + The path to the temporary testing directory for monitor tests. + """ + return create_testing_dir(temp_output_dir, "monitor_testing") + + +@pytest.fixture(scope="session") +def monitor_setup( + monitor_testing_dir: FixtureStr, + path_to_test_specs: FixtureStr, +) -> MonitorSetup: + """ + Fixture for setting up the environment required for testing the monitor command. + + This fixture prepares the necessary configuration and paths for executing tests related + to the monitor command. It aggregates the required parameters into a single + [`MonitorSetup`][fixture_data_classes.MonitorSetup] data class instance, which + simplifies the management of these parameters in tests. + + Args: + monitor_testing_dir: The path to the temporary output directory where monitor + command tests will store their results. + path_to_test_specs: The base path to the Merlin test specs directory. + + Returns: + A [`MonitorSetup`][fixture_data_classes.MonitorSetup] instance containing + information needed for the monitor command tests. + """ + auto_restart_yaml_path = os.path.join(path_to_test_specs, "monitor_auto_restart_test.yaml") + return MonitorSetup( + testing_dir=monitor_testing_dir, + auto_restart_yaml=auto_restart_yaml_path, + ) diff --git a/tests/fixtures/run_command.py b/tests/fixtures/run_command.py new file mode 100644 index 000000000..fe8edfb97 --- /dev/null +++ b/tests/fixtures/run_command.py @@ -0,0 +1,29 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Fixtures specifically for help testing the `merlin run` command. +""" + +import pytest + +from tests.fixture_types import FixtureCallable, FixtureStr + + +@pytest.fixture(scope="session") +def run_command_testing_dir(create_testing_dir: FixtureCallable, temp_output_dir: FixtureStr) -> FixtureStr: + """ + Fixture to create a temporary output directory for tests related to testing the + `merlin run` functionality. + + Args: + create_testing_dir: A fixture which returns a function that creates the testing directory. + temp_output_dir: The path to the temporary ouptut directory we'll be using for this test run. + + Returns: + The path to the temporary testing directory for `merlin run` tests. + """ + return create_testing_dir(temp_output_dir, "run_command_testing") diff --git a/tests/fixtures/server.py b/tests/fixtures/server.py index 4f4a07a2c..a75df22d8 100644 --- a/tests/fixtures/server.py +++ b/tests/fixtures/server.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Fixtures specifically for help testing the modules in the server/ directory. """ @@ -9,27 +15,29 @@ import pytest import yaml +from tests.fixture_types import FixtureCallable, FixtureDict, FixtureNamespace, FixtureStr + # pylint: disable=redefined-outer-name @pytest.fixture(scope="session") -def server_testing_dir(temp_output_dir: str) -> str: +def server_testing_dir(create_testing_dir: FixtureCallable, temp_output_dir: FixtureStr) -> FixtureStr: """ Fixture to create a temporary output directory for tests related to the server functionality. - :param temp_output_dir: The path to the temporary output directory we'll be using for this test run - :returns: The path to the temporary testing directory for server tests - """ - testing_dir = f"{temp_output_dir}/server_testing" - if not os.path.exists(testing_dir): - os.mkdir(testing_dir) + Args: + create_testing_dir: A fixture which returns a function that creates the testing directory. + temp_output_dir: The path to the temporary ouptut directory we'll be using for this test run. - return testing_dir + Returns: + The path to the temporary testing directory for server tests. + """ + return create_testing_dir(temp_output_dir, "server_testing") @pytest.fixture(scope="session") -def server_redis_conf_file(server_testing_dir: str) -> str: +def server_redis_conf_file(server_testing_dir: FixtureStr) -> FixtureStr: """ Fixture to write a redis.conf file to the temporary output directory. @@ -77,7 +85,7 @@ def server_redis_conf_file(server_testing_dir: str) -> str: @pytest.fixture(scope="session") -def server_redis_pass_file(server_testing_dir: str) -> str: +def server_redis_pass_file(server_testing_dir: FixtureStr) -> FixtureStr: """ Fixture to create a redis password file in the temporary output directory. @@ -96,7 +104,7 @@ def server_redis_pass_file(server_testing_dir: str) -> str: @pytest.fixture(scope="session") -def server_users() -> Dict[str, Dict[str, str]]: +def server_users() -> FixtureDict[str, Dict[str, str]]: """ Create a dictionary of two test users with identical configuration settings. @@ -122,7 +130,7 @@ def server_users() -> Dict[str, Dict[str, str]]: @pytest.fixture(scope="session") -def server_redis_users_file(server_testing_dir: str, server_users: dict) -> str: +def server_redis_users_file(server_testing_dir: FixtureStr, server_users: FixtureDict[str, Dict[str, str]]) -> FixtureStr: """ Fixture to write a redis.users file to the temporary output directory. @@ -143,11 +151,11 @@ def server_redis_users_file(server_testing_dir: str, server_users: dict) -> str: @pytest.fixture(scope="class") def server_container_config_data( - server_testing_dir: str, - server_redis_conf_file: str, - server_redis_pass_file: str, - server_redis_users_file: str, -) -> Dict[str, str]: + server_testing_dir: FixtureStr, + server_redis_conf_file: FixtureStr, + server_redis_pass_file: FixtureStr, + server_redis_users_file: FixtureStr, +) -> FixtureDict[str, str]: """ Fixture to provide sample data for ContainerConfig tests. @@ -172,7 +180,7 @@ def server_container_config_data( @pytest.fixture(scope="class") -def server_container_format_config_data() -> Dict[str, str]: +def server_container_format_config_data() -> FixtureDict[str, str]: """ Fixture to provide sample data for ContainerFormatConfig tests @@ -187,7 +195,7 @@ def server_container_format_config_data() -> Dict[str, str]: @pytest.fixture(scope="class") -def server_process_config_data() -> Dict[str, str]: +def server_process_config_data() -> FixtureDict[str, str]: """ Fixture to provide sample data for ProcessConfig tests @@ -201,10 +209,10 @@ def server_process_config_data() -> Dict[str, str]: @pytest.fixture(scope="class") def server_server_config( - server_container_config_data: Dict[str, str], - server_process_config_data: Dict[str, str], - server_container_format_config_data: Dict[str, str], -) -> Dict[str, Dict[str, str]]: + server_container_config_data: FixtureDict[str, str], + server_process_config_data: FixtureDict[str, str], + server_container_format_config_data: FixtureDict[str, str], +) -> FixtureDict[str, FixtureDict[str, str]]: """ Fixture to provide sample data for ServerConfig tests @@ -222,10 +230,10 @@ def server_server_config( @pytest.fixture(scope="function") def server_app_yaml_contents( - server_redis_pass_file: str, - server_container_config_data: Dict[str, str], - server_process_config_data: Dict[str, str], -) -> Dict[str, Union[str, int]]: + server_redis_pass_file: FixtureStr, + server_container_config_data: FixtureDict[str, str], + server_process_config_data: FixtureDict[str, str], +) -> FixtureDict[str, Union[str, int]]: """ Fixture to create the contents of an app.yaml file. @@ -260,7 +268,7 @@ def server_app_yaml_contents( @pytest.fixture(scope="function") -def server_app_yaml(server_testing_dir: str, server_app_yaml_contents: dict) -> str: +def server_app_yaml(server_testing_dir: FixtureStr, server_app_yaml_contents: FixtureDict[str, Union[str, int]]) -> FixtureStr: """ Fixture to create an app.yaml file in the temporary output directory. @@ -280,13 +288,13 @@ def server_app_yaml(server_testing_dir: str, server_app_yaml_contents: dict) -> @pytest.fixture(scope="function") -def server_process_file_contents() -> str: +def server_process_file_contents() -> FixtureDict[str, Union[str, int]]: """Fixture to represent process file contents.""" return {"parent_pid": 123, "image_pid": 456, "port": 6379, "hostname": "dummy_server"} @pytest.fixture(scope="function") -def server_config_server_args() -> Namespace: +def server_config_server_args() -> FixtureNamespace: """ Setup an argparse Namespace with all args that the `config_server` function will need. These can be modified on a test-by-test basis. diff --git a/tests/fixtures/status.py b/tests/fixtures/status.py index 39a36f9bf..3f67d1a40 100644 --- a/tests/fixtures/status.py +++ b/tests/fixtures/status.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Fixtures specifically for help testing the functionality related to status/detailed-status. @@ -11,6 +17,7 @@ import pytest import yaml +from tests.fixture_types import FixtureCallable, FixtureNamespace, FixtureStr from tests.unit.study.status_test_files import status_test_variables @@ -18,22 +25,22 @@ @pytest.fixture(scope="session") -def status_testing_dir(temp_output_dir: str) -> str: +def status_testing_dir(create_testing_dir: FixtureCallable, temp_output_dir: FixtureStr) -> FixtureStr: """ A pytest fixture to set up a temporary directory to write files to for testing status. - :param temp_output_dir: The path to the temporary output directory we'll be using for this test run - :returns: The path to the temporary testing directory for status testing - """ - testing_dir = f"{temp_output_dir}/status_testing/" - if not os.path.exists(testing_dir): - os.mkdir(testing_dir) + Args: + create_testing_dir: A fixture which returns a function that creates the testing directory. + temp_output_dir: The path to the temporary ouptut directory we'll be using for this test run. - return testing_dir + Returns: + The path to the temporary testing directory for status tests. + """ + return create_testing_dir(temp_output_dir, "status_testing") @pytest.fixture(scope="class") -def status_empty_file(status_testing_dir: str) -> str: +def status_empty_file(status_testing_dir: FixtureStr) -> FixtureStr: """ A pytest fixture to create an empty status file. @@ -49,7 +56,7 @@ def status_empty_file(status_testing_dir: str) -> str: @pytest.fixture(scope="session") -def status_spec_path(status_testing_dir: str) -> str: # pylint: disable=W0621 +def status_spec_path(status_testing_dir: FixtureStr) -> FixtureStr: # pylint: disable=W0621 """ Copy the test spec to the temp directory and modify the OUTPUT_PATH in the spec to point to the temp location. @@ -94,7 +101,7 @@ def set_sample_path(output_workspace: str): @pytest.fixture(scope="session") -def status_output_workspace(status_testing_dir: str) -> str: # pylint: disable=W0621 +def status_output_workspace(status_testing_dir: FixtureStr) -> FixtureStr: # pylint: disable=W0621 """ A pytest fixture to copy the test output workspace for status to the temporary status testing directory. @@ -110,7 +117,7 @@ def status_output_workspace(status_testing_dir: str) -> str: # pylint: disable= @pytest.fixture(scope="function") -def status_args(): +def status_args() -> FixtureNamespace: """ A pytest fixture to set up a namespace with all the arguments necessary for the Status object. @@ -130,7 +137,7 @@ def status_args(): @pytest.fixture(scope="session") -def status_nested_workspace(status_testing_dir: str) -> str: # pylint: disable=W0621 +def status_nested_workspace(status_testing_dir: FixtureStr) -> FixtureStr: # pylint: disable=W0621 """ Create an output workspace that contains another output workspace within one of its steps. In this case it will copy the status test workspace then within the 'just_samples' diff --git a/tests/fixtures/stores.py b/tests/fixtures/stores.py new file mode 100644 index 000000000..a3a410d89 --- /dev/null +++ b/tests/fixtures/stores.py @@ -0,0 +1,122 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Fixtures related to database stores. + +These can be used by any kind of store (e.g. Redis, SQLite). +""" + +from unittest.mock import MagicMock + +import pytest +from pytest_mock import MockerFixture +from redis import Redis + +from merlin.db_scripts.data_models import LogicalWorkerModel, PhysicalWorkerModel, RunModel, StudyModel +from tests.fixture_types import FixtureCallable, FixtureTuple + + +@pytest.fixture +def mock_redis(mocker): + """Create a mock Redis client.""" + redis_mock = mocker.MagicMock(spec=Redis) + return redis_mock + + +@pytest.fixture +def test_models(): + """Create test model instances for tests.""" + + # Sample study model + study = StudyModel( + id="study1", + name="Test Study", + runs=["run1"], + ) + + # Sample run model + run = RunModel( + id="run1", + study_id="study1", + workers=["lw1"], + ) + + # Sample logical worker model + logical_worker = LogicalWorkerModel( + id="lw1", + name="logical_worker", + queues=["queue1", "queue2"], + physical_workers=["pw1"], + runs=["run1"], + ) + + # Sample physical worker model + physical_worker = PhysicalWorkerModel( + id="pw1", + name="Worker 1", + logical_worker_id="lw1", + ) + + return {"study": study, "run": run, "logical_worker": logical_worker, "physical_worker": physical_worker} + + +@pytest.fixture(scope="session") +def create_redis_hash_data() -> FixtureCallable: + """ + Pytest fixture that provides a helper function to simulate Redis hash data + from a model object. + + Returns: + A function that takes an object (typically a model instance) and converts + its attributes into a dictionary formatted as Redis hash data, with all values + serialized as strings. + """ + + def _create_redis_hash_data(obj): + """Create a dict that simulates Redis hash data for a model.""" + # Simple conversion - in real code, this would be more complex + result = {} + for key, value in obj.__dict__.items(): + if isinstance(value, (str, int, float, bool, type(None))): + result[key] = str(value) + elif isinstance(value, dict): + result[key] = str(value) # In real code, this would be JSON + elif isinstance(value, list): + result[key] = str(value) # In real code, this would be JSON + return result + + return _create_redis_hash_data + + +@pytest.fixture +def mock_sqlite_connection(mocker: MockerFixture) -> FixtureTuple[MagicMock]: + """ + Create a mocked SQLiteConnection context manager. + + Args: + mocker: PyTest mocker fixture. + + Returns: + A tuple of (mock_connection, mock_cursor) for easy access in tests. + """ + # Create mock cursor + mock_cursor = mocker.MagicMock() + + # Create mock connection + mock_conn = mocker.MagicMock() + mock_conn.execute.return_value = mock_cursor + + # Create mock context manager + mock_context_manager = mocker.MagicMock() + mock_context_manager.__enter__.return_value = mock_conn + mock_context_manager.__exit__.return_value = None + + # Mock the SQLiteConnection class + mock_sqlite_conn = mocker.patch("merlin.backends.sqlite.sqlite_store_base.SQLiteConnection") + mock_sqlite_conn.return_value = mock_context_manager + + return mock_conn, mock_cursor diff --git a/tests/integration/__init__.py b/tests/integration/__init__.py index e69de29bb..3232b50b9 100644 --- a/tests/integration/__init__.py +++ b/tests/integration/__init__.py @@ -0,0 +1,5 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## diff --git a/tests/integration/commands/__init__.py b/tests/integration/commands/__init__.py new file mode 100644 index 000000000..3232b50b9 --- /dev/null +++ b/tests/integration/commands/__init__.py @@ -0,0 +1,5 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## diff --git a/tests/integration/commands/pgen.py b/tests/integration/commands/pgen.py new file mode 100644 index 000000000..b0dd89206 --- /dev/null +++ b/tests/integration/commands/pgen.py @@ -0,0 +1,42 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This file contains pgen functionality for testing purposes. +It's specifically set up to work with the feature demo example. +""" + +import random + +from maestrowf.datastructures.core import ParameterGenerator + + +# pylint complains about unused argument `env` but it's necessary for Maestro +def get_custom_generator(env, **kwargs): # pylint: disable=unused-argument + """ + Custom parameter generator that's used for testing the `--pgen` flag + of the `merlin run` command. + """ + p_gen = ParameterGenerator() + + # Unpack any pargs passed in + x2_min = int(kwargs.get("X2_MIN", "0")) + x2_max = int(kwargs.get("X2_MAX", "1")) + n_name_min = int(kwargs.get("N_NAME_MIN", "0")) + n_name_max = int(kwargs.get("N_NAME_MAX", "10")) + + # We'll only have two parameter entries each just for testing + num_points = 2 + + params = { + "X2": {"values": [random.uniform(x2_min, x2_max) for _ in range(num_points)], "label": "X2.%%"}, + "N_NEW": {"values": [random.randint(n_name_min, n_name_max) for _ in range(num_points)], "label": "N_NEW.%%"}, + } + + for key, value in params.items(): + p_gen.add_parameter(key, value["values"], value["label"]) + + return p_gen diff --git a/tests/integration/commands/test_monitor.py b/tests/integration/commands/test_monitor.py new file mode 100644 index 000000000..c3a413871 --- /dev/null +++ b/tests/integration/commands/test_monitor.py @@ -0,0 +1,120 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module will contain the testing logic for the `merlin monitor` command. +""" + +import subprocess +from time import sleep + +from tests.context_managers.celery_task_manager import CeleryTaskManager +from tests.context_managers.celery_workers_manager import CeleryWorkersManager +from tests.fixture_data_classes import MonitorSetup, RedisBrokerAndBackend +from tests.fixture_types import FixtureStr +from tests.integration.conditions import HasRegex, StepFileExists +from tests.integration.helper_funcs import check_test_conditions, copy_app_yaml_to_cwd + + +class TestMonitor: + """ + Tests for the `merlin monitor` command. + """ + + # TODO this is huge, can we split it up somehow? + def test_auto_restart( + self, monitor_setup: MonitorSetup, redis_broker_and_backend_class: RedisBrokerAndBackend, merlin_server_dir: FixtureStr + ): + """ + Test that the monitor automatically restarts the workflow when: + 1. There are no tasks in the queues + 2. There are no workers processing tasks + 3. The workflow has not yet finished + + This test is accomplished by: + 1. Sending tasks to the queues + 2. Starting workers so that they begin processing the workflow + 3. Starting the monitor so that it begins monitoring the workflow + 4. Purging the tasks so that there's nothing left in the queues and the workflow cannot finish + without a restart that the monitor must provide + + The result of this process will produce the necessary conditions for the monitor to + restart the workflow. + + Args: + monitor_setup: A fixture that returns a + [`MonitorSetup`][fixture_data_classes.MonitorSetup] instance. + redis_broker_and_backend_class: Fixture for setting up Redis broker and + backend for class-scoped tests. + merlin_server_dir: A fixture to provide the path to the merlin_server directory that will be + created by the [`redis_server`][conftest.redis_server] fixture. + """ + from merlin.celery import app as celery_app # pylint: disable=import-outside-toplevel + + # Need to copy app.yaml to cwd so we can connect to redis server + copy_app_yaml_to_cwd(merlin_server_dir) + + run_workers_proc = monitor_stdout = monitor_stderr = None + with CeleryTaskManager(celery_app, redis_broker_and_backend_class.client): + # Send the tasks to the server + try: + subprocess.run( + f"merlin run {monitor_setup.auto_restart_yaml} --vars OUTPUT_PATH={monitor_setup.testing_dir}", + shell=True, + text=True, + timeout=15, + ) + except subprocess.TimeoutExpired as exc: + raise TimeoutError("Could not send tasks to the server within the allotted time.") from exc + + # We use a context manager to start workers so that they'll safely stop even if this test fails + with CeleryWorkersManager(celery_app) as celery_worker_manager: + # Start the workers then add them to the context manager so they can be stopped safely later + # This worker will start processing the workflow but we don't want it to finish processing it + run_workers_proc = subprocess.Popen( # pylint: disable=consider-using-with + f"merlin run-workers {monitor_setup.auto_restart_yaml}".split(), + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + start_new_session=True, + ) + celery_worker_manager.add_run_workers_process(run_workers_proc.pid) + sleep(5) + + # Start the monitor and give it a 3 second sleep interval + monitor_proc = subprocess.Popen( # pylint: disable=consider-using-with + f"merlin monitor {monitor_setup.auto_restart_yaml} --sleep 3".split(), + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + start_new_session=True, + ) + + # Purge the tasks that are in the queues + purge_proc = subprocess.run( + f"merlin purge -f {monitor_setup.auto_restart_yaml}".split(), capture_output=True, text=True + ) + + monitor_stdout, monitor_stderr = monitor_proc.communicate() + + # Define our test conditions + study_name = "monitor_auto_restart_test" + conditions = [ + HasRegex("Purged 1 message from 2 known task queues."), + HasRegex("Monitor: Restarting workflow for run with workspace"), + HasRegex("Monitor: Workflow restarted successfully:"), + HasRegex("Monitor: Failed to restart workflow:", negate=True), + StepFileExists("process_samples", "MERLIN_FINISHED", study_name, monitor_setup.testing_dir, samples=True), + StepFileExists("funnel_step", "MERLIN_FINISHED", study_name, monitor_setup.testing_dir), + ] + + # Check our test conditions + info = { + "return_code": monitor_proc.returncode, + "stdout": monitor_stdout + purge_proc.stdout + run_workers_proc.stdout.read(), + "stderr": monitor_stderr + purge_proc.stderr + run_workers_proc.stderr.read(), + } + check_test_conditions(conditions, info) diff --git a/tests/integration/commands/test_purge.py b/tests/integration/commands/test_purge.py new file mode 100644 index 000000000..fba0e09e7 --- /dev/null +++ b/tests/integration/commands/test_purge.py @@ -0,0 +1,369 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module will contain the testing logic +for the `merlin purge` command. +""" + +import os +import subprocess +from typing import Dict, List, Tuple, Union + +from merlin.spec.expansion import get_spec_with_expansion +from tests.context_managers.celery_task_manager import CeleryTaskManager +from tests.fixture_data_classes import RedisBrokerAndBackend +from tests.fixture_types import FixtureRedis, FixtureStr +from tests.integration.conditions import HasRegex, HasReturnCode +from tests.integration.helper_funcs import check_test_conditions, copy_app_yaml_to_cwd + + +class TestPurgeCommand: + """ + Tests for the `merlin purge` command. + """ + + demo_workflow = os.path.join("examples", "workflows", "feature_demo", "feature_demo.yaml") + + def setup_test(self, path_to_merlin_codebase: FixtureStr, merlin_server_dir: FixtureStr) -> str: + """ + Setup the test environment for these tests by: + 1. Copying the app.yaml file created by the `redis_server` fixture to the cwd so that + Merlin can connect to the test server. + 2. Obtaining the path to the feature_demo spec that we'll use for these tests. + + Args: + path_to_merlin_codebase: + A fixture to provide the path to the directory containing Merlin's core + functionality. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + + Returns: + The path to the feature_demo spec file. + """ + copy_app_yaml_to_cwd(merlin_server_dir) + return os.path.join(path_to_merlin_codebase, self.demo_workflow) + + def setup_tasks(self, celery_task_manager: CeleryTaskManager, spec_file: str) -> Tuple[Dict[str, str], int]: + """ + Helper method to setup tasks in the specified queues. + + This method sends tasks named 'task_for_{queue}' to each queue defined in the + provided spec file and returns the total number of queues that received tasks. + + Args: + celery_task_manager: + A context manager for managing Celery tasks, used to send tasks to the server. + spec_file: + The path to the spec file from which queues will be extracted. + + Returns: + A tuple with: + - A dictionary where the keys are step names and values are their associated queues. + - The number of queues that received tasks + """ + spec = get_spec_with_expansion(spec_file) + queues_in_spec = spec.get_task_queues() + + for queue in queues_in_spec.values(): + celery_task_manager.send_task(f"task_for_{queue}", queue=queue) + + return queues_in_spec, len(queues_in_spec.values()) + + def run_purge( + self, + spec_file: str, + input_value: str = None, + force: bool = False, + steps_to_purge: List[str] = None, + ) -> Dict[str, Union[str, int]]: + """ + Helper method to run the purge command. + + Args: + spec_file: The path to the spec file from which queues will be purged. + input_value: Any input we need to send to the subprocess. + force: If True, add the `-f` option to the purge command. + steps_to_purge: An optional list of steps to send to the purge command. + + Returns: + The result from executing the command in a subprocess. + """ + purge_cmd = ( + "merlin purge" + + (" -f" if force else "") + + f" {spec_file}" + + (f" --steps {' '.join(steps_to_purge)}" if steps_to_purge is not None else "") + ) + result = subprocess.run(purge_cmd, shell=True, capture_output=True, text=True, input=input_value) + return { + "stdout": result.stdout, + "stderr": result.stderr, + "return_code": result.returncode, + } + + def check_queues( + self, + redis_client: FixtureRedis, + queues_in_spec: Dict[str, str], + expected_task_count: int, + steps_to_purge: List[str] = None, + ): + """ + Check the state of queues in Redis against expected task counts. + + When `steps_to_purge` is set, the `expected_task_count` will represent the + number of expected tasks in the queues that _are not_ associated with the + steps in the `steps_to_purge` list. + + Args: + redis_client: The Redis client instance. + queues_in_spec: A dictionary of queues to check. + expected_task_count: The expected number of tasks in the queues (0 or 1). + steps_to_purge: Optional list of steps to determine which queues should be purged. + """ + for queue in queues_in_spec.values(): + # Brackets are special chars in regex so we have to add \ to make them literal + queue = queue.replace("[", "\\[").replace("]", "\\]") + matching_queues_on_server = redis_client.keys(pattern=f"{queue}*") + + for matching_queue in matching_queues_on_server: + tasks = redis_client.lrange(matching_queue, 0, -1) + if steps_to_purge and matching_queue in [queues_in_spec[step] for step in steps_to_purge]: + assert len(tasks) == 0, f"Expected 0 tasks in {matching_queue}, found {len(tasks)}." + else: + assert ( + len(tasks) == expected_task_count + ), f"Expected {expected_task_count} tasks in {matching_queue}, found {len(tasks)}." + + def test_no_options_tasks_exist_y( + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + path_to_merlin_codebase: FixtureStr, + merlin_server_dir: FixtureStr, + ): + """ + Test the `merlin purge` command with no options added and + tasks sent to the server. This should come up with a y/N + prompt in which we type 'y'. This should then purge the + tasks from the server. + + Args: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + path_to_merlin_codebase: + A fixture to provide the path to the directory containing Merlin's core + functionality. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + """ + from merlin.celery import app as celery_app # pylint: disable=import-outside-toplevel + + feature_demo = self.setup_test(path_to_merlin_codebase, merlin_server_dir) + + with CeleryTaskManager(celery_app, redis_broker_and_backend_function.client) as celery_task_manager: + # Send tasks to the server for every queue in the spec + queues_in_spec, num_queues = self.setup_tasks(celery_task_manager, feature_demo) + + # Run the purge test + test_info = self.run_purge(feature_demo, input_value="y") + + # Make sure the subprocess ran and the correct output messages are given + conditions = [ + HasReturnCode(), + HasRegex("Are you sure you want to delete all tasks?"), + HasRegex(f"Purged {num_queues} messages from {num_queues} known task queues."), + ] + check_test_conditions(conditions, test_info) + + # Check on the Redis queues to ensure they were purged + self.check_queues(redis_broker_and_backend_function.client, queues_in_spec, expected_task_count=0) + + def test_no_options_no_tasks_y( + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + path_to_merlin_codebase: FixtureStr, + merlin_server_dir: FixtureStr, + ): + """ + Test the `merlin purge` command with no options added and + no tasks sent to the server. This should come up with a y/N + prompt in which we type 'y'. This should then give us a "No + messages purged" log. + + Args: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + path_to_merlin_codebase: + A fixture to provide the path to the directory containing Merlin's core + functionality. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + """ + from merlin.celery import app as celery_app # pylint: disable=import-outside-toplevel + + feature_demo = self.setup_test(path_to_merlin_codebase, merlin_server_dir) + + with CeleryTaskManager(celery_app, redis_broker_and_backend_function.client): + # Get the queues from the spec file + spec = get_spec_with_expansion(feature_demo) + queues_in_spec = spec.get_task_queues() + num_queues = len(queues_in_spec.values()) + + # Check that there are no tasks in the queues before we run the purge command + self.check_queues(redis_broker_and_backend_function.client, queues_in_spec, expected_task_count=0) + + # Run the purge test + test_info = self.run_purge(feature_demo, input_value="y") + + # Make sure the subprocess ran and the correct output messages are given + conditions = [ + HasReturnCode(), + HasRegex("Are you sure you want to delete all tasks?"), + HasRegex(f"No messages purged from {num_queues} queues."), + ] + check_test_conditions(conditions, test_info) + + # Check that the Redis server still has no tasks + self.check_queues(redis_broker_and_backend_function.client, queues_in_spec, expected_task_count=0) + + def test_no_options_n( + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + path_to_merlin_codebase: FixtureStr, + merlin_server_dir: FixtureStr, + ): + """ + Test the `merlin purge` command with no options added and + tasks sent to the server. This should come up with a y/N + prompt in which we type 'N'. This should take us out of the + command without purging the tasks. + + Args: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + path_to_merlin_codebase: + A fixture to provide the path to the directory containing Merlin's core + functionality. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + """ + from merlin.celery import app as celery_app # pylint: disable=import-outside-toplevel + + feature_demo = self.setup_test(path_to_merlin_codebase, merlin_server_dir) + + with CeleryTaskManager(celery_app, redis_broker_and_backend_function.client) as celery_task_manager: + # Send tasks to the server for every queue in the spec + queues_in_spec, num_queues = self.setup_tasks(celery_task_manager, feature_demo) + + # Run the purge test + test_info = self.run_purge(feature_demo, input_value="N") + + # Make sure the subprocess ran and the correct output messages are given + conditions = [ + HasReturnCode(), + HasRegex("Are you sure you want to delete all tasks?"), + HasRegex(f"Purged {num_queues} messages from {num_queues} known task queues.", negate=True), + ] + check_test_conditions(conditions, test_info) + + # Check on the Redis queues to ensure they were not purged + self.check_queues(redis_broker_and_backend_function.client, queues_in_spec, expected_task_count=1) + + def test_force_option( + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + path_to_merlin_codebase: FixtureStr, + merlin_server_dir: FixtureStr, + ): + """ + Test the `merlin purge` command with the `--force` option + enabled. This should not bring up a y/N prompt and should + immediately purge all tasks. + + Args: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + path_to_merlin_codebase: + A fixture to provide the path to the directory containing Merlin's core + functionality. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + """ + from merlin.celery import app as celery_app # pylint: disable=import-outside-toplevel + + feature_demo = self.setup_test(path_to_merlin_codebase, merlin_server_dir) + + with CeleryTaskManager(celery_app, redis_broker_and_backend_function.client) as celery_task_manager: + # Send tasks to the server for every queue in the spec + queues_in_spec, num_queues = self.setup_tasks(celery_task_manager, feature_demo) + + # Run the purge test + test_info = self.run_purge(feature_demo, force=True) + + # Make sure the subprocess ran and the correct output messages are given + conditions = [ + HasReturnCode(), + HasRegex("Are you sure you want to delete all tasks?", negate=True), + HasRegex(f"Purged {num_queues} messages from {num_queues} known task queues."), + ] + check_test_conditions(conditions, test_info) + + # Check on the Redis queues to ensure they were purged + self.check_queues(redis_broker_and_backend_function.client, queues_in_spec, expected_task_count=0) + + def test_steps_option( + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + path_to_merlin_codebase: FixtureStr, + merlin_server_dir: FixtureStr, + ): + """ + Test the `merlin purge` command with the `--steps` option + enabled. This should only purge the tasks in the task queues + associated with the steps provided. + + Args: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + path_to_merlin_codebase: + A fixture to provide the path to the directory containing Merlin's core + functionality. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + """ + from merlin.celery import app as celery_app # pylint: disable=import-outside-toplevel + + feature_demo = self.setup_test(path_to_merlin_codebase, merlin_server_dir) + + with CeleryTaskManager(celery_app, redis_broker_and_backend_function.client) as celery_task_manager: + # Send tasks to the server for every queue in the spec + queues_in_spec, _ = self.setup_tasks(celery_task_manager, feature_demo) + + # Run the purge test + steps_to_purge = ["hello", "collect"] + test_info = self.run_purge(feature_demo, input_value="y", steps_to_purge=steps_to_purge) + + # Make sure the subprocess ran and the correct output messages are given + num_steps_to_purge = len(steps_to_purge) + conditions = [ + HasReturnCode(), + HasRegex("Are you sure you want to delete all tasks?"), + HasRegex(f"Purged {num_steps_to_purge} messages from {num_steps_to_purge} known task queues."), + ] + check_test_conditions(conditions, test_info) + + # Check on the Redis queues to ensure they were not purged + self.check_queues( + redis_broker_and_backend_function.client, queues_in_spec, expected_task_count=1, steps_to_purge=steps_to_purge + ) diff --git a/tests/integration/commands/test_run.py b/tests/integration/commands/test_run.py new file mode 100644 index 000000000..d35894bcc --- /dev/null +++ b/tests/integration/commands/test_run.py @@ -0,0 +1,474 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module will contain the testing logic +for the `merlin run` command. +""" + +import csv +import os +import re +import shutil +import subprocess +from typing import Dict, Union + +from merlin.spec.expansion import get_spec_with_expansion +from tests.context_managers.celery_task_manager import CeleryTaskManager +from tests.fixture_data_classes import RedisBrokerAndBackend +from tests.fixture_types import FixtureStr +from tests.integration.conditions import HasReturnCode, PathExists, StepFinishedFilesCount +from tests.integration.helper_funcs import check_test_conditions, copy_app_yaml_to_cwd + + +# pylint: disable=import-outside-toplevel,unused-argument + + +class TestRunCommand: + """ + Base class for testing the `merlin run` command. + """ + + demo_workflow = os.path.join("examples", "workflows", "feature_demo", "feature_demo.yaml") + + def setup_test_environment( + self, path_to_merlin_codebase: FixtureStr, merlin_server_dir: FixtureStr, run_command_testing_dir: FixtureStr + ) -> str: + """ + Setup the test environment for these tests by: + 1. Moving into the temporary output directory created specifically for these tests. + 2. Copying the app.yaml file created by the `redis_server` fixture to the cwd so that + Merlin can connect to the test server. + 3. Obtaining the path to the feature_demo spec that we'll use for these tests. + + Args: + path_to_merlin_codebase: + A fixture to provide the path to the directory containing Merlin's core + functionality. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + run_command_testing_dir: + The path to the the temp output directory for `merlin run` tests. + + Returns: + The path to the feature_demo spec file. + """ + os.chdir(run_command_testing_dir) + copy_app_yaml_to_cwd(merlin_server_dir) + return os.path.join(path_to_merlin_codebase, self.demo_workflow) + + def run_merlin_command(self, command: str) -> Dict[str, Union[str, int]]: + """ + Open a subprocess and run the command specified by the `command` parameter. + Ensure this command runs successfully and return the process results. + + Args: + command: The command to execute in a subprocess. + + Returns: + The results from executing the command in a subprocess. + + Raises: + AssertionError: If the command fails (non-zero return code). + """ + result = subprocess.run(command, shell=True, capture_output=True, text=True) + return { + "stdout": result.stdout, + "stderr": result.stderr, + "return_code": result.returncode, + } + + def get_output_workspace_from_logs(self, test_info: Dict[str, Union[str, int]]) -> str: + """ + Extracts the workspace path from the provided standard output and error logs. + + This method searches for a specific message indicating the study workspace + in the combined logs (both stdout and stderr). The expected message format + is: "Study workspace is ''". If the message is found, + the method returns the extracted workspace path. If the message is not + found, an assertion error is raised. + + Args: + test_info: The results from executing our test. + + Returns: + The extracted workspace path from the logs. + + Raises: + AssertionError: If the expected message is not found in the combined logs. + """ + workspace_pattern = re.compile(r"Study workspace is '(\S+)'") + combined_output = test_info["stdout"] + test_info["stderr"] + match = workspace_pattern.search(combined_output) + assert match, "No 'Study workspace is...' message found in command output." + return match.group(1) + + +class TestRunCommandDistributed(TestRunCommand): + """ + Tests for the `merlin run` command that are run in a distributed manner + rather than being run locally. + """ + + def test_distributed_run( + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + path_to_merlin_codebase: FixtureStr, + merlin_server_dir: FixtureStr, + run_command_testing_dir: FixtureStr, + ): + """ + This test verifies that tasks can be successfully sent to a Redis server + using the `merlin run` command with no flags. + + Args: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + path_to_merlin_codebase: + A fixture to provide the path to the directory containing Merlin's core + functionality. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + run_command_testing_dir: + The path to the the temp output directory for `merlin run` tests. + """ + from merlin.celery import app as celery_app + + # Setup the testing environment + feature_demo = self.setup_test_environment(path_to_merlin_codebase, merlin_server_dir, run_command_testing_dir) + + with CeleryTaskManager(celery_app, redis_broker_and_backend_function.client): + # Send tasks to the server + test_info = self.run_merlin_command(f"merlin run {feature_demo} --vars NAME=run_command_test_distributed_run") + + # Check that the test ran properly + check_test_conditions([HasReturnCode()], test_info) + + # Get the queues we need to query + spec = get_spec_with_expansion(feature_demo) + queues_in_spec = spec.get_task_queues() + + for queue in queues_in_spec.values(): + # Brackets are special chars in regex so we have to add \ to make them literal + queue = queue.replace("[", "\\[").replace("]", "\\]") + matching_queues_on_server = redis_broker_and_backend_function.client.keys(pattern=f"{queue}*") + + # Make sure any queues that exist on the server have tasks in them + for matching_queue in matching_queues_on_server: + tasks = redis_broker_and_backend_function.client.lrange(matching_queue, 0, -1) + assert len(tasks) > 0 + + def test_samplesfile_option( + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + path_to_merlin_codebase: FixtureStr, + merlin_server_dir: FixtureStr, + run_command_testing_dir: FixtureStr, + ): + """ + This test verifies that passing in a samples filepath from the command line will + substitute in the file properly. It should copy the samples file that's passed + in to the merlin_info subdirectory. + + Args: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + path_to_merlin_codebase: + A fixture to provide the path to the directory containing Merlin's core + functionality. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + run_command_testing_dir: + The path to the the temp output directory for `merlin run` tests. + """ + from merlin.celery import app as celery_app + + # Setup the testing environment + feature_demo = self.setup_test_environment(path_to_merlin_codebase, merlin_server_dir, run_command_testing_dir) + + # Create a new samples file to pass into our test workflow + data = [ + ["X1, Value 1", "X2, Value 1"], + ["X1, Value 2", "X2, Value 2"], + ["X1, Value 3", "X2, Value 3"], + ] + sample_filename = "test_samplesfile.csv" + new_samples_file = os.path.join(run_command_testing_dir, sample_filename) + with open(new_samples_file, mode="w", newline="") as file: + writer = csv.writer(file) + writer.writerows(data) + + with CeleryTaskManager(celery_app, redis_broker_and_backend_function.client): + # Send tasks to the server + test_info = self.run_merlin_command( + f"merlin run {feature_demo} --vars NAME=run_command_test_samplesfile_option --samplesfile {new_samples_file}" + ) + + # Check that the test ran properly and created the correct directories/files + expected_workspace_path = self.get_output_workspace_from_logs(test_info) + conditions = [ + HasReturnCode(), + PathExists(expected_workspace_path), + PathExists(os.path.join(expected_workspace_path, "merlin_info", sample_filename)), + ] + check_test_conditions(conditions, test_info) + + def test_pgen_and_pargs_options( # pylint: disable=too-many-locals + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + path_to_merlin_codebase: FixtureStr, + merlin_server_dir: FixtureStr, + run_command_testing_dir: FixtureStr, + ): + """ + Test the `--pgen` and `--pargs` options with the `merlin run` command. + This should update the parameter block of the expanded yaml file to have + 2 entries for both `X2` and `N_NEW`. The `X2` parameter should be between + `X2_MIN` and `X2_MAX`, and the `N_NEW` parameter should be between `N_NEW_MIN` + and `N_NEW_MAX`. + + Args: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + path_to_merlin_codebase: + A fixture to provide the path to the directory containing Merlin's core + functionality. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + run_command_testing_dir: + The path to the the temp output directory for `merlin run` tests. + """ + from merlin.celery import app as celery_app + + # Setup test vars and the testing environment + bounds = {"X2": (1, 2), "N_NEW": (5, 15)} + pgen_filepath = os.path.join( + os.path.abspath(os.path.expandvars(os.path.expanduser(os.path.dirname(__file__)))), "pgen.py" + ) + feature_demo = self.setup_test_environment(path_to_merlin_codebase, merlin_server_dir, run_command_testing_dir) + + with CeleryTaskManager(celery_app, redis_broker_and_backend_function.client): + # Send tasks to the server + test_info = self.run_merlin_command( + f"merlin run {feature_demo} " + "--vars NAME=run_command_test_pgen_and_pargs_options " + f"--pgen {pgen_filepath} " + f'--parg "X2_MIN:{bounds["X2"][0]}" ' + f'--parg "X2_MAX:{bounds["X2"][1]}" ' + f'--parg "N_NAME_MIN:{bounds["N_NEW"][0]}" ' + f'--parg "N_NAME_MAX:{bounds["N_NEW"][1]}"' + ) + + # Check that the test ran properly and created the correct directories/files + expected_workspace_path = self.get_output_workspace_from_logs(test_info) + expanded_yaml = os.path.join(expected_workspace_path, "merlin_info", "feature_demo.expanded.yaml") + conditions = [HasReturnCode(), PathExists(expected_workspace_path), PathExists(os.path.join(expanded_yaml))] + check_test_conditions(conditions, test_info) + + # Read in the parameters from the expanded yaml and ensure they're within the new bounds we provided + params = get_spec_with_expansion(expanded_yaml).get_parameters() + for param_name, (min_val, max_val) in bounds.items(): + for param in params.parameters[param_name]: + assert min_val <= param <= max_val + + +class TestRunCommandLocal(TestRunCommand): + """ + Tests for the `merlin run` command that are run in a locally rather + than in a distributed manner. + """ + + def test_dry_run( # pylint: disable=too-many-locals + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + path_to_merlin_codebase: FixtureStr, + merlin_server_dir: FixtureStr, + run_command_testing_dir: FixtureStr, + ): + """ + Test the `merlin run` command's `--dry` option. This should create all the output + subdirectories for each step but it shouldn't execute anything for the steps. In + other words, the only file in each step subdirectory should be the .sh file. + + Note: + This test will run locally so that we don't have to worry about starting + & stopping workers. + + Args: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + path_to_merlin_codebase: + A fixture to provide the path to the directory containing Merlin's core + functionality. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + run_command_testing_dir: + The path to the the temp output directory for `merlin run` tests. + """ + # Setup the test environment + feature_demo = self.setup_test_environment(path_to_merlin_codebase, merlin_server_dir, run_command_testing_dir) + + # Run the test and grab the output workspace generated from it + test_info = self.run_merlin_command(f"merlin run {feature_demo} --vars NAME=run_command_test_dry_run --local --dry") + + # Check that the test ran properly and created the correct directories/files + expected_workspace_path = self.get_output_workspace_from_logs(test_info) + check_test_conditions([HasReturnCode(), PathExists(expected_workspace_path)], test_info) + + # Check that every step was ran by looking for an existing output workspace + for step in get_spec_with_expansion(feature_demo).get_study_steps(): + step_directory = os.path.join(expected_workspace_path, step.name) + assert os.path.exists(step_directory), f"Output directory for step '{step.name}' not found: {step_directory}" + + allowed_dry_run_files = {"MERLIN_STATUS.json", "status.lock"} + for dirpath, dirnames, filenames in os.walk(step_directory): + # Check if the current directory has no subdirectories (leaf directory) + if not dirnames: + # Check for unexpected files + unexpected_files = [ + file for file in filenames if file not in allowed_dry_run_files and not file.endswith(".sh") + ] + assert not unexpected_files, ( + f"Unexpected files found in {dirpath}: {unexpected_files}. " + f"Expected only .sh files or {allowed_dry_run_files}." + ) + + # Check that there is exactly one .sh file + sh_file_count = sum(1 for file in filenames if file.endswith(".sh")) + assert ( + sh_file_count == 1 + ), f"Expected exactly one .sh file in {dirpath} but found {sh_file_count} .sh files." + + def test_local_run( + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + path_to_merlin_codebase: FixtureStr, + merlin_server_dir: FixtureStr, + run_command_testing_dir: FixtureStr, + ): + """ + This test verifies that tasks can be successfully executed locally using + the `merlin run` command with the `--local` flag. + + Args: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + path_to_merlin_codebase: + A fixture to provide the path to the directory containing Merlin's core + functionality. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + run_command_testing_dir: + The path to the the temp output directory for `merlin run` tests. + """ + # Setup the test environment + feature_demo = self.setup_test_environment(path_to_merlin_codebase, merlin_server_dir, run_command_testing_dir) + + # Run the test and grab the output workspace generated from it + study_name = "run_command_test_local_run" + num_samples = 8 + vars_dict = {"NAME": study_name, "OUTPUT_PATH": run_command_testing_dir, "N_SAMPLES": num_samples} + vars_str = " ".join(f"{key}={value}" for key, value in vars_dict.items()) + command = f"merlin run {feature_demo} --vars {vars_str} --local" + test_info = self.run_merlin_command(command) + + # Check that the test ran properly and created the correct directories/files + expected_workspace_path = self.get_output_workspace_from_logs(test_info) + conditions = [ + HasReturnCode(), + PathExists(expected_workspace_path), + StepFinishedFilesCount( # The rest of the conditions will ensure every step ran to completion + step="hello", + study_name=study_name, + output_path=run_command_testing_dir, + num_parameters=1, + num_samples=num_samples, + ), + StepFinishedFilesCount( + step="python3_hello", + study_name=study_name, + output_path=run_command_testing_dir, + num_parameters=1, + num_samples=0, + ), + StepFinishedFilesCount( + step="collect", + study_name=study_name, + output_path=run_command_testing_dir, + num_parameters=1, + num_samples=0, + ), + StepFinishedFilesCount( + step="translate", + study_name=study_name, + output_path=run_command_testing_dir, + num_parameters=1, + num_samples=0, + ), + StepFinishedFilesCount( + step="learn", + study_name=study_name, + output_path=run_command_testing_dir, + num_parameters=1, + num_samples=0, + ), + StepFinishedFilesCount( + step="make_new_samples", + study_name=study_name, + output_path=run_command_testing_dir, + num_parameters=1, + num_samples=0, + ), + StepFinishedFilesCount( + step="predict", + study_name=study_name, + output_path=run_command_testing_dir, + num_parameters=1, + num_samples=0, + ), + StepFinishedFilesCount( + step="verify", + study_name=study_name, + output_path=run_command_testing_dir, + num_parameters=1, + num_samples=0, + ), + ] + + # GitHub actions doesn't have a python2 path so we'll conditionally add this check + if shutil.which("python2"): + conditions.append( + StepFinishedFilesCount( + step="python2_hello", + study_name=study_name, + output_path=run_command_testing_dir, + num_parameters=1, + num_samples=0, + ) + ) + + check_test_conditions(conditions, test_info) + + # # Check that every step was ran by looking for an existing output workspace and MERLIN_FINISHED files + # for step in get_spec_with_expansion(feature_demo).get_study_steps(): + # step_directory = os.path.join(expected_workspace_path, step.name) + # assert os.path.exists(step_directory), f"Output directory for step '{step.name}' not found: {step_directory}" + # for dirpath, dirnames, filenames in os.walk(step_directory): + # # Check if the current directory has no subdirectories (leaf directory) + # if not dirnames: + # # Check for the existence of the MERLIN_FINISHED file + # assert ( + # "MERLIN_FINISHED" in filenames + # ), f"Expected a MERLIN_FINISHED file in list of files for {dirpath} but did not find one" + + +# pylint: enable=import-outside-toplevel,unused-argument diff --git a/tests/integration/commands/test_stop_and_query_workers.py b/tests/integration/commands/test_stop_and_query_workers.py new file mode 100644 index 000000000..a2f5d3376 --- /dev/null +++ b/tests/integration/commands/test_stop_and_query_workers.py @@ -0,0 +1,386 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module will contain the testing logic for +the `stop-workers` and `query-workers` commands. +""" + +import os +import subprocess +from contextlib import contextmanager +from enum import Enum +from typing import List + +import pytest + +from tests.context_managers.celery_workers_manager import CeleryWorkersManager +from tests.fixture_data_classes import RedisBrokerAndBackend +from tests.fixture_types import FixtureStr +from tests.integration.conditions import Condition, HasRegex +from tests.integration.helper_funcs import check_test_conditions, copy_app_yaml_to_cwd, load_workers_from_spec + + +# pylint: disable=unused-argument,import-outside-toplevel + + +class WorkerMessages(Enum): + """ + Enumerated strings to help keep track of the messages + that we're expecting (or not expecting) to see from the + tests in this module. + """ + + NO_WORKERS_MSG_STOP = "No workers found to stop" + NO_WORKERS_MSG_QUERY = "No workers found!" + STEP_1_WORKER = "step_1_merlin_test_worker" + STEP_2_WORKER = "step_2_merlin_test_worker" + OTHER_WORKER = "other_merlin_test_worker" + + +class TestStopAndQueryWorkersCommands: + """ + Tests for the `merlin stop-workers` and `merlin query-workers` commands. + Most of these tests will: + 1. Start workers from a spec file used for testing + - Use CeleryWorkerManager for this to ensure safe stoppage of workers + if something goes wrong + 2. Run the test command from a subprocess + """ + + @contextmanager + def run_test_with_workers( # pylint: disable=too-many-arguments + self, + path_to_test_specs: FixtureStr, + merlin_server_dir: FixtureStr, + conditions: List[Condition], + command: str, + flag: str = None, + ): + """ + Helper method to run common testing logic for tests with workers started. + This method must also be a context manager so we can check the status of the + workers prior to the CeleryWorkersManager running it's exit code that shuts down + all active workers. + + This method will: + 0. Read in the necessary fixtures as parameters. These fixtures grab paths to + our test specs and the merlin server directory created from starting the + containerized redis server. + 1. Load in the worker specifications from the `multiple_workers.yaml` file. + 2. Use a context manager to start up the workers on the celery app connected to + the containerized redis server + 3. Copy the app.yaml file for the containerized redis server to the current working + directory so that merlin will connect to it when we run our test + 4. Run the test command that's provided and check that the conditions given are + passing. + 5. Yield control back to the calling method. + 6. Safely terminate workers that may have not been stopped once the calling method + completes. + + Parameters: + path_to_test_specs: + A fixture to provide the path to the directory containing test specifications. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + conditions: + A list of `Condition` instances that need to pass in order for this test to + be successful. + command: + The command that we're testing. E.g. "merlin stop-workers" + flag: + An optional flag to add to the command that we're testing so we can test + different functionality for the command. + """ + from merlin.celery import app as celery_app + + # Grab worker configurations from the spec file + multiple_worker_spec = os.path.join(path_to_test_specs, "multiple_workers.yaml") + workers_from_spec = load_workers_from_spec(multiple_worker_spec) + + # We use a context manager to start workers so that they'll safely stop even if this test fails + with CeleryWorkersManager(celery_app) as workers_manager: + workers_manager.launch_workers(workers_from_spec) + + # Copy the app.yaml to the cwd so merlin will connect to the testing server + copy_app_yaml_to_cwd(merlin_server_dir) + + # Run the test + cmd_to_test = f"{command} {flag}" if flag else command + result = subprocess.run(cmd_to_test, capture_output=True, text=True, shell=True) + + info = { + "stdout": result.stdout, + "stderr": result.stderr, + "return_code": result.returncode, + } + + # Ensure all test conditions are satisfied + check_test_conditions(conditions, info) + + yield + + def get_no_workers_msg(self, command_to_test: str) -> WorkerMessages: + """ + Retrieve the appropriate "no workers" found message. + + This method checks the command to test and returns a corresponding + message based on whether the command is to stop workers or query for them. + + Returns: + The message indicating that no workers are available, depending on the + command being tested. + """ + no_workers_msg = None + if command_to_test == "merlin stop-workers": + no_workers_msg = WorkerMessages.NO_WORKERS_MSG_STOP.value + else: + no_workers_msg = WorkerMessages.NO_WORKERS_MSG_QUERY.value + return no_workers_msg + + @pytest.mark.parametrize("command_to_test", ["merlin stop-workers", "merlin query-workers"]) + def test_no_workers( + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + merlin_server_dir: FixtureStr, + command_to_test: str, + ): + """ + Test the `merlin stop-workers` and `merlin query-workers` commands with no workers + started in the first place. + + This test will: + 0. Setup the pytest fixtures which include: + - starting a containerized Redis server + - updating the CONFIG object to point to the containerized Redis server + - obtaining the path to the merlin server directory created from starting + the containerized Redis server + 1. Copy the app.yaml file for the containerized redis server to the current working + directory so that merlin will connect to it when we run our test + 2. Run the test command that's provided and check that the conditions given are + passing. + + Parameters: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + command_to_test: + The command that we're testing, obtained from the parametrize call. + """ + conditions = [ + HasRegex(self.get_no_workers_msg(command_to_test)), + HasRegex(WorkerMessages.STEP_1_WORKER.value, negate=True), + HasRegex(WorkerMessages.STEP_2_WORKER.value, negate=True), + HasRegex(WorkerMessages.OTHER_WORKER.value, negate=True), + ] + + # Copy the app.yaml to the cwd so merlin will connect to the testing server + copy_app_yaml_to_cwd(merlin_server_dir) + + # Run the test + result = subprocess.run(command_to_test, capture_output=True, text=True, shell=True) + info = { + "stdout": result.stdout, + "stderr": result.stderr, + "return_code": result.returncode, + } + + # Ensure all test conditions are satisfied + check_test_conditions(conditions, info) + + @pytest.mark.parametrize("command_to_test", ["merlin stop-workers", "merlin query-workers"]) + def test_no_flags( + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + path_to_test_specs: FixtureStr, + merlin_server_dir: FixtureStr, + command_to_test: str, + ): + """ + Test the `merlin stop-workers` and `merlin query-workers` commands with no flags. + + Run the commands referenced above and ensure the text output from Merlin is correct. + For the `stop-workers` command, we check if all workers are stopped as well. + To see more information on exactly what this test is doing, see the + `run_test_with_workers()` method. + + Parameters: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + path_to_test_specs: + A fixture to provide the path to the directory containing test specifications. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + command_to_test: + The command that we're testing, obtained from the parametrize call. + """ + conditions = [ + HasRegex(self.get_no_workers_msg(command_to_test), negate=True), + HasRegex(WorkerMessages.STEP_1_WORKER.value), + HasRegex(WorkerMessages.STEP_2_WORKER.value), + HasRegex(WorkerMessages.OTHER_WORKER.value), + ] + with self.run_test_with_workers(path_to_test_specs, merlin_server_dir, conditions, command_to_test): + if command_to_test == "merlin stop-workers": + # After the test runs and before the CeleryWorkersManager exits, ensure there are no workers on the app + from merlin.celery import app as celery_app + + active_queues = celery_app.control.inspect().active_queues() + assert active_queues is None + + @pytest.mark.parametrize("command_to_test", ["merlin stop-workers", "merlin query-workers"]) + def test_spec_flag( + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + path_to_test_specs: FixtureStr, + merlin_server_dir: FixtureStr, + command_to_test: str, + ): + """ + Test the `merlin stop-workers` and `merlin query-workers` commands with the `--spec` + flag. + + Run the commands referenced above with the `--spec` flag and ensure the text output + from Merlin is correct. For the `stop-workers` command, we check if all workers defined + in the spec file are stopped as well. To see more information on exactly what this test + is doing, see the `run_test_with_workers()` method. + + Parameters: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + path_to_test_specs: + A fixture to provide the path to the directory containing test specifications. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + command_to_test: + The command that we're testing, obtained from the parametrize call. + """ + conditions = [ + HasRegex(self.get_no_workers_msg(command_to_test), negate=True), + HasRegex(WorkerMessages.STEP_1_WORKER.value), + HasRegex(WorkerMessages.STEP_2_WORKER.value), + HasRegex(WorkerMessages.OTHER_WORKER.value), + ] + with self.run_test_with_workers( + path_to_test_specs, + merlin_server_dir, + conditions, + command_to_test, + flag=f"--spec {os.path.join(path_to_test_specs, 'multiple_workers.yaml')}", + ): + if command_to_test == "merlin stop-workers": + from merlin.celery import app as celery_app + + active_queues = celery_app.control.inspect().active_queues() + assert active_queues is None + + @pytest.mark.parametrize("command_to_test", ["merlin stop-workers", "merlin query-workers"]) + def test_workers_flag( + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + path_to_test_specs: FixtureStr, + merlin_server_dir: FixtureStr, + command_to_test: str, + ): + """ + Test the `merlin stop-workers` and `merlin query-workers` commands with the `--workers` + flag. + + Run the commands referenced above with the `--workers` flag and ensure the text output + from Merlin is correct. For the `stop-workers` command, we check to make sure that all + workers given with this flag are stopped. To see more information on exactly what this + test is doing, see the `run_test_with_workers()` method. + + Parameters: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + path_to_test_specs: + A fixture to provide the path to the directory containing test specifications. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + command_to_test: + The command that we're testing, obtained from the parametrize call. + """ + conditions = [ + HasRegex(self.get_no_workers_msg(command_to_test), negate=True), + HasRegex(WorkerMessages.STEP_1_WORKER.value), + HasRegex(WorkerMessages.STEP_2_WORKER.value), + HasRegex(WorkerMessages.OTHER_WORKER.value, negate=True), + ] + with self.run_test_with_workers( + path_to_test_specs, + merlin_server_dir, + conditions, + command_to_test, + flag=f"--workers {WorkerMessages.STEP_1_WORKER.value} {WorkerMessages.STEP_2_WORKER.value}", + ): + if command_to_test == "merlin stop-workers": + from merlin.celery import app as celery_app + + active_queues = celery_app.control.inspect().active_queues() + worker_name = f"celery@{WorkerMessages.OTHER_WORKER.value}" + assert worker_name in active_queues + + @pytest.mark.parametrize("command_to_test", ["merlin stop-workers", "merlin query-workers"]) + def test_queues_flag( + self, + redis_broker_and_backend_function: RedisBrokerAndBackend, + path_to_test_specs: FixtureStr, + merlin_server_dir: FixtureStr, + command_to_test: str, + ): + """ + Test the `merlin stop-workers` and `merlin query-workers` commands with the `--queues` + flag. + + Run the commands referenced above with the `--queues` flag and ensure the text output + from Merlin is correct. For the `stop-workers` command, we check that only the workers + attached to the given queues are stopped. To see more information on exactly what this + test is doing, see the `run_test_with_workers()` method. + + Parameters: + redis_broker_and_backend_function: Fixture for setting up Redis broker and + backend for function-scoped tests. + path_to_test_specs: + A fixture to provide the path to the directory containing test specifications. + merlin_server_dir: + A fixture to provide the path to the merlin_server directory that will be + created by the `redis_server` fixture. + command_to_test: + The command that we're testing, obtained from the parametrize call. + """ + conditions = [ + HasRegex(self.get_no_workers_msg(command_to_test), negate=True), + HasRegex(WorkerMessages.STEP_1_WORKER.value), + HasRegex(WorkerMessages.STEP_2_WORKER.value, negate=True), + HasRegex(WorkerMessages.OTHER_WORKER.value, negate=True), + ] + with self.run_test_with_workers( + path_to_test_specs, + merlin_server_dir, + conditions, + command_to_test, + flag="--queues hello_queue", + ): + if command_to_test == "merlin stop-workers": + from merlin.celery import app as celery_app + + active_queues = celery_app.control.inspect().active_queues() + workers_that_should_be_alive = [ + f"celery@{WorkerMessages.OTHER_WORKER.value}", + f"celery@{WorkerMessages.STEP_2_WORKER.value}", + ] + for worker_name in workers_that_should_be_alive: + assert worker_name in active_queues + + +# pylint: enable=unused-argument,import-outside-toplevel diff --git a/tests/integration/conditions.py b/tests/integration/conditions.py index 1c852b064..a46fd6a8f 100644 --- a/tests/integration/conditions.py +++ b/tests/integration/conditions.py @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """This module defines the different conditions to test against.""" import os from abc import ABC, abstractmethod @@ -34,7 +11,6 @@ from re import search -# TODO when moving command line tests to pytest, change Condition boolean returns to assertions class Condition(ABC): """Abstract Condition class that other conditions will inherit from""" @@ -106,6 +82,9 @@ def __str__(self): return f"{__class__.__name__} expected no '{self.regex}' regex match, but match was found." return f"{__class__.__name__} expected '{self.regex}' regex match, but match was not found." + def __repr__(self): + return f"HasRegex(regex={self.regex}, negate={self.negate})" + def is_within(self, text): """ :param `text`: text in which to search for a regex match @@ -131,7 +110,7 @@ def __init__(self, study_name, output_path): """ self.study_name = study_name self.output_path = output_path - self.dirpath_glob = f"{self.output_path}/{self.study_name}" f"_[0-9]*-[0-9]*" + self.dirpath_glob = os.path.join(self.output_path, f"{self.study_name}_[0-9]*-[0-9]*") def glob(self, glob_string): """ @@ -154,7 +133,7 @@ class StepFileExists(StudyOutputAware): A StudyOutputAware that checks for a particular file's existence. """ - def __init__(self, step, filename, study_name, output_path, params=False): # pylint: disable=R0913 + def __init__(self, step, filename, study_name, output_path, params=False, samples=False): # pylint: disable=R0913 """ :param `step`: the name of a step :param `filename`: name of file to search for in step's workspace directory @@ -165,6 +144,7 @@ def __init__(self, step, filename, study_name, output_path, params=False): # py self.step = step self.filename = filename self.params = params + self.samples = samples def __str__(self): return f"{__class__.__name__} expected to find file '{self.glob_string}', but file did not exist" @@ -174,10 +154,9 @@ def glob_string(self): """ Returns a regex string for the glob library to recursively find files with. """ - param_glob = "" - if self.params: - param_glob = "*/" - return f"{self.dirpath_glob}/{self.step}/{param_glob}{self.filename}" + param_glob = "*" if self.params else "" + samples_glob = "**" if self.samples else "" + return os.path.join(self.dirpath_glob, self.step, param_glob, samples_glob, self.filename) def file_exists(self): """Check if the file path created by glob_string exists""" @@ -229,7 +208,7 @@ def contains(self): with open(filename, "r") as textfile: filetext = textfile.read() return self.is_within(filetext) - except Exception: # pylint: disable=W0718 + except Exception: # pylint: disable=broad-except return False def is_within(self, text): @@ -243,6 +222,108 @@ def passes(self): return self.contains() +# TODO when writing API docs for tests make sure this looks correct and has functioning links +# - Do we want to list expected_count, glob_string, and passes as methods since they're already attributes? +class StepFinishedFilesCount(StudyOutputAware): + """ + A [`StudyOutputAware`][integration.conditions.StudyOutputAware] that checks for the + exact number of `MERLIN_FINISHED` files in a specified step's output directory based + on the number of parameters and samples. + + Attributes: + step: The name of the step to check. + study_name: The name of the study. + output_path: The output path of the study. + num_parameters: The number of parameters for the step. + num_samples: The number of samples for the step. + expected_count: The expected number of `MERLIN_FINISHED` files based on parameters and samples or explicitly set. + glob_string: The glob pattern to find `MERLIN_FINISHED` files in the specified step's output directory. + passes: Checks if the count of `MERLIN_FINISHED` files matches the expected count. + + Methods: + expected_count: Calculates the expected number of `MERLIN_FINISHED` files. + glob_string: Constructs the glob pattern for searching `MERLIN_FINISHED` files. + count_finished_files: Counts the number of `MERLIN_FINISHED` files found. + passes: Checks if the count of `MERLIN_FINISHED` files matches the expected count. + """ + + # All of these parameters are necessary for this Condition so we'll ignore pylint + def __init__( + self, + step: str, + study_name: str, + output_path: str, + num_parameters: int = 0, + num_samples: int = 0, + expected_count: int = None, + ): # pylint: disable=too-many-arguments + super().__init__(study_name, output_path) + self.step = step + self.num_parameters = num_parameters + self.num_samples = num_samples + self._expected_count = expected_count + + @property + def expected_count(self) -> int: + """ + Calculate the expected number of `MERLIN_FINISHED` files. + + Returns: + The expected number of `MERLIN_FINISHED` files. + """ + # Return the explicitly set expected count if given + if self._expected_count is not None: + return self._expected_count + + # Otherwise calculate the correct number of MERLIN_FINISHED files to expect + if self.num_parameters > 0 and self.num_samples > 0: + return self.num_parameters * self.num_samples + if self.num_parameters > 0: + return self.num_parameters + if self.num_samples > 0: + return self.num_samples + + return 1 # Default case when there are no parameters or samples + + @property + def glob_string(self) -> str: + """ + Glob pattern to find `MERLIN_FINISHED` files in the specified step's output directory. + + Returns: + A glob pattern to find `MERLIN_FINISHED` files. + """ + param_glob = "*" if self.num_parameters > 0 else "" + samples_glob = "**" if self.num_samples > 0 else "" + return os.path.join(self.dirpath_glob, self.step, param_glob, samples_glob, "MERLIN_FINISHED") + + def count_finished_files(self) -> int: + """ + Count the number of `MERLIN_FINISHED` files found. + + Returns: + The actual number of `MERLIN_FINISHED` files that exist in the step's output directory. + """ + finished_files = glob(self.glob_string) # Adjust the glob pattern as needed + return len(finished_files) + + @property + def passes(self) -> bool: + """ + Check if the count of `MERLIN_FINISHED` files matches the expected count. + + Returns: + True if the expected count matches the actual count. False otherwise. + """ + return self.count_finished_files() == self.expected_count + + def __str__(self) -> str: + return ( + f"{__class__.__name__} expected {self.expected_count} `MERLIN_FINISHED` " + f"files, but found {self.count_finished_files()}" + ) + + class ProvenanceYAMLFileHasRegex(HasRegex): """ A condition that a Merlin provenance yaml spec in the 'merlin_info' directory @@ -339,7 +420,7 @@ def contains(self) -> bool: with open(self.filename, "r") as f: # pylint: disable=C0103 filetext = f.read() return self.is_within(filetext) - except Exception: # pylint: disable=W0718 + except Exception: # pylint: disable=broad-except return False def is_within(self, text): diff --git a/tests/integration/database/test_data_models.py b/tests/integration/database/test_data_models.py new file mode 100644 index 000000000..4c9280478 --- /dev/null +++ b/tests/integration/database/test_data_models.py @@ -0,0 +1,74 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Integration tests to test interactions between models. +""" + +from merlin.db_scripts.data_models import LogicalWorkerModel, PhysicalWorkerModel, RunModel, StudyModel + + +class TestModelInteractions: + """Tests for interactions between models.""" + + def test_study_run_relationship(self): + """Test the relationship between StudyModel and RunModel.""" + # Create a study + study = StudyModel(id="study-123", name="Test Study") + + # Create runs associated with the study + run1 = RunModel(id="run-1", study_id=study.id) + run2 = RunModel(id="run-2", study_id=study.id) + + # Update study with run IDs + study.update_fields({"runs": [run1.id, run2.id]}) + + # Verify the relationship + assert run1.id in study.runs + assert run2.id in study.runs + assert run1.study_id == study.id + assert run2.study_id == study.id + + def test_logical_physical_worker_relationship(self): + """Test the relationship between LogicalWorkerModel and PhysicalWorkerModel.""" + # Create a logical worker + logical_worker = LogicalWorkerModel(name="worker1", queues={"queue1"}) + + # Create physical workers associated with the logical worker + physical_worker1 = PhysicalWorkerModel(id="pw-1", logical_worker_id=logical_worker.id, name="celery@worker1.host1") + + physical_worker2 = PhysicalWorkerModel(id="pw-2", logical_worker_id=logical_worker.id, name="celery@worker1.host2") + + # Update logical worker with physical worker IDs + logical_worker.update_fields({"physical_workers": [physical_worker1.id, physical_worker2.id]}) + + # Verify the relationship + assert physical_worker1.id in logical_worker.physical_workers + assert physical_worker2.id in logical_worker.physical_workers + assert physical_worker1.logical_worker_id == logical_worker.id + assert physical_worker2.logical_worker_id == logical_worker.id + + def test_run_worker_relationship(self): + """Test the relationship between RunModel and LogicalWorkerModel.""" + # Create a run + run = RunModel(id="run-test", study_id="study-test") + + # Create logical workers for the run + worker1 = LogicalWorkerModel(name="worker1", queues={"queue1"}) + worker2 = LogicalWorkerModel(name="worker2", queues={"queue2"}) + + # Update run with worker IDs + run.update_fields({"workers": [worker1.id, worker2.id]}) + + # Update workers with run ID + worker1.update_fields({"runs": [run.id]}) + worker2.update_fields({"runs": [run.id]}) + + # Verify the relationship + assert worker1.id in run.workers + assert worker2.id in run.workers + assert run.id in worker1.runs + assert run.id in worker2.runs diff --git a/tests/integration/definitions.py b/tests/integration/definitions.py index 18435d461..108664b50 100644 --- a/tests/integration/definitions.py +++ b/tests/integration/definitions.py @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ This module defines all the integration tests to be ran through run_tests.py. @@ -40,6 +17,7 @@ } """ +import os import shutil # Pylint complains that we didn't install this module but it's defined locally so ignore @@ -108,16 +86,12 @@ def define_tests(): # pylint: disable=R0914,R0915 workers_lsf = get_worker_by_cmd("jsrun", workers) run = f"merlin {err_lvl} run" restart = f"merlin {err_lvl} restart" - purge = "merlin purge" - stop = "merlin stop-workers" - query = "merlin query-workers" # Shortcuts for example workflow paths examples = "merlin/examples/workflows" dev_examples = "merlin/examples/dev_workflows" test_specs = "tests/integration/test_specs" demo = f"{examples}/feature_demo/feature_demo.yaml" - remote_demo = f"{examples}/remote_feature_demo/remote_feature_demo.yaml" demo_pgen = f"{examples}/feature_demo/scripts/pgen.py" simple = f"{examples}/simple_chain/simple_chain.yaml" slurm = f"{test_specs}/slurm_test.yaml" @@ -126,25 +100,78 @@ def define_tests(): # pylint: disable=R0914,R0915 flux_restart = f"{examples}/flux/flux_par_restart.yaml" flux_native = f"{test_specs}/flux_par_native_test.yaml" lsf = f"{examples}/lsf/lsf_par.yaml" - mul_workers_demo = f"{dev_examples}/multiple_workers.yaml" cli_substitution_wf = f"{test_specs}/cli_substitution_test.yaml" - chord_err_wf = f"{test_specs}/chord_err.yaml" # Other shortcuts black = "black --check --target-version py36" config_dir = "./CLI_TEST_MERLIN_CONFIG" release_dependencies = "./requirements/release.txt" + app_yaml_path = os.path.join(config_dir, "app.yaml") + basic_checks = { "merlin": {"cmds": "merlin", "conditions": HasReturnCode(1), "run type": "local"}, "merlin help": {"cmds": "merlin --help", "conditions": HasReturnCode(), "run type": "local"}, "merlin version": {"cmds": "merlin --version", "conditions": HasReturnCode(), "run type": "local"}, - "merlin config": { - "cmds": f"merlin config -o {config_dir}", - "conditions": HasReturnCode(), + "merlin config create": { + "cmds": f"merlin config --test create -o {app_yaml_path}", + "conditions": [ + HasReturnCode(), + PathExists(app_yaml_path), + PathExists(os.path.join(config_dir, "config_path.txt")), + FileHasRegex(os.path.join(config_dir, "config_path.txt"), app_yaml_path), + ], "run type": "local", "cleanup": f"rm -rf {config_dir}", }, + "merlin config update-broker": { + "cmds": f"""merlin config --test create -o {app_yaml_path}; merlin config update-broker \ + -t rabbitmq --cf {app_yaml_path} -u rabbitmq_user --pf rabbitmq_password_file \ + -s rabbitmq_server -p 5672 -v rabbitmq_vhost""", + "conditions": [ + HasReturnCode(), + FileHasRegex(app_yaml_path, "name: rabbitmq"), + FileHasRegex(app_yaml_path, "username: rabbitmq_user"), + FileHasRegex(app_yaml_path, "password: rabbitmq_password_file"), + FileHasRegex(app_yaml_path, "server: rabbitmq_server"), + FileHasRegex(app_yaml_path, "port: 5672"), + FileHasRegex(app_yaml_path, "vhost: rabbitmq_vhost"), + ], + "run type": "local", + "cleanup": f"rm -rf {config_dir}", + }, + "merlin config update-backend": { + "cmds": f"""merlin config --test create -o {app_yaml_path}; merlin config update-backend \ + -t redis --cf {app_yaml_path} --pf redis_password_file -s redis_server -p 6379""", + "conditions": [ + HasReturnCode(), + FileHasRegex(app_yaml_path, "name: rediss"), + FileHasRegex(app_yaml_path, "password: redis_password_file"), + FileHasRegex(app_yaml_path, "server: redis_server"), + FileHasRegex(app_yaml_path, "port: 6379"), + ], + "run type": "local", + "cleanup": f"rm -rf {config_dir}", + }, + "merlin config use": { # create two app.yaml files then choose to use the first one + "cmds": f"""merlin config --test create -o {app_yaml_path}; + merlin config --test create -o {os.path.join(config_dir, "second_app.yaml")}; + merlin config --test use {app_yaml_path}""", + "conditions": [ + HasReturnCode(), + PathExists(app_yaml_path), + PathExists(os.path.join(config_dir, "second_app.yaml")), + PathExists(os.path.join(config_dir, "config_path.txt")), + FileHasRegex(os.path.join(config_dir, "config_path.txt"), app_yaml_path), + ], + "run type": "local", + "cleanup": f"rm -rf {config_dir}", + }, + "merlin info": { + "cmds": "merlin info", + "conditions": HasReturnCode(), + "run type": "local", + }, } server_basic_tests = { "merlin server init": { @@ -650,213 +677,6 @@ def define_tests(): # pylint: disable=R0914,R0915 "run type": "local", }, } - stop_workers_tests = { - "stop workers no workers": { - "cmds": f"{stop}", - "conditions": [ - HasReturnCode(), - HasRegex("No workers found to stop"), - HasRegex("step_1_merlin_test_worker", negate=True), - HasRegex("step_2_merlin_test_worker", negate=True), - HasRegex("other_merlin_test_worker", negate=True), - ], - "run type": "distributed", - }, - "stop workers no flags": { - "cmds": [ - f"{workers} {mul_workers_demo}", - f"{stop}", - ], - "conditions": [ - HasReturnCode(), - HasRegex("No workers found to stop", negate=True), - HasRegex("step_1_merlin_test_worker"), - HasRegex("step_2_merlin_test_worker"), - HasRegex("other_merlin_test_worker"), - ], - "run type": "distributed", - "cleanup": KILL_WORKERS, - "num procs": 2, - }, - "stop workers with spec flag": { - "cmds": [ - f"{workers} {mul_workers_demo}", - f"{stop} --spec {mul_workers_demo}", - ], - "conditions": [ - HasReturnCode(), - HasRegex("No workers found to stop", negate=True), - HasRegex("step_1_merlin_test_worker"), - HasRegex("step_2_merlin_test_worker"), - HasRegex("other_merlin_test_worker"), - ], - "run type": "distributed", - "cleanup": KILL_WORKERS, - "num procs": 2, - }, - "stop workers with workers flag": { - "cmds": [ - f"{workers} {mul_workers_demo}", - f"{stop} --workers step_1_merlin_test_worker step_2_merlin_test_worker", - ], - "conditions": [ - HasReturnCode(), - HasRegex("No workers found to stop", negate=True), - HasRegex("step_1_merlin_test_worker"), - HasRegex("step_2_merlin_test_worker"), - HasRegex("other_merlin_test_worker", negate=True), - ], - "run type": "distributed", - "cleanup": KILL_WORKERS, - "num procs": 2, - }, - "stop workers with queues flag": { - "cmds": [ - f"{workers} {mul_workers_demo}", - f"{stop} --queues hello_queue", - ], - "conditions": [ - HasReturnCode(), - HasRegex("No workers found to stop", negate=True), - HasRegex("step_1_merlin_test_worker"), - HasRegex("step_2_merlin_test_worker", negate=True), - HasRegex("other_merlin_test_worker", negate=True), - ], - "run type": "distributed", - "cleanup": KILL_WORKERS, - "num procs": 2, - }, - } - query_workers_tests = { - "query workers no workers": { - "cmds": f"{query}", - "conditions": [ - HasReturnCode(), - HasRegex("No workers found!"), - HasRegex("step_1_merlin_test_worker", negate=True), - HasRegex("step_2_merlin_test_worker", negate=True), - HasRegex("other_merlin_test_worker", negate=True), - ], - "run type": "distributed", - }, - "query workers no flags": { - "cmds": [ - f"{workers} {mul_workers_demo}", - f"{query}", - ], - "conditions": [ - HasReturnCode(), - HasRegex("No workers found!", negate=True), - HasRegex("step_1_merlin_test_worker"), - HasRegex("step_2_merlin_test_worker"), - HasRegex("other_merlin_test_worker"), - ], - "run type": "distributed", - "cleanup": KILL_WORKERS, - "num procs": 2, - }, - "query workers with spec flag": { - "cmds": [ - f"{workers} {mul_workers_demo}", - f"{query} --spec {mul_workers_demo}", - ], - "conditions": [ - HasReturnCode(), - HasRegex("No workers found!", negate=True), - HasRegex("step_1_merlin_test_worker"), - HasRegex("step_2_merlin_test_worker"), - HasRegex("other_merlin_test_worker"), - ], - "run type": "distributed", - "cleanup": KILL_WORKERS, - "num procs": 2, - }, - "query workers with workers flag": { - "cmds": [ - f"{workers} {mul_workers_demo}", - f"{query} --workers step_1_merlin_test_worker step_2_merlin_test_worker", - ], - "conditions": [ - HasReturnCode(), - HasRegex("No workers found!", negate=True), - HasRegex("step_1_merlin_test_worker"), - HasRegex("step_2_merlin_test_worker"), - HasRegex("other_merlin_test_worker", negate=True), - ], - "run type": "distributed", - "cleanup": KILL_WORKERS, - "num procs": 2, - }, - "query workers with queues flag": { - "cmds": [ - f"{workers} {mul_workers_demo}", - f"{query} --queues hello_queue", - ], - "conditions": [ - HasReturnCode(), - HasRegex("No workers found!", negate=True), - HasRegex("step_1_merlin_test_worker"), - HasRegex("step_2_merlin_test_worker", negate=True), - HasRegex("other_merlin_test_worker", negate=True), - ], - "run type": "distributed", - "cleanup": KILL_WORKERS, - "num procs": 2, - }, - } - distributed_tests = { # noqa: F841 - "run and purge feature_demo": { - "cmds": f"{run} {demo}; {purge} {demo} -f", - "conditions": HasReturnCode(), - "run type": "distributed", - }, - "remote feature_demo": { - "cmds": f"""{run} {remote_demo} --vars OUTPUT_PATH=./{OUTPUT_DIR} WORKER_NAME=cli_test_demo_workers; - {workers} {remote_demo} --vars OUTPUT_PATH=./{OUTPUT_DIR} WORKER_NAME=cli_test_demo_workers""", - "conditions": [ - HasReturnCode(), - ProvenanceYAMLFileHasRegex( - regex="cli_test_demo_workers:", - spec_file_name="remote_feature_demo", - study_name="feature_demo", - output_path=OUTPUT_DIR, - provenance_type="expanded", - ), - StepFileExists( - "verify", - "MERLIN_FINISHED", - "feature_demo", - OUTPUT_DIR, - params=True, - ), - ], - "run type": "distributed", - }, - } - distributed_error_checks = { - "check chord error continues wf": { - "cmds": [ - f"{workers} {chord_err_wf} --vars OUTPUT_PATH=./{OUTPUT_DIR}", - f"{run} {chord_err_wf} --vars OUTPUT_PATH=./{OUTPUT_DIR}; sleep 40; tree {OUTPUT_DIR}", - ], - "conditions": [ - HasReturnCode(), - PathExists( # Check that the sample that's supposed to raise an error actually raises an error - f"{OUTPUT_DIR}/process_samples/01/MERLIN_FINISHED", - negate=True, - ), - StepFileExists( # Check that step 3 is actually started and completes - "step_3", - "MERLIN_FINISHED", - "chord_err", - OUTPUT_DIR, - ), - ], - "run type": "distributed", - "cleanup": KILL_WORKERS, - "num procs": 2, - } - } # combine and return test dictionaries all_tests = {} @@ -876,10 +696,6 @@ def define_tests(): # pylint: disable=R0914,R0915 # provenence_equality_checks, # omitting provenance equality check because it is broken # style_checks, # omitting style checks due to different results on different machines dependency_checks, - stop_workers_tests, - query_workers_tests, - distributed_tests, - distributed_error_checks, ]: all_tests.update(test_dict) diff --git a/tests/integration/helper_funcs.py b/tests/integration/helper_funcs.py new file mode 100644 index 000000000..30e767d83 --- /dev/null +++ b/tests/integration/helper_funcs.py @@ -0,0 +1,170 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module contains helper functions for the integration +test suite. +""" + +import os +import re +import shutil +import subprocess +from time import sleep +from typing import Dict, List + +from merlin.spec.expansion import get_spec_with_expansion +from tests.context_managers.celery_task_manager import CeleryTaskManager +from tests.context_managers.celery_workers_manager import CeleryWorkersManager +from tests.fixture_types import FixtureRedis +from tests.integration.conditions import Condition + + +def load_workers_from_spec(spec_filepath: str) -> dict: + """ + Load worker specifications from a YAML file. + + This function reads a YAML file containing study specifications and + extracts the worker information under the "merlin" section. It + constructs a dictionary in the form that + [`CeleryWorkersManager.launch_workers`][context_managers.celery_workers_manager.CeleryWorkersManager.launch_workers] + requires. + + Args: + spec_filepath: The file path to the YAML specification file. + + Returns: + A dictionary containing the worker specifications from the + "merlin" section of the YAML file. + """ + worker_info = {} + spec = get_spec_with_expansion(spec_filepath) + steps_and_queues = spec.get_task_queues(omit_tag=True) + + for worker_name, worker_settings in spec.merlin["resources"]["workers"].items(): + match = re.search(r"--concurrency\s+(\d+)", worker_settings["args"]) + concurrency = int(match.group(1)) if match else 1 + worker_info[worker_name] = {"concurrency": concurrency} + if worker_settings["steps"] == ["all"]: + worker_info[worker_name]["queues"] = list(steps_and_queues.values()) + else: + worker_info[worker_name]["queues"] = [steps_and_queues[step] for step in worker_settings["steps"]] + + return worker_info + + +def copy_app_yaml_to_cwd(merlin_server_dir: str): + """ + Copy the app.yaml file from the directory provided to the current working + directory. + + Grab the app.yaml file from `merlin_server_dir` and copy it to the current + working directory so that Merlin will read this in as the server configuration + for whatever test is calling this. + + Args: + merlin_server_dir: The path to the `merlin_server` directory that should be created by the + [`redis_server`][conftest.redis_server] fixture. + """ + copied_app_yaml = os.path.join(os.getcwd(), "app.yaml") + if not os.path.exists(copied_app_yaml): + server_app_yaml = os.path.join(merlin_server_dir, "app.yaml") + shutil.copy(server_app_yaml, copied_app_yaml) + + +def check_test_conditions(conditions: List[Condition], info: Dict[str, str]): + """ + Ensure all specified test conditions are satisfied based on the output + from a subprocess. + + This function iterates through a list of [`Condition`][integration.conditions.Condition] + instances, ingests the provided information (stdout, stderr, and return + code) for each condition, and checks if each condition passes. If any + condition fails, an AssertionError is raised with a detailed message that + includes the condition that failed, along with the captured output and + return code. + + Args: + conditions: A list of Condition instances that define the expectations for the test. + info: A dictionary containing the output from the subprocess, which should + include the following keys:\n + - 'stdout': The standard output captured from the subprocess. + - 'stderr': The standard error output captured from the subprocess. + - 'return_code': The return code of the subprocess, indicating success + or failure of the command executed. + + Raises: + AssertionError: If any of the conditions do not pass, an AssertionError is raised with + a detailed message including the failed condition and the subprocess + output. + """ + for condition in conditions: + condition.ingest_info(info) + try: + assert condition.passes + except AssertionError as exc: + error_message = ( + f"Condition failed: {condition}\n" + f"Captured stdout: {info['stdout']}\n" + f"Captured stderr: {info['stderr']}\n" + f"Return code: {info['return_code']}\n" + ) + raise AssertionError(error_message) from exc + + +def run_workflow(redis_client: FixtureRedis, workflow_path: str, vars_to_substitute: List[str]) -> subprocess.CompletedProcess: + """ + Run a Merlin workflow using the `merlin run` and `merlin run-workers` commands. + + This function executes a Merlin workflow using a specified path to a study and variables to + configure the study with. It utilizes context managers to safely send tasks to the server + and start up workers. The tasks are given 15 seconds to be sent to the server. Once tasks + exist on the server, the workflow is given 30 seconds to run to completion, which should be + plenty of time. + + Args: + redis_client: A fixture that connects us to a redis client that we can interact with. + workflow_path: The path to the study that we're going to run here + vars_to_substitute: A list of variables in the form ["VAR_NAME=var_value"] to be modified + in the workflow. + + Returns: + The completed process object containing information about the execution of the workflow, including + return code, stdout, and stderr. + """ + from merlin.celery import app as celery_app # pylint: disable=import-outside-toplevel + + run_workers_proc = None + + with CeleryTaskManager(celery_app, redis_client): + # Send the tasks to the server + try: + subprocess.run( + f"merlin run {workflow_path} --vars {' '.join(vars_to_substitute)}", + shell=True, + capture_output=True, + text=True, + timeout=15, + ) + except subprocess.TimeoutExpired as exc: + raise TimeoutError("Could not send tasks to the server within the allotted time.") from exc + + # We use a context manager to start workers so that they'll safely stop even if this test fails + with CeleryWorkersManager(celery_app) as celery_worker_manager: + # Start the workers then add them to the context manager so they can be stopped safely later + run_workers_proc = subprocess.Popen( # pylint: disable=consider-using-with + f"merlin run-workers {workflow_path}".split(), + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + start_new_session=True, + ) + celery_worker_manager.add_run_workers_process(run_workers_proc.pid) + + # Let the workflow try to run for 30 seconds + sleep(30) + + return run_workers_proc diff --git a/tests/integration/run_tests.py b/tests/integration/run_tests.py index 868ec1c3a..cb1d2e0ee 100644 --- a/tests/integration/run_tests.py +++ b/tests/integration/run_tests.py @@ -1,32 +1,8 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## """ Script for running Merlin command line interface tests. @@ -39,10 +15,9 @@ from contextlib import suppress from subprocess import TimeoutExpired, run +from definitions import OUTPUT_DIR, define_tests # pylint: disable=E0401 from tabulate import tabulate -from tests.integration.definitions import OUTPUT_DIR, define_tests # pylint: disable=E0401 - def get_definition_issues(test): """ diff --git a/tests/integration/test_celeryadapter.py b/tests/integration/test_celeryadapter.py index a40f1054b..2073e0bad 100644 --- a/tests/integration/test_celeryadapter.py +++ b/tests/integration/test_celeryadapter.py @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the celeryadapter module. """ @@ -34,12 +11,9 @@ import json import os from datetime import datetime -from time import sleep from typing import Dict -import pytest from celery import Celery -from celery.canvas import Signature from deepdiff import DeepDiff from merlin.config import Config @@ -49,150 +23,153 @@ from tests.unit.study.status_test_files.status_test_variables import SPEC_PATH -@pytest.mark.order(before="TestInactive") -class TestActive: - """ - This class will test functions in the celeryadapter.py module. - It will run tests where we need active queues/workers to interact with. - - NOTE: The tests in this class must be ran before the TestInactive class or else the - Celery workers needed for this class don't start - - TODO: fix the bug noted above and then check if we still need pytest-order - """ - - def test_query_celery_queues( - self, celery_app: Celery, launch_workers: "Fixture", worker_queue_map: Dict[str, str] # noqa: F821 - ): - """ - Test the query_celery_queues function by providing it with a list of active queues. - This should return a dict where keys are queue names and values are more dicts containing - the number of jobs and consumers in that queue. - - :param `celery_app`: A pytest fixture for the test Celery app - :param launch_workers: A pytest fixture that launches celery workers for us to interact with - :param worker_queue_map: A pytest fixture that returns a dict of workers and queues - """ - # Set up a dummy configuration to use in the test - dummy_config = Config({"broker": {"name": "redis"}}) - - # Get the actual output - queues_to_query = list(worker_queue_map.values()) - actual_queue_info = celeryadapter.query_celery_queues(queues_to_query, app=celery_app, config=dummy_config) - - # Ensure all 3 queues in worker_queue_map were queried before looping - assert len(actual_queue_info) == 3 - - # Ensure each queue has a worker attached - for queue_name, queue_info in actual_queue_info.items(): - assert queue_name in worker_queue_map.values() - assert queue_info == {"consumers": 1, "jobs": 0} - - def test_get_running_queues(self, launch_workers: "Fixture", worker_queue_map: Dict[str, str]): # noqa: F821 - """ - Test the get_running_queues function with queues active. - This should return a list of active queues. - - :param `launch_workers`: A pytest fixture that launches celery workers for us to interact with - :param `worker_queue_map`: A pytest fixture that returns a dict of workers and queues - """ - result = celeryadapter.get_running_queues("merlin_test_app", test_mode=True) - assert sorted(result) == sorted(list(worker_queue_map.values())) - - def test_get_active_celery_queues( - self, celery_app: Celery, launch_workers: "Fixture", worker_queue_map: Dict[str, str] # noqa: F821 - ): - """ - Test the get_active_celery_queues function with queues active. - This should return a tuple where the first entry is a dict of queue info - and the second entry is a list of worker names. - - :param `celery_app`: A pytest fixture for the test Celery app - :param `launch_workers`: A pytest fixture that launches celery workers for us to interact with - :param `worker_queue_map`: A pytest fixture that returns a dict of workers and queues - """ - # Start the queues and run the test - queue_result, worker_result = celeryadapter.get_active_celery_queues(celery_app) - - # Ensure we got output before looping - assert len(queue_result) == len(worker_result) == 3 - - for worker, queue in worker_queue_map.items(): - # Check that the entry in the queue_result dict for this queue is correct - assert queue in queue_result - assert len(queue_result[queue]) == 1 - assert worker in queue_result[queue][0] - - # Remove this entry from the queue_result dict - del queue_result[queue] - - # Check that this worker was added to the worker_result list - worker_found = False - for worker_name in worker_result[:]: - if worker in worker_name: - worker_found = True - worker_result.remove(worker_name) - break - assert worker_found - - # Ensure there was no extra output that we weren't expecting - assert queue_result == {} - assert worker_result == [] - - def test_build_set_of_queues( - self, celery_app: Celery, launch_workers: "Fixture", worker_queue_map: Dict[str, str] # noqa: F821 - ): - """ - Test the build_set_of_queues function with queues active. - This should return a set of queues (the queues defined in setUp). - """ - # Run the test - result = celeryadapter.build_set_of_queues( - steps=["all"], spec=None, specific_queues=None, verbose=False, app=celery_app - ) - assert result == set(worker_queue_map.values()) - - @pytest.mark.order(index=1) - def test_check_celery_workers_processing_tasks( - self, - celery_app: Celery, - sleep_sig: Signature, - launch_workers: "Fixture", # noqa: F821 - ): - """ - Test the check_celery_workers_processing function with workers active and a task in a queue. - This function will query workers for any tasks they're still processing. We'll send a - a task that sleeps for 3 seconds to our workers before we run this test so that there should be - a task for this function to find. - - NOTE: the celery app fixture shows strange behavior when using app.control.inspect() calls (which - check_celery_workers_processing uses) so we have to run this test first in this class in order to - have it run properly. - - :param celery_app: A pytest fixture for the test Celery app - :param sleep_sig: A pytest fixture for a celery signature of a task that sleeps for 3 sec - :param launch_workers: A pytest fixture that launches celery workers for us to interact with - """ - # Our active workers/queues are test_worker_[0-2]/test_queue_[0-2] so we're - # sending this to test_queue_0 for test_worker_0 to process - queue_for_signature = "test_queue_0" - sleep_sig.set(queue=queue_for_signature) - result = sleep_sig.delay() - - # We need to give the task we just sent to the server a second to get picked up by the worker - sleep(1) - - # Run the test now that the task should be getting processed - active_queue_test = celeryadapter.check_celery_workers_processing([queue_for_signature], celery_app) - assert active_queue_test is True - - # Now test that a queue without any tasks returns false - # We sent the signature to task_queue_0 so task_queue_1 shouldn't have any tasks to find - non_active_queue_test = celeryadapter.check_celery_workers_processing(["test_queue_1"], celery_app) - assert non_active_queue_test is False - - # Wait for the worker to finish running the task - result.get() +# from time import sleep +# import pytest +# from celery.canvas import Signature +# @pytest.mark.order(before="TestInactive") +# class TestActive: +# """ +# This class will test functions in the celeryadapter.py module. +# It will run tests where we need active queues/workers to interact with. + +# NOTE: The tests in this class must be ran before the TestInactive class or else the +# Celery workers needed for this class don't start + +# TODO: fix the bug noted above and then check if we still need pytest-order +# """ + +# def test_query_celery_queues( +# self, celery_app: Celery, launch_workers: "Fixture", worker_queue_map: Dict[str, str] # noqa: F821 +# ): +# """ +# Test the query_celery_queues function by providing it with a list of active queues. +# This should return a dict where keys are queue names and values are more dicts containing +# the number of jobs and consumers in that queue. + +# :param `celery_app`: A pytest fixture for the test Celery app +# :param launch_workers: A pytest fixture that launches celery workers for us to interact with +# :param worker_queue_map: A pytest fixture that returns a dict of workers and queues +# """ +# # Set up a dummy configuration to use in the test +# dummy_config = Config({"broker": {"name": "redis"}}) + +# # Get the actual output +# queues_to_query = list(worker_queue_map.values()) +# actual_queue_info = celeryadapter.query_celery_queues(queues_to_query, app=celery_app, config=dummy_config) + +# # Ensure all 3 queues in worker_queue_map were queried before looping +# assert len(actual_queue_info) == 3 + +# # Ensure each queue has a worker attached +# for queue_name, queue_info in actual_queue_info.items(): +# assert queue_name in worker_queue_map.values() +# assert queue_info == {"consumers": 1, "jobs": 0} + +# def test_get_running_queues(self, launch_workers: "Fixture", worker_queue_map: Dict[str, str]): # noqa: F821 +# """ +# Test the get_running_queues function with queues active. +# This should return a list of active queues. + +# :param `launch_workers`: A pytest fixture that launches celery workers for us to interact with +# :param `worker_queue_map`: A pytest fixture that returns a dict of workers and queues +# """ +# result = celeryadapter.get_running_queues("merlin_test_app", test_mode=True) +# assert sorted(result) == sorted(list(worker_queue_map.values())) + +# def test_get_active_celery_queues( +# self, celery_app: Celery, launch_workers: "Fixture", worker_queue_map: Dict[str, str] # noqa: F821 +# ): +# """ +# Test the get_active_celery_queues function with queues active. +# This should return a tuple where the first entry is a dict of queue info +# and the second entry is a list of worker names. + +# :param `celery_app`: A pytest fixture for the test Celery app +# :param `launch_workers`: A pytest fixture that launches celery workers for us to interact with +# :param `worker_queue_map`: A pytest fixture that returns a dict of workers and queues +# """ +# # Start the queues and run the test +# queue_result, worker_result = celeryadapter.get_active_celery_queues(celery_app) + +# # Ensure we got output before looping +# assert len(queue_result) == len(worker_result) == 3 + +# for worker, queue in worker_queue_map.items(): +# # Check that the entry in the queue_result dict for this queue is correct +# assert queue in queue_result +# assert len(queue_result[queue]) == 1 +# assert worker in queue_result[queue][0] + +# # Remove this entry from the queue_result dict +# del queue_result[queue] + +# # Check that this worker was added to the worker_result list +# worker_found = False +# for worker_name in worker_result[:]: +# if worker in worker_name: +# worker_found = True +# worker_result.remove(worker_name) +# break +# assert worker_found + +# # Ensure there was no extra output that we weren't expecting +# assert queue_result == {} +# assert worker_result == [] + +# def test_build_set_of_queues( +# self, celery_app: Celery, launch_workers: "Fixture", worker_queue_map: Dict[str, str] # noqa: F821 +# ): +# """ +# Test the build_set_of_queues function with queues active. +# This should return a set of queues (the queues defined in setUp). +# """ +# # Run the test +# result = celeryadapter.build_set_of_queues( +# steps=["all"], spec=None, specific_queues=None, verbose=False, app=celery_app +# ) +# assert result == set(worker_queue_map.values()) + +# @pytest.mark.order(index=1) +# def test_check_celery_workers_processing_tasks( +# self, +# celery_app: Celery, +# sleep_sig: Signature, +# launch_workers: "Fixture", # noqa: F821 +# ): +# """ +# Test the check_celery_workers_processing function with workers active and a task in a queue. +# This function will query workers for any tasks they're still processing. We'll send a +# a task that sleeps for 3 seconds to our workers before we run this test so that there should be +# a task for this function to find. + +# NOTE: the celery app fixture shows strange behavior when using app.control.inspect() calls (which +# check_celery_workers_processing uses) so we have to run this test first in this class in order to +# have it run properly. + +# :param celery_app: A pytest fixture for the test Celery app +# :param sleep_sig: A pytest fixture for a celery signature of a task that sleeps for 3 sec +# :param launch_workers: A pytest fixture that launches celery workers for us to interact with +# """ +# # Our active workers/queues are test_worker_[0-2]/test_queue_[0-2] so we're +# # sending this to test_queue_0 for test_worker_0 to process +# queue_for_signature = "test_queue_0" +# sleep_sig.set(queue=queue_for_signature) +# result = sleep_sig.delay() + +# # We need to give the task we just sent to the server a second to get picked up by the worker +# sleep(1) + +# # Run the test now that the task should be getting processed +# active_queue_test = celeryadapter.check_celery_workers_processing([queue_for_signature], celery_app) +# assert active_queue_test is True + +# # Now test that a queue without any tasks returns false +# # We sent the signature to task_queue_0 so task_queue_1 shouldn't have any tasks to find +# non_active_queue_test = celeryadapter.check_celery_workers_processing(["test_queue_1"], celery_app) +# assert non_active_queue_test is False + +# # Wait for the worker to finish running the task +# result.get() class TestInactive: @@ -250,7 +227,7 @@ def test_get_running_queues(self): This should return an empty list. """ result = celeryadapter.get_running_queues("merlin_test_app", test_mode=True) - assert result == [] + assert not result def test_get_active_celery_queues(self, celery_app: Celery): """ @@ -261,8 +238,8 @@ def test_get_active_celery_queues(self, celery_app: Celery): :param `celery_app`: A pytest fixture for the test Celery app """ queue_result, worker_result = celeryadapter.get_active_celery_queues(celery_app) - assert queue_result == {} - assert worker_result == [] + assert not queue_result + assert not worker_result def test_check_celery_workers_processing_tasks(self, celery_app: Celery, worker_queue_map: Dict[str, str]): """ @@ -476,7 +453,7 @@ def test_dump_celery_queue_info_csv(self, worker_queue_map: Dict[str, str]): # Make sure the rest of the csv file was created as expected dump_diff = DeepDiff(csv_dump_output, expected_output) - assert dump_diff == {} + assert not dump_diff finally: try: os.remove(outfile) @@ -513,7 +490,7 @@ def test_dump_celery_queue_info_json(self, worker_queue_map: Dict[str, str]): # There should only be one entry in the json dump file so this will only 'loop' once for dump_entry in json_df_contents.values(): json_dump_diff = DeepDiff(dump_entry, expected_output) - assert json_dump_diff == {} + assert not json_dump_diff finally: try: os.remove(outfile) diff --git a/tests/integration/test_specs/chord_err.yaml b/tests/integration/test_specs/chord_err.yaml index 3da99ae03..9fe7d55ea 100644 --- a/tests/integration/test_specs/chord_err.yaml +++ b/tests/integration/test_specs/chord_err.yaml @@ -1,10 +1,11 @@ description: - name: chord_err + name: $(NAME) description: test the chord err problem env: variables: OUTPUT_PATH: ./studies + NAME: chord_err global.parameters: TEST_PARAM: diff --git a/tests/integration/test_specs/monitor_auto_restart_test.yaml b/tests/integration/test_specs/monitor_auto_restart_test.yaml new file mode 100644 index 000000000..ca03d9c53 --- /dev/null +++ b/tests/integration/test_specs/monitor_auto_restart_test.yaml @@ -0,0 +1,36 @@ +description: + name: monitor_auto_restart_test + description: a spec that helps test the monitor's auto restart functionality + +env: + variables: + N_SAMPLES: 3 + OUTPUT_PATH: . + +study: + - name: process_samples + description: Run samples for a certain duration + run: + cmd: | + echo $(SAMPLE) + sleep 10 + task_queue: sim_queue + + - name: funnel_step + description: print a success message + run: + cmd: echo "all finished" + depends: [process_samples_*] + task_queue: seq_queue + +merlin: + resources: + workers: + worker2: + args: -l INFO --concurrency 2 --prefetch-multiplier 1 -Ofair + steps: [all] + samples: + generate: + cmd: spellbook make-samples -dims 1 -n $(N_SAMPLES) -outfile=$(MERLIN_INFO)/samples.npy + file: $(MERLIN_INFO)/samples.npy + column_labels: [SAMPLE] \ No newline at end of file diff --git a/tests/integration/test_specs/multiple_workers.yaml b/tests/integration/test_specs/multiple_workers.yaml new file mode 100644 index 000000000..967582a53 --- /dev/null +++ b/tests/integration/test_specs/multiple_workers.yaml @@ -0,0 +1,56 @@ +description: + name: multiple_workers + description: a very simple merlin workflow with multiple workers + +global.parameters: + GREET: + values : ["hello","hola"] + label : GREET.%% + WORLD: + values : ["world","mundo"] + label : WORLD.%% + +study: + - name: step_1 + description: say hello + run: + cmd: | + echo "$(GREET), $(WORLD)!" + task_queue: hello_queue + + - name: step_2 + description: step 2 + run: + cmd: | + echo "step_2" + depends: [step_1_*] + task_queue: echo_queue + + - name: step_3 + description: stop workers + run: + cmd: | + echo "stop workers" + depends: [step_2] + task_queue: other_queue + + - name: step_4 + description: another step + run: + cmd: | + echo "another step" + depends: [step_3] + task_queue: other_queue + +merlin: + resources: + workers: + step_1_merlin_test_worker: + args: -l INFO --concurrency 1 + steps: [step_1] + step_2_merlin_test_worker: + args: -l INFO --concurrency 1 + steps: [step_2] + other_merlin_test_worker: + args: -l INFO --concurrency 1 + steps: [step_3, step_4] diff --git a/tests/integration/workflows/test_chord_error.py b/tests/integration/workflows/test_chord_error.py new file mode 100644 index 000000000..70e2af768 --- /dev/null +++ b/tests/integration/workflows/test_chord_error.py @@ -0,0 +1,69 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module contains tests for the feature_demo workflow. +""" + +import subprocess + +from tests.fixture_data_classes import ChordErrorSetup +from tests.integration.conditions import HasRegex, StepFinishedFilesCount +from tests.integration.helper_funcs import check_test_conditions + + +class TestChordError: + """ + Tests for the chord error workflow. + """ + + def test_chord_error_continues( + self, + chord_err_setup: ChordErrorSetup, + chord_err_run_workflow: subprocess.CompletedProcess, + ): + """ + Test that this workflow continues through to the end of its execution, even + though a ChordError will be raised. + + Args: + chord_err_setup: A fixture that returns a [`ChordErrorSetup`][fixture_data_classes.ChordErrorSetup] + instance. + chord_err_run_workflow: A fixture to run the chord error study. + """ + + conditions = [ + HasRegex("Exception raised by request from the user"), + StepFinishedFilesCount( # Check that the `process_samples` step has only 2 MERLIN_FINISHED files + step="process_samples", + study_name=chord_err_setup.name, + output_path=chord_err_setup.testing_dir, + expected_count=2, + num_samples=3, + ), + StepFinishedFilesCount( # Check that the `samples_and_params` step has all of its MERLIN_FINISHED files + step="samples_and_params", + study_name=chord_err_setup.name, + output_path=chord_err_setup.testing_dir, + num_parameters=2, + num_samples=3, + ), + StepFinishedFilesCount( # Check that the final step has a MERLIN_FINISHED file + step="step_3", + study_name=chord_err_setup.name, + output_path=chord_err_setup.testing_dir, + num_parameters=0, + num_samples=0, + ), + ] + + info = { + "return_code": chord_err_run_workflow.returncode, + "stdout": chord_err_run_workflow.stdout.read(), + "stderr": chord_err_run_workflow.stderr.read(), + } + + check_test_conditions(conditions, info) diff --git a/tests/integration/workflows/test_feature_demo.py b/tests/integration/workflows/test_feature_demo.py new file mode 100644 index 000000000..dbe11bffc --- /dev/null +++ b/tests/integration/workflows/test_feature_demo.py @@ -0,0 +1,138 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module contains tests for the feature_demo workflow. +""" + +import shutil +import subprocess + +from tests.fixture_data_classes import FeatureDemoSetup +from tests.integration.conditions import ProvenanceYAMLFileHasRegex, StepFinishedFilesCount + + +class TestFeatureDemo: + """ + Tests for the feature_demo workflow. + """ + + def test_end_to_end_run( + self, feature_demo_setup: FeatureDemoSetup, feature_demo_run_workflow: subprocess.CompletedProcess + ): + """ + Test that the workflow runs from start to finish with no problems. + + This will check that each step has the proper amount of `MERLIN_FINISHED` files. + The workflow will be run via the + [`feature_demo_run_workflow`][fixtures.feature_demo.feature_demo_run_workflow] + fixture. + + Args: + feature_demo_setup: A fixture that returns a + [`FeatureDemoSetup`][fixture_data_classes.FeatureDemoSetup] instance. + feature_demo_run_workflow: A fixture to run the feature demo study. + """ + conditions = [ + ProvenanceYAMLFileHasRegex( # This condition will check that variable substitution worked + regex=f"N_SAMPLES: {feature_demo_setup.num_samples}", + spec_file_name="feature_demo", + study_name=feature_demo_setup.name, + output_path=feature_demo_setup.testing_dir, + provenance_type="expanded", + ), + StepFinishedFilesCount( # The rest of the conditions will ensure every step ran to completion + step="hello", + study_name=feature_demo_setup.name, + output_path=feature_demo_setup.testing_dir, + num_parameters=1, + num_samples=feature_demo_setup.num_samples, + ), + StepFinishedFilesCount( + step="python3_hello", + study_name=feature_demo_setup.name, + output_path=feature_demo_setup.testing_dir, + num_parameters=1, + num_samples=0, + ), + StepFinishedFilesCount( + step="collect", + study_name=feature_demo_setup.name, + output_path=feature_demo_setup.testing_dir, + num_parameters=1, + num_samples=0, + ), + StepFinishedFilesCount( + step="translate", + study_name=feature_demo_setup.name, + output_path=feature_demo_setup.testing_dir, + num_parameters=1, + num_samples=0, + ), + StepFinishedFilesCount( + step="learn", + study_name=feature_demo_setup.name, + output_path=feature_demo_setup.testing_dir, + num_parameters=1, + num_samples=0, + ), + StepFinishedFilesCount( + step="make_new_samples", + study_name=feature_demo_setup.name, + output_path=feature_demo_setup.testing_dir, + num_parameters=1, + num_samples=0, + ), + StepFinishedFilesCount( + step="predict", + study_name=feature_demo_setup.name, + output_path=feature_demo_setup.testing_dir, + num_parameters=1, + num_samples=0, + ), + StepFinishedFilesCount( + step="verify", + study_name=feature_demo_setup.name, + output_path=feature_demo_setup.testing_dir, + num_parameters=1, + num_samples=0, + ), + ] + + # GitHub actions doesn't have a python2 path so we'll conditionally add this check + if shutil.which("python2"): + conditions.append( + StepFinishedFilesCount( + step="python2_hello", + study_name=feature_demo_setup.name, + output_path=feature_demo_setup.testing_dir, + num_parameters=1, + num_samples=0, + ) + ) + + for condition in conditions: + assert condition.passes + + # TODO implement the below tests + # def test_step_execution_order(self): + # """ + # Test that steps are executed in the correct order. + # """ + # # TODO build a list with the correct order that steps should be ran + # # TODO compare the list against the logs from the worker + + # def test_workflow_error_handling(self): + # """ + # Test the behavior when errors arise during the worfklow. + + # TODO should this test both soft and hard fails? should this test all return codes? + # """ + + # def test_data_passing(self): + # """ + # Test that data can be successfully passed between steps using built-in Merlin variables. + # """ diff --git a/tests/unit/backends/redis/test_redis_backend.py b/tests/unit/backends/redis/test_redis_backend.py new file mode 100644 index 000000000..40efc9064 --- /dev/null +++ b/tests/unit/backends/redis/test_redis_backend.py @@ -0,0 +1,133 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `redis_backend.py` module. +""" + +from unittest.mock import MagicMock + +import pytest +from pytest_mock import MockerFixture +from redis import Redis + +from merlin.backends.redis.redis_backend import RedisBackend +from tests.fixture_types import FixtureModification, FixtureStr + + +@pytest.fixture +def redis_backend_connection_string() -> FixtureStr: + """ + Fixture to provide a mock Redis connection string. + + This fixture returns a Redis connection string that can be used in tests to simulate + connecting to a Redis server. It ensures that tests relying on a Redis connection + string do not require a real Redis instance and remain isolated. + + Returns: + A mock Redis connection string. + """ + return "redis://localhost:6379" + + +@pytest.fixture +def redis_backend_mock_redis_client(mocker: MockerFixture) -> MagicMock: + """ + Mocks the Redis client. + + Args: + mocker (MockerFixture): Used to create a mock Redis client. + + Returns: + A mocked Redis client. + """ + return mocker.MagicMock(spec=Redis) + + +@pytest.fixture +def redis_backend_instance( + mocker: MockerFixture, + redis_results_backend_config_class: FixtureModification, + redis_backend_connection_string: FixtureStr, + redis_backend_mock_redis_client: MagicMock, +) -> RedisBackend: + """ + Fixture to create a `RedisBackend` instance with mocked dependencies. + + This fixture sets up a `RedisBackend` instance with its Redis client and store mappings mocked, + allowing tests to run without requiring an actual Redis server. It uses the `mocker` library + to patch external dependencies such as the Redis client, connection string retrieval, and + configuration settings. + + Args: + mocker (MockerFixture): The pytest-mock fixture used for mocking objects and patching + external dependencies. + redis_results_backend_config_class (FixtureModification): A fixture that sets the `CONFIG` + object to point to a Redis backend. + + Returns: + A `RedisBackend` instance with mocked Redis client and stores. + """ + # Mock the `info()` return value + redis_backend_mock_redis_client.info.return_value = {"redis_version": "6.2.5"} + + # Patch Redis.from_url to return the mock client + mocker.patch("merlin.backends.redis.redis_backend.Redis.from_url", return_value=redis_backend_mock_redis_client) + + # Patch the connection string retrieval + mocker.patch("merlin.config.results_backend.get_connection_string", return_value=redis_backend_connection_string) + + # Initialize RedisBackend + backend = RedisBackend("redis") + + # Override the client and stores with mocked objects + backend.client = redis_backend_mock_redis_client + backend.stores = { + "study": mocker.MagicMock(), + "run": mocker.MagicMock(), + "logical_worker": mocker.MagicMock(), + "physical_worker": mocker.MagicMock(), + } + + return backend + + +class TestRedisBackend: + """ + Test suite for the `RedisBackend` class. + + This class contains unit tests to validate the functionality of the `RedisBackend` implementation, + which provides an interface for interacting with Redis as a backend for storing and retrieving entities. + + Fixtures and mocking are used to isolate the tests from the actual Redis implementation, ensuring that the tests focus + on the behavior of the `RedisBackend` class. + + Parametrization is employed to reduce redundancy and ensure comprehensive coverage across different entity types + and operations. + """ + + def test_get_version(self, redis_backend_instance: RedisBackend): + """ + Test that RedisBackend correctly retrieves the Redis version. + + Args: + redis_backend_instance (RedisBackend): A fixture representing a `RedisBackend` instance with + mocked Redis client and stores. + """ + redis_backend_instance.client.info.return_value = {"redis_version": "6.2.6"} + version = redis_backend_instance.get_version() + assert version == "6.2.6", "Redis version should be correctly retrieved." + + def test_flush_database(self, redis_backend_instance: RedisBackend): + """ + Test that RedisBackend flushes the database. + + Args: + redis_backend_instance (RedisBackend): A fixture representing a `RedisBackend` instance with + mocked Redis client and stores. + """ + redis_backend_instance.flush_database() + redis_backend_instance.client.flushdb.assert_called_once() diff --git a/tests/unit/backends/redis/test_redis_store_base.py b/tests/unit/backends/redis/test_redis_store_base.py new file mode 100644 index 000000000..ba612d06e --- /dev/null +++ b/tests/unit/backends/redis/test_redis_store_base.py @@ -0,0 +1,495 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `merlin/backends/redis/redis_store_base.py` module. +""" + +from unittest.mock import MagicMock + +import pytest +from pytest_mock import MockerFixture + +from merlin.backends.redis.redis_store_base import NameMappingMixin, RedisStoreBase +from merlin.db_scripts.data_models import RunModel, StudyModel +from merlin.exceptions import RunNotFoundError, StudyNotFoundError +from tests.fixture_types import FixtureCallable, FixtureDict, FixtureRedis + + +class TestRedisStoreBase: + """Tests for the RedisStoreBase class.""" + + @pytest.fixture + def simple_store(self, mock_redis: FixtureRedis) -> RedisStoreBase: + """ + Create a simple store instance for testing. + + Args: + mock_redis: A fixture providing a mocked Redis client. + + Returns: + A simple RedisStoreBase implementation for testing. + """ + + # Create a simple subclass of RedisStoreBase for testing + class TestStore(RedisStoreBase): + pass + + store = TestStore(mock_redis, "test", RunModel) + return store + + def test_initialization(self, mock_redis: FixtureRedis): + """ + Test that the store initializes correctly. + + Args: + mock_redis: A fixture providing a mocked Redis client. + """ + store = RedisStoreBase(mock_redis, "test_key", RunModel) + + assert store.client == mock_redis + assert store.key == "test_key" + assert store.model_class == RunModel + + def test_get_full_key(self, simple_store: RedisStoreBase): + """ + Test the _get_full_key method. + + Args: + simple_store: A fixture providing a RedisStoreBase instance. + """ + # When the ID already has the prefix + key = simple_store._get_full_key("test:123") + assert key == "test:123" + + # When the ID doesn't have the prefix + key = simple_store._get_full_key("123") + assert key == "test:123" + + def test_save_new_object( + self, + mocker: MockerFixture, + test_models: FixtureRedis, + simple_store: RedisStoreBase, + ): + """ + Test saving a new object to Redis. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + simple_store: A fixture providing a RedisStoreBase instance. + """ + run = test_models["run"] + simple_store.client.exists.return_value = False + mock_serialize = mocker.patch( + "merlin.backends.redis.redis_store_base.serialize_entity", return_value={"id": run.id, "foo": "bar"} + ) + + simple_store.save(run) + + simple_store.client.exists.assert_called_once_with(f"{simple_store.key}:{run.id}") + mock_serialize.assert_called_once_with(run) + simple_store.client.hset.assert_called_once_with(f"{simple_store.key}:{run.id}", mapping={"id": run.id, "foo": "bar"}) + + def test_save_existing_object( + self, + mocker: MockerFixture, + test_models: FixtureRedis, + simple_store: RedisStoreBase, + ): + """ + Test updating an existing object in Redis. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + simple_store: A fixture providing a RedisStoreBase instance. + """ + run = test_models["run"] + simple_store.client.exists.return_value = True + # Mock deserialization of existing data + mock_deserialize = mocker.patch("merlin.backends.redis.redis_store_base.deserialize_entity") + mock_existing = MagicMock() + mock_deserialize.return_value = mock_existing + simple_store.client.hgetall.return_value = {"id": run.id, "foo": "old"} + # Mock update_fields and to_dict + mock_existing.update_fields = MagicMock() + mock_existing.to_dict.return_value = {"id": run.id, "foo": "bar"} + mock_serialize = mocker.patch( + "merlin.backends.redis.redis_store_base.serialize_entity", return_value={"id": run.id, "foo": "bar"} + ) + + simple_store.save(run) + + simple_store.client.exists.assert_called_once_with(f"{simple_store.key}:{run.id}") + simple_store.client.hgetall.assert_called_once_with(f"{simple_store.key}:{run.id}") + mock_deserialize.assert_called_once_with({"id": run.id, "foo": "old"}, RunModel) + mock_existing.update_fields.assert_called_once_with(run.to_dict()) + mock_serialize.assert_called_once_with(mock_existing) + simple_store.client.hset.assert_called_once_with(f"{simple_store.key}:{run.id}", mapping={"id": run.id, "foo": "bar"}) + + def test_retrieve_existing_object( + self, + mocker: MockerFixture, + test_models: FixtureRedis, + create_redis_hash_data: FixtureCallable, + simple_store: RedisStoreBase, + ): + """ + Test retrieving an existing object from Redis. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + create_redis_hash_data: A fixture that creates Redis hash data. + simple_store: A fixture providing a RedisStoreBase instance. + """ + run = test_models["run"] + simple_store.client.exists.return_value = True + redis_data = create_redis_hash_data(run) + simple_store.client.hgetall.return_value = redis_data + mock_deserialize = mocker.patch("merlin.backends.redis.redis_store_base.deserialize_entity", return_value=run) + + result = simple_store.retrieve(run.id) + + simple_store.client.exists.assert_called_once_with(f"{simple_store.key}:{run.id}") + simple_store.client.hgetall.assert_called_once_with(f"{simple_store.key}:{run.id}") + mock_deserialize.assert_called_once_with(redis_data, RunModel) + assert result == run + + def test_retrieve_nonexistent_object(self, simple_store: RedisStoreBase): + """ + Test retrieving a non-existent object from Redis. + + Args: + simple_store: A fixture providing a RedisStoreBase instance. + """ + # Setup + simple_store.client.exists.return_value = False + + # Call the method + result = simple_store.retrieve("nonexistent_id") + + # Assertions + simple_store.client.exists.assert_called_once_with(f"{simple_store.key}:nonexistent_id") + simple_store.client.hgetall.assert_not_called() + assert result is None + + def test_retrieve_all( + self, + mocker: MockerFixture, + test_models: FixtureRedis, + simple_store: RedisStoreBase, + ): + """ + Test retrieving all objects from Redis. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + simple_store: A fixture providing a RedisStoreBase instance. + """ + # Setup + run1 = test_models["run"] + run2 = RunModel(id="run2", study_id="study1") + + keys = [f"{simple_store.key}:{run1.id}", f"{simple_store.key}:{run2.id}"] + simple_store.client.scan_iter.return_value = keys + + # Mock the retrieve method to return our test models + mocker.patch.object(simple_store, "retrieve", side_effect=[run1, run2]) + + # Call the method + results = simple_store.retrieve_all() + + # Assertions + simple_store.client.scan_iter.assert_called_once_with(match=f"{simple_store.key}:*") + assert simple_store.retrieve.call_count == 2 + assert len(results) == 2 + assert run1 in results + assert run2 in results + + def test_delete_existing_object( + self, + mocker: MockerFixture, + test_models: FixtureRedis, + simple_store: RedisStoreBase, + ): + """ + Test deleting an existing object from Redis. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + simple_store: A fixture providing a RedisStoreBase instance. + """ + run = test_models["run"] + mocker.patch.object(simple_store, "retrieve", return_value=run) + mocker.patch("merlin.backends.redis.redis_store_base.get_not_found_error_class", return_value=RunNotFoundError) + + simple_store.delete(run.id) + + simple_store.retrieve.assert_called_once_with(run.id) + simple_store.client.delete.assert_called_once_with(f"{simple_store.key}:{run.id}") + + def test_delete_nonexistent_object( + self, + mocker: MockerFixture, + simple_store: RedisStoreBase, + ): + """ + Test deleting a non-existent object from Redis. + + Args: + mocker: PyTest mocker fixture. + simple_store: A fixture providing a RedisStoreBase instance. + """ + mocker.patch.object(simple_store, "retrieve", return_value=None) + mocker.patch("merlin.backends.redis.redis_store_base.get_not_found_error_class", return_value=RunNotFoundError) + + with pytest.raises(RunNotFoundError): + simple_store.delete("nonexistent_id") + + simple_store.retrieve.assert_called_once_with("nonexistent_id") + simple_store.client.delete.assert_not_called() + + +class TestNameMappingMixin: + """Tests for the NameMappingMixin class.""" + + @pytest.fixture + def name_mapping_store(self, mock_redis: FixtureRedis) -> RedisStoreBase: + """ + Create a store with NameMappingMixin for testing. + + Args: + mock_redis: A fixture providing a mocked Redis client. + + Returns: + A store instance that implements NameMappingMixin. + """ + + # Create a test class that uses the NameMappingMixin + class TestStore(NameMappingMixin, RedisStoreBase): + pass + + store = TestStore(mock_redis, "test", StudyModel) + return store + + def test_save_new_object( + self, + mocker: MockerFixture, + test_models: FixtureDict, + name_mapping_store: RedisStoreBase, + ): + """ + Test saving a new object with name mapping. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + name_mapping_store: A fixture providing a store with NameMappingMixin. + """ + # Setup + study = test_models["study"] + name_mapping_store.client.hget.return_value = None # No existing object with this name + + # Mock the parent class's save method + mocker.patch.object(RedisStoreBase, "save") + + # Call the method + name_mapping_store.save(study) + + # Assertions + name_mapping_store.client.hget.assert_called_once_with(f"{name_mapping_store.key}:name", study.name) + RedisStoreBase.save.assert_called_once_with(study) + name_mapping_store.client.hset.assert_called_once_with(f"{name_mapping_store.key}:name", study.name, study.id) + + def test_save_existing_object( + self, + mocker: MockerFixture, + test_models: FixtureDict, + name_mapping_store: RedisStoreBase, + ): + """ + Test updating an existing object with name mapping. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + name_mapping_store: A fixture providing a store with NameMappingMixin. + """ + # Setup + study = test_models["study"] + name_mapping_store.client.hget.return_value = study.id # Existing object with this name + + # Mock the parent class's save method + mocker.patch.object(RedisStoreBase, "save") + + # Call the method + name_mapping_store.save(study) + + # Assertions + name_mapping_store.client.hget.assert_called_once_with(f"{name_mapping_store.key}:name", study.name) + RedisStoreBase.save.assert_called_once_with(study) + name_mapping_store.client.hset.assert_not_called() # Should not update name mapping + + def test_retrieve_by_id( + self, + mocker: MockerFixture, + test_models: FixtureDict, + name_mapping_store: RedisStoreBase, + ): + """ + Test retrieving an object by ID with name mapping. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + name_mapping_store: A fixture providing a store with NameMappingMixin. + """ + # Setup + study = test_models["study"] + + # Mock the parent class's retrieve method + mocker.patch.object(RedisStoreBase, "retrieve", return_value=study) + + # Call the method + result = name_mapping_store.retrieve(study.id, by_name=False) + + # Assertions + RedisStoreBase.retrieve.assert_called_once_with(study.id) + assert result == study + + def test_retrieve_by_name_existing( + self, + mocker: MockerFixture, + test_models: FixtureDict, + name_mapping_store: RedisStoreBase, + ): + """ + Test retrieving an existing object by name with name mapping. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + name_mapping_store: A fixture providing a store with NameMappingMixin. + """ + # Setup + study = test_models["study"] + name_mapping_store.client.hget.return_value = study.id + + # Mock the parent class's retrieve method + mocker.patch.object(RedisStoreBase, "retrieve", return_value=study) + + # Call the method + result = name_mapping_store.retrieve(study.name, by_name=True) + + # Assertions + name_mapping_store.client.hget.assert_called_once_with(f"{name_mapping_store.key}:name", study.name) + RedisStoreBase.retrieve.assert_called_once_with(study.id) + assert result == study + + def test_retrieve_by_name_nonexistent(self, name_mapping_store: RedisStoreBase): + """ + Test retrieving a non-existent object by name with name mapping. + + Args: + name_mapping_store: A fixture providing a store with NameMappingMixin. + """ + # Setup + name_mapping_store.client.hget.return_value = None + + # Call the method + result = name_mapping_store.retrieve("nonexistent_name", by_name=True) + + # Assertions + name_mapping_store.client.hget.assert_called_once_with(f"{name_mapping_store.key}:name", "nonexistent_name") + assert result is None + + def test_delete_by_id( + self, + mocker: MockerFixture, + test_models: FixtureDict, + name_mapping_store: RedisStoreBase, + ): + """ + Test deleting an object by ID with name mapping. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + name_mapping_store: A fixture providing a store with NameMappingMixin. + """ + study = test_models["study"] + + # Patch get_not_found_error_class at the correct location + mocker.patch("merlin.backends.redis.redis_store_base.get_not_found_error_class", return_value=StudyNotFoundError) + + # Patch retrieve to call the parent method (simulate as if the object exists) + mocker.patch.object(NameMappingMixin, "retrieve", return_value=study) + + # Call the method + name_mapping_store.delete(study.id, by_name=False) + + # Assertions + NameMappingMixin.retrieve.assert_called_once_with(study.id, by_name=False) + name_mapping_store.client.hdel.assert_called_once_with(f"{name_mapping_store.key}:name", study.name) + name_mapping_store.client.delete.assert_called_once_with(f"{name_mapping_store.key}:{study.id}") + + def test_delete_by_name( + self, + mocker: MockerFixture, + test_models: FixtureDict, + name_mapping_store: RedisStoreBase, + ): + """ + Test deleting an object by name with name mapping. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + name_mapping_store: A fixture providing a store with NameMappingMixin. + """ + study = test_models["study"] + + # Patch get_not_found_error_class at the correct location + mocker.patch("merlin.backends.redis.redis_store_base.get_not_found_error_class", return_value=StudyNotFoundError) + + # Patch retrieve to call the parent method (simulate as if the object exists) + mocker.patch.object(NameMappingMixin, "retrieve", return_value=study) + + # Call the method + name_mapping_store.delete(study.name, by_name=True) + + # Assertions + NameMappingMixin.retrieve.assert_called_once_with(study.name, by_name=True) + name_mapping_store.client.hdel.assert_called_once_with(f"{name_mapping_store.key}:name", study.name) + name_mapping_store.client.delete.assert_called_once_with(f"{name_mapping_store.key}:{study.id}") + + def test_delete_nonexistent_object(self, mocker: MockerFixture, name_mapping_store: RedisStoreBase): + """ + Test deleting a non-existent object with name mapping. + + Args: + mocker: PyTest mocker fixture. + name_mapping_store: A fixture providing a store with NameMappingMixin. + """ + # Patch get_not_found_error_class at the correct location + mocker.patch("merlin.backends.redis.redis_store_base.get_not_found_error_class", return_value=StudyNotFoundError) + + # Patch retrieve to return None + mocker.patch.object(NameMappingMixin, "retrieve", return_value=None) + + # Call the method and assert it raises the correct exception + with pytest.raises(StudyNotFoundError): + name_mapping_store.delete("nonexistent_name", by_name=True) + + # Assertions + NameMappingMixin.retrieve.assert_called_once_with("nonexistent_name", by_name=True) + name_mapping_store.client.hdel.assert_not_called() + name_mapping_store.client.delete.assert_not_called() diff --git a/tests/unit/backends/redis/test_redis_stores.py b/tests/unit/backends/redis/test_redis_stores.py new file mode 100644 index 000000000..80e33ec1d --- /dev/null +++ b/tests/unit/backends/redis/test_redis_stores.py @@ -0,0 +1,86 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `merlin/backends/redis/redis_stores.py` module. +""" + +from merlin.backends.redis.redis_stores import ( + RedisLogicalWorkerStore, + RedisPhysicalWorkerStore, + RedisRunStore, + RedisStudyStore, +) +from merlin.db_scripts.data_models import LogicalWorkerModel, PhysicalWorkerModel, RunModel, StudyModel +from tests.fixture_types import FixtureRedis + + +class TestRedisLogicalWorkerStore: + """Tests for the RedisLogicalWorkerStore implementation.""" + + def test_initialization(self, mock_redis: FixtureRedis): + """ + Test that the store initializes with the correct parameters. + + Args: + mock_redis: A fixture providing a mocked Redis client. + """ + store = RedisLogicalWorkerStore(mock_redis) + + assert store.client == mock_redis + assert store.key == "logical_worker" + assert store.model_class == LogicalWorkerModel + + +class TestRedisPhysicalWorkerStore: + """Tests for the RedisPhysicalWorkerStore implementation.""" + + def test_initialization(self, mock_redis: FixtureRedis): + """ + Test that the store initializes with the correct parameters. + + Args: + mock_redis: A fixture providing a mocked Redis client. + """ + store = RedisPhysicalWorkerStore(mock_redis) + + assert store.client == mock_redis + assert store.key == "physical_worker" + assert store.model_class == PhysicalWorkerModel + + +class TestRedisRunStore: + """Tests for the RedisRunStore implementation.""" + + def test_initialization(self, mock_redis: FixtureRedis): + """ + Test that the store initializes with the correct parameters. + + Args: + mock_redis: A fixture providing a mocked Redis client. + """ + store = RedisRunStore(mock_redis) + + assert store.client == mock_redis + assert store.key == "run" + assert store.model_class == RunModel + + +class TestRedisStudyStore: + """Tests for the RedisStudyStore implementation.""" + + def test_initialization(self, mock_redis: FixtureRedis): + """ + Test that the store initializes with the correct parameters. + + Args: + mock_redis: A fixture providing a mocked Redis client. + """ + store = RedisStudyStore(mock_redis) + + assert store.client == mock_redis + assert store.key == "study" + assert store.model_class == StudyModel diff --git a/tests/unit/backends/sqlite/test_sqlite_backend.py b/tests/unit/backends/sqlite/test_sqlite_backend.py new file mode 100644 index 000000000..d9f16cf23 --- /dev/null +++ b/tests/unit/backends/sqlite/test_sqlite_backend.py @@ -0,0 +1,88 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `sqlite_backend.py` module. +""" + +import pytest +from pytest_mock import MockerFixture + +from merlin.backends.sqlite.sqlite_backend import SQLiteBackend + + +@pytest.fixture +def sqlite_backend_instance(mocker: MockerFixture) -> SQLiteBackend: + """ + Fixture to create a `SQLiteBackend` instance with mocked dependencies. + + Args: + mocker (MockerFixture): The pytest-mock fixture used for mocking. + + Returns: + SQLiteBackend: An instance with mocked store methods and schema creation. + """ + # Patch the SQLite store classes to return mocks + mocker.patch("merlin.backends.sqlite.sqlite_backend.SQLiteStudyStore", return_value=mocker.MagicMock()) + mocker.patch("merlin.backends.sqlite.sqlite_backend.SQLiteRunStore", return_value=mocker.MagicMock()) + mocker.patch("merlin.backends.sqlite.sqlite_backend.SQLiteLogicalWorkerStore", return_value=mocker.MagicMock()) + mocker.patch("merlin.backends.sqlite.sqlite_backend.SQLitePhysicalWorkerStore", return_value=mocker.MagicMock()) + + # Patch the initialization method to avoid real DB operations + mocker.patch.object(SQLiteBackend, "_initialize_schema", return_value=None) + + backend = SQLiteBackend("sqlite") + return backend + + +class TestSQLiteBackend: + """ + Test suite for the `SQLiteBackend` class. + + Validates core functionality such as retrieving version information and flushing the database. + SQLite connections are mocked to prevent real filesystem or database interactions. + """ + + def test_get_version(self, mocker: MockerFixture, sqlite_backend_instance: SQLiteBackend): + """ + Test that SQLiteBackend correctly retrieves the SQLite version. + + Args: + sqlite_backend_instance (SQLiteBackend): The mocked backend. + """ + mock_conn = mocker.MagicMock() + mock_cursor = mocker.MagicMock() + mock_cursor.fetchone.return_value = ["3.43.2"] + mock_conn.execute.return_value = mock_cursor + + # Patch context manager to return mock connection + mocker.patch("merlin.backends.sqlite.sqlite_backend.SQLiteConnection", return_value=mock_conn) + mock_conn.__enter__.return_value = mock_conn + + version = sqlite_backend_instance.get_version() + assert version == "3.43.2", "SQLite version should be correctly retrieved." + + def test_flush_database(self, mocker: MockerFixture, sqlite_backend_instance: SQLiteBackend): + """ + Test that SQLiteBackend flushes the database by dropping all tables. + + Args: + sqlite_backend_instance (SQLiteBackend): The mocked backend. + """ + mock_conn = mocker.MagicMock() + mock_cursor = mocker.MagicMock() + mock_cursor.fetchall.return_value = [("table1",), ("table2",)] + mock_conn.execute.side_effect = lambda sql: mock_cursor if "SELECT name" in sql else None + + # Patch context manager to return mock connection + mocker.patch("merlin.backends.sqlite.sqlite_backend.SQLiteConnection", return_value=mock_conn) + mock_conn.__enter__.return_value = mock_conn + + sqlite_backend_instance.flush_database() + + # Ensure DROP TABLE commands were called for the expected tables + mock_conn.execute.assert_any_call("DROP TABLE IF EXISTS table1") + mock_conn.execute.assert_any_call("DROP TABLE IF EXISTS table2") diff --git a/tests/unit/backends/sqlite/test_sqlite_connection.py b/tests/unit/backends/sqlite/test_sqlite_connection.py new file mode 100644 index 000000000..82f1ba329 --- /dev/null +++ b/tests/unit/backends/sqlite/test_sqlite_connection.py @@ -0,0 +1,92 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `sqlite_connection.py` module. +""" + +import os +import sqlite3 +import sys +from unittest.mock import MagicMock + +import pytest +from pytest_mock import MockerFixture + +from merlin.backends.sqlite.sqlite_store_base import SQLiteConnection +from tests.fixture_types import FixtureDict + + +@pytest.fixture +def mock_sqlite_components(mocker: MockerFixture) -> FixtureDict[str, MagicMock]: + """ + Fixture to patch all external dependencies used by SQLiteConnection. + + Args: + mocker: PyTest mocker fixture. + + Returns: + A dictionary of mocked sqlite connection components. + """ + # Patch get_connection_string to return a fake path + mock_conn_str = os.path.join("tmp", "fake", "merlin.db") + mock_get_conn_str = mocker.patch("merlin.config.results_backend.get_connection_string", return_value=mock_conn_str) + + # Patch Path.mkdir so it doesn't touch the filesystem + mock_mkdir = mocker.patch("pathlib.Path.mkdir") + + # Patch sqlite3.connect + mock_conn = MagicMock(spec=sqlite3.Connection) + mock_connect = mocker.patch("sqlite3.connect", return_value=mock_conn) + + return { + "mock_conn_str": mock_conn_str, + "mock_get_conn_str": mock_get_conn_str, + "mock_mkdir": mock_mkdir, + "mock_connect": mock_connect, + "mock_conn": mock_conn, + } + + +def test_connection_enter_sets_up_connection(mock_sqlite_components: FixtureDict[str, MagicMock]): + """ + Test that `__enter__` correctly initializes the SQLite connection with expected settings. + + Args: + mock_sqlite_components: A dictionary of mocked sqlite connection components. + """ + conn_mock = mock_sqlite_components["mock_conn"] + conn_mock.execute = MagicMock() + + with SQLiteConnection() as conn: + assert conn is conn_mock + + mock_sqlite_components["mock_get_conn_str"].assert_called_once() + autocommit_kwargs = {"autocommit": True} if sys.version_info >= (3, 12) else {"isolation_level": None} + mock_sqlite_components["mock_connect"].assert_called_once_with( + mock_sqlite_components["mock_conn_str"], + check_same_thread=False, + **autocommit_kwargs, + ) + conn_mock.execute.assert_any_call("PRAGMA journal_mode=WAL") + conn_mock.execute.assert_any_call("PRAGMA foreign_keys=ON") + assert conn_mock.row_factory == sqlite3.Row + + +def test_connection_exit_closes_connection(mock_sqlite_components: FixtureDict[str, MagicMock]): + """ + Test that `__exit__` closes the SQLite connection. + + Args: + mock_sqlite_components: A dictionary of mocked sqlite connection components. + """ + conn_mock = mock_sqlite_components["mock_conn"] + sqlite_conn = SQLiteConnection() + sqlite_conn.conn = conn_mock + + sqlite_conn.__exit__(None, None, None) + + conn_mock.close.assert_called_once() diff --git a/tests/unit/backends/sqlite/test_sqlite_store_base.py b/tests/unit/backends/sqlite/test_sqlite_store_base.py new file mode 100644 index 000000000..390333b7a --- /dev/null +++ b/tests/unit/backends/sqlite/test_sqlite_store_base.py @@ -0,0 +1,457 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `sqlite_store_base.py` module. +""" + +from datetime import datetime +from unittest.mock import MagicMock + +import pytest +from pytest_mock import MockerFixture + +from merlin.backends.sqlite.sqlite_store_base import SQLiteStoreBase +from merlin.db_scripts.data_models import RunModel +from merlin.exceptions import RunNotFoundError +from tests.fixture_types import FixtureDict, FixtureTuple + + +class TestSQLiteStoreBase: + """Tests for the SQLiteStoreBase class.""" + + @pytest.fixture + def simple_store(self, mocker: MockerFixture) -> SQLiteStoreBase: + """ + Create a simple store instance for testing. + + Args: + mocker: PyTest mocker fixture. + + Returns: + A simple SQLiteStoreBase implementation for testing. + """ + # Mock the create_table_if_not_exists method to avoid actual table creation + mocker.patch.object(SQLiteStoreBase, "create_table_if_not_exists") + + # Create a simple subclass of SQLiteStoreBase for testing + class TestStore(SQLiteStoreBase): + pass + + store = TestStore("test_table", RunModel) + return store + + def test_initialization(self, mocker: MockerFixture): + """ + Test that the store initializes correctly. + + Args: + mocker: PyTest mocker fixture. + """ + # Mock the create_table_if_not_exists method + mock_create_table = mocker.patch.object(SQLiteStoreBase, "create_table_if_not_exists") + + store = SQLiteStoreBase("test_table", RunModel) + + assert store.table_name == "test_table" + assert store.model_class == RunModel + mock_create_table.assert_called_once() + + def test_get_sqlite_type(self, simple_store: SQLiteStoreBase): + """ + Test the _get_sqlite_type method with various Python types. + + Args: + simple_store: A fixture providing a SQLiteStoreBase instance. + """ + # Test basic types + assert simple_store._get_sqlite_type(str) == "TEXT" + assert simple_store._get_sqlite_type(int) == "INTEGER" + assert simple_store._get_sqlite_type(float) == "REAL" + assert simple_store._get_sqlite_type(bool) == "INTEGER" + assert simple_store._get_sqlite_type(datetime) == "TEXT" + + # Test generic types (lists, dicts, sets) + from typing import Dict, List, Set + + assert simple_store._get_sqlite_type(List[str]) == "TEXT" + assert simple_store._get_sqlite_type(Dict[str, int]) == "TEXT" + assert simple_store._get_sqlite_type(Set[str]) == "TEXT" + + # Test unknown type (should default to TEXT) + class CustomType: + pass + + assert simple_store._get_sqlite_type(CustomType) == "TEXT" + + def testcreate_table_if_not_exists(self, mock_sqlite_connection: FixtureTuple[MagicMock]): + """ + Test the create_table_if_not_exists method. + + Args: + mock_sqlite_connection: Fixture providing mocked SQLite connection and cursor. + """ + # Grab the mocked connection from the SQLiteConnection fixture + mock_conn, _ = mock_sqlite_connection + + # Mock model class fields + mock_field1 = MagicMock() + mock_field1.name = "id" + mock_field1.type = str + mock_field2 = MagicMock() + mock_field2.name = "study_id" + mock_field2.type = str + + mock_model_class = MagicMock() + mock_model_class.get_class_fields.return_value = [mock_field1, mock_field2] + + # Create store (this will call create_table_if_not_exists) + SQLiteStoreBase("test_table", mock_model_class) + + # Verify the SQL execution + expected_sql = "CREATE TABLE IF NOT EXISTS test_table (id TEXT, study_id TEXT);" + mock_conn.execute.assert_called_once_with(expected_sql) + + def test_save_new_object( + self, + mocker: MockerFixture, + test_models: FixtureDict, # Assuming similar fixture structure + simple_store: SQLiteStoreBase, + mock_sqlite_connection: FixtureTuple[MagicMock], + ): + """ + Test saving a new object to SQLite. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + simple_store: A fixture providing a SQLiteStoreBase instance. + mock_sqlite_connection: Fixture providing mocked SQLite connection and cursor. + """ + run = test_models["run"] + + # Mock retrieve to return None (object doesn't exist) + mocker.patch.object(simple_store, "retrieve", return_value=None) + + # Grab the mocked connection from the SQLiteConnection fixture + mock_conn, _ = mock_sqlite_connection + + # Mock serialization + mock_serialize = mocker.patch( + "merlin.backends.sqlite.sqlite_store_base.serialize_entity", return_value={"id": run.id, "study_id": "study1"} + ) + + # Mock model class fields + mock_field1 = MagicMock() + mock_field1.name = "id" + mock_field2 = MagicMock() + mock_field2.name = "study_id" + mocker.patch.object(simple_store.model_class, "get_class_fields", return_value=[mock_field1, mock_field2]) + + simple_store.save(run) + + # Verify method calls + simple_store.retrieve.assert_called_once_with(run.id) + mock_serialize.assert_called_once_with(run) + mock_conn.execute.assert_called_once() + call_args = mock_conn.execute.call_args + # Check that the SQL contains the expected structure (whitespace may vary) + assert "INSERT INTO test_table" in call_args[0][0] + assert call_args[0][1] == {"id": run.id, "study_id": "study1"} + + def test_save_existing_object( + self, + mocker: MockerFixture, + test_models: FixtureDict, + simple_store: SQLiteStoreBase, + mock_sqlite_connection: FixtureTuple[MagicMock], + ): + """ + Test updating an existing object in SQLite. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + simple_store: A fixture providing a SQLiteStoreBase instance. + mock_sqlite_connection: Fixture providing mocked SQLite connection and cursor. + """ + run = test_models["run"] + + # Mock existing object + mock_existing = MagicMock() + mock_existing.update_fields = MagicMock() + mocker.patch.object(simple_store, "retrieve", return_value=mock_existing) + + # Grab the mocked connection from the SQLiteConnection fixture + mock_conn, _ = mock_sqlite_connection + + # Mock serialization + mock_serialize = mocker.patch( + "merlin.backends.sqlite.sqlite_store_base.serialize_entity", return_value={"id": run.id, "study_id": "study1"} + ) + + # Mock model class fields + mock_field1 = MagicMock() + mock_field1.name = "id" + mock_field2 = MagicMock() + mock_field2.name = "study_id" + mocker.patch.object(simple_store.model_class, "get_class_fields", return_value=[mock_field1, mock_field2]) + + simple_store.save(run) + + # Verify method calls + simple_store.retrieve.assert_called_once_with(run.id) + mock_existing.update_fields.assert_called_once_with(run.to_dict()) + mock_serialize.assert_called_once_with(mock_existing) + + # Verify UPDATE SQL was executed + mock_conn.execute.assert_called_once() + call_args = mock_conn.execute.call_args + assert "UPDATE test_table" in call_args[0][0] + assert "WHERE id = :id" in call_args[0][0] + + @pytest.mark.parametrize("by_name, identifier_key", [(False, "id"), (True, "name")]) + def test_retrieve_existing_object( + self, + mocker: MockerFixture, + test_models: FixtureDict, + simple_store: SQLiteStoreBase, + mock_sqlite_connection: FixtureTuple[MagicMock], + by_name: bool, + identifier_key: str, + ): + """ + Parametrized test for retrieving an existing object by ID or name from SQLite. + + Args: + mocker: PyTest mocker fixture. + test_models: Fixture providing test model instances. + simple_store: A SQLiteStoreBase instance. + mock_sqlite_connection: Fixture providing mocked SQLite connection and cursor. + by_name: Whether to retrieve by name (True) or by ID (False). + identifier_key: Either "id" or "name", depending on lookup method. + """ + run = test_models["run"] + identifier = "test_run" if by_name else run.id + + # Mock connection and cursor + mock_conn, mock_cursor = mock_sqlite_connection + mock_row = {"id": run.id, "study_id": "study1", "name": "test_run"} + mock_cursor.fetchone.return_value = mock_row + + # Mock deserialization + mock_deserialize = mocker.patch("merlin.backends.sqlite.sqlite_store_base.deserialize_entity", return_value=run) + + result = simple_store.retrieve(identifier, by_name=by_name) + + mock_conn.execute.assert_called_once_with( + f"SELECT * FROM test_table WHERE {identifier_key} = :identifier", {"identifier": identifier} + ) + mock_cursor.fetchone.assert_called_once() + mock_deserialize.assert_called_once_with(mock_row, simple_store.model_class) + assert result == run + + def test_retrieve_nonexistent_object( + self, + simple_store: SQLiteStoreBase, + mock_sqlite_connection: FixtureTuple[MagicMock], + ): + """ + Test retrieving a non-existent object from SQLite. + + Args: + simple_store: A fixture providing a SQLiteStoreBase instance. + mock_sqlite_connection: Fixture providing mocked SQLite connection and cursor. + """ + # Mock SQLiteConnection and cursor + mock_conn, mock_cursor = mock_sqlite_connection + mock_cursor.fetchone.return_value = None + + result = simple_store.retrieve("nonexistent_id") + + # Verify method calls + mock_conn.execute.assert_called_once_with( + "SELECT * FROM test_table WHERE id = :identifier", {"identifier": "nonexistent_id"} + ) + mock_cursor.fetchone.assert_called_once() + assert result is None + + def test_retrieve_all( + self, + mocker: MockerFixture, + test_models: FixtureDict, + simple_store: SQLiteStoreBase, + mock_sqlite_connection: FixtureTuple[MagicMock], + ): + """ + Test retrieving all objects from SQLite. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + simple_store: A fixture providing a SQLiteStoreBase instance. + mock_sqlite_connection: Fixture providing mocked SQLite connection and cursor. + """ + run1 = test_models["run"] + run2 = RunModel(id="run2", study_id="study1") + + # Mock SQLiteConnection and cursor + mock_conn, mock_cursor = mock_sqlite_connection + mock_rows = [{"id": run1.id, "study_id": "study1"}, {"id": run2.id, "study_id": "study1"}] + mock_cursor.fetchall.return_value = mock_rows + + # Mock deserialization + mock_deserialize = mocker.patch( + "merlin.backends.sqlite.sqlite_store_base.deserialize_entity", side_effect=[run1, run2] + ) + + results = simple_store.retrieve_all() + + # Verify method calls + mock_conn.execute.assert_called_once_with("SELECT * FROM test_table") + mock_cursor.fetchall.assert_called_once() + assert mock_deserialize.call_count == 2 + assert len(results) == 2 + assert run1 in results + assert run2 in results + + def test_retrieve_all_with_deserialization_error( + self, + mocker: MockerFixture, + simple_store: SQLiteStoreBase, + mock_sqlite_connection: FixtureTuple[MagicMock], + ): + """ + Test retrieving all objects when deserialization fails for some objects. + + Args: + mocker: PyTest mocker fixture. + simple_store: A fixture providing a SQLiteStoreBase instance. + mock_sqlite_connection: Fixture providing mocked SQLite connection and cursor. + """ + # Mock SQLiteConnection and cursor + _, mock_cursor = mock_sqlite_connection + mock_rows = [{"id": "run1", "study_id": "study1"}, {"id": "run2", "study_id": "study1"}] # This one will fail + mock_cursor.fetchall.return_value = mock_rows + + # Mock deserialization - first succeeds, second fails + run1 = RunModel(id="run1", study_id="study1") + mocker.patch( + "merlin.backends.sqlite.sqlite_store_base.deserialize_entity", + side_effect=[run1, Exception("Deserialization error")], + ) + + results = simple_store.retrieve_all() + + # Verify that only the successful object is returned + assert len(results) == 1 + assert results[0] == run1 + + @pytest.mark.parametrize( + "identifier, by_name, identifier_key", + [ + ("run1", False, "id"), + ("test_run", True, "name"), + ], + ) + def test_delete_existing_object( + self, + mocker: MockerFixture, + test_models: FixtureDict, + simple_store: SQLiteStoreBase, + mock_sqlite_connection: FixtureTuple[MagicMock], + identifier: str, + by_name: bool, + identifier_key: str, + ): + """ + Parametrized test for deleting an existing object from SQLite + by ID or name. + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + simple_store: A fixture providing a SQLiteStoreBase instance. + mock_sqlite_connection: Fixture providing mocked SQLite connection and cursor. + identifier: The value used to identify the object (id or name). + by_name: Whether the deletion is by name or by id. + identifier_key: The column to delete on ("id" or "name"). + """ + run = test_models["run"] + + # Mock retrieve to return the object + mocker.patch.object(simple_store, "retrieve", return_value=run) + + # Mock SQLiteConnection and cursor + mock_conn, mock_cursor = mock_sqlite_connection + mock_cursor.rowcount = 1 + + # Mock error class + mocker.patch("merlin.backends.sqlite.sqlite_store_base.get_not_found_error_class", return_value=RunNotFoundError) + + simple_store.delete(identifier, by_name=by_name) + + # Verify calls + simple_store.retrieve.assert_called_once_with(identifier, by_name=by_name) + mock_conn.execute.assert_called_once_with( + f"DELETE FROM test_table WHERE {identifier_key} = :identifier", {"identifier": identifier} + ) + + def test_delete_nonexistent_object( + self, + mocker: MockerFixture, + simple_store: SQLiteStoreBase, + ): + """ + Test deleting a non-existent object from SQLite. + + Args: + mocker: PyTest mocker fixture. + simple_store: A fixture providing a SQLiteStoreBase instance. + """ + # Mock retrieve to return None + mocker.patch.object(simple_store, "retrieve", return_value=None) + + # Mock error class + mocker.patch("merlin.backends.sqlite.sqlite_store_base.get_not_found_error_class", return_value=RunNotFoundError) + + with pytest.raises(RunNotFoundError, match="Test_table with id 'nonexistent_id' does not exist in the database."): + simple_store.delete("nonexistent_id") + + # Verify that retrieve was called but execute was not + simple_store.retrieve.assert_called_once_with("nonexistent_id", by_name=False) + + def test_delete_with_zero_rowcount( + self, + mocker: MockerFixture, + test_models: FixtureDict, + simple_store: SQLiteStoreBase, + mock_sqlite_connection: FixtureTuple[MagicMock], + ): + """ + Test deleting an object that exists during retrieve but fails to delete (rowcount = 0). + + Args: + mocker: PyTest mocker fixture. + test_models: A fixture providing test model instances. + simple_store: A fixture providing a SQLiteStoreBase instance. + mock_sqlite_connection: Fixture providing mocked SQLite connection and cursor. + """ + run = test_models["run"] + + # Mock retrieve to return the object + mocker.patch.object(simple_store, "retrieve", return_value=run) + + # Mock SQLiteConnection and cursor + mock_conn, mock_cursor = mock_sqlite_connection + mock_cursor.rowcount = 0 + + # This should not raise an exception, just log a warning + simple_store.delete(run.id) + + # Verify method calls + simple_store.retrieve.assert_called_once_with(run.id, by_name=False) + mock_conn.execute.assert_called_once_with("DELETE FROM test_table WHERE id = :identifier", {"identifier": run.id}) diff --git a/tests/unit/backends/sqlite/test_sqlite_stores.py b/tests/unit/backends/sqlite/test_sqlite_stores.py new file mode 100644 index 000000000..7d84ba629 --- /dev/null +++ b/tests/unit/backends/sqlite/test_sqlite_stores.py @@ -0,0 +1,96 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `merlin/backends/sqlite/sqlite_stores.py` module. +""" + +import pytest +from pytest_mock import MockerFixture + +from merlin.backends.sqlite.sqlite_stores import ( + SQLiteLogicalWorkerStore, + SQLitePhysicalWorkerStore, + SQLiteRunStore, + SQLiteStudyStore, +) +from merlin.db_scripts.data_models import LogicalWorkerModel, PhysicalWorkerModel, RunModel, StudyModel + + +@pytest.fixture +def mock_create_table(mocker: MockerFixture): + """ + A fixture to mock the `create_table_if_not_exists` method so we don't actually + try to create a connection to the database. + + Args: + mocker: PyTest mocker fixture. + """ + mocker.patch("merlin.backends.sqlite.sqlite_store_base.SQLiteStoreBase.create_table_if_not_exists") + + +class TestSQLiteLogicalWorkerStore: + """Tests for the SQLiteLogicalWorkerStore implementation.""" + + def test_initialization(self, mock_create_table): + """ + Test that the store initializes with the correct attributes. + + Args: + mock_create_table: Fixture that patches the `create_table_if_not_exists` method. + """ + store = SQLiteLogicalWorkerStore() + + assert store.table_name == "logical_worker" + assert store.model_class == LogicalWorkerModel + + +class TestSQLitePhysicalWorkerStore: + """Tests for the SQLitePhysicalWorkerStore implementation.""" + + def test_initialization(self, mock_create_table): + """ + Test that the store initializes with the correct attributes. + + Args: + mock_create_table: Fixture that patches the `create_table_if_not_exists` method. + """ + store = SQLitePhysicalWorkerStore() + + assert store.table_name == "physical_worker" + assert store.model_class == PhysicalWorkerModel + + +class TestSQLiteRunStore: + """Tests for the SQLiteRunStore implementation.""" + + def test_initialization(self, mock_create_table): + """ + Test that the store initializes with the correct attributes. + + Args: + mock_create_table: Fixture that patches the `create_table_if_not_exists` method. + """ + store = SQLiteRunStore() + + assert store.table_name == "run" + assert store.model_class == RunModel + + +class TestSQLiteStudyStore: + """Tests for the SQLiteStudyStore implementation.""" + + def test_initialization(self, mock_create_table): + """ + Test that the store initializes with the correct attributes. + + Args: + mock_create_table: Fixture that patches the `create_table_if_not_exists` method. + """ + store = SQLiteStudyStore() + + assert store.table_name == "study" + assert store.model_class == StudyModel diff --git a/tests/unit/backends/test_backend_factory.py b/tests/unit/backends/test_backend_factory.py new file mode 100644 index 000000000..4f51ae4f7 --- /dev/null +++ b/tests/unit/backends/test_backend_factory.py @@ -0,0 +1,86 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `backend_factory.py` module. +""" + +import pytest +from pytest_mock import MockerFixture + +from merlin.backends.backend_factory import backend_factory +from merlin.backends.redis.redis_backend import RedisBackend +from merlin.exceptions import BackendNotSupportedError + + +class TestBackendFactory: + """ + Test suite for the `backend_factory` module. + + This class contains unit tests to validate the functionality of the `backend_factory`, which is responsible + for managing backend instances and providing an interface for retrieving supported backends and resolving + backend aliases. + + Fixtures and mocking are used to isolate the tests from the actual backend implementations, ensuring that + the tests focus on the behavior of the `backend_factory` module. + + These tests ensure the robustness and correctness of the `backend_factory` module, which is critical for + backend management in the Merlin framework. + """ + + def test_get_supported_backends(self): + """ + Test that `get_supported_backends` returns the correct list of supported backends. + """ + supported_backends = backend_factory.get_supported_backends() + assert supported_backends == ["redis", "sqlite"] + + def test_get_backend_with_valid_backend(self, mocker: MockerFixture): + """ + Test that `get_backend` returns the correct backend instance for a valid backend. + + Args: + mocker (MockerFixture): A built-in fixture from the pytest-mock library to create a Mock object. + """ + backend_name = "redis" + + # Mock the RedisBackend class to avoid instantiation issues + RedisBackendMock = mocker.MagicMock(spec=RedisBackend) + backend_factory._backends["redis"] = RedisBackendMock + + backend_instance = backend_factory.get_backend(backend_name) + + # Verify the backend instance is created correctly + RedisBackendMock.assert_called_once_with(backend_name) + assert backend_instance == RedisBackendMock(backend_name), "Backend instance should match the mocked backend." + + def test_get_backend_with_alias(self, mocker: MockerFixture): + """ + Test that `get_backend` correctly resolves aliases to canonical backend names. + + Args: + mocker (MockerFixture): A built-in fixture from the pytest-mock library to create a Mock object. + """ + alias_name = "rediss" + + # Mock the RedisBackend class to avoid instantiation issues + RedisBackendMock = mocker.MagicMock(spec=RedisBackend) + backend_factory._backends["redis"] = RedisBackendMock + + backend_instance = backend_factory.get_backend(alias_name) + + # Verify the alias resolves and the backend instance is created correctly + RedisBackendMock.assert_called_once_with("redis") + assert backend_instance == RedisBackendMock("redis"), "Backend instance should match the mocked backend." + + def test_get_backend_with_invalid_backend(self): + """ + Test that get_backend raises BackendNotSupportedError for an unsupported backend. + """ + invalid_backend_name = "unsupported_backend" + + with pytest.raises(BackendNotSupportedError, match=f"Backend unsupported by Merlin: {invalid_backend_name}."): + backend_factory.get_backend(invalid_backend_name) diff --git a/tests/unit/backends/test_results_backend.py b/tests/unit/backends/test_results_backend.py new file mode 100644 index 000000000..e38ca32fa --- /dev/null +++ b/tests/unit/backends/test_results_backend.py @@ -0,0 +1,344 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `results_backend.py` file. +""" + +import uuid + +import pytest +from pytest_mock import MockerFixture + +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.data_models import BaseDataModel, LogicalWorkerModel, PhysicalWorkerModel, RunModel, StudyModel +from merlin.exceptions import UnsupportedDataModelError +from tests.fixture_types import FixtureStr + + +@pytest.fixture() +def results_backend_test_name() -> FixtureStr: + """ + Defines a specific name to use for the results backend tests. This helps ensure + that even if changes were made to the tests, as long as this fixture is still used + the tests should expect the same backend name. + + Returns: + A string representing the name of the test backend. + """ + return "test-backend" + + +@pytest.fixture() +def results_backend_test_instance(mocker: MockerFixture, results_backend_test_name: FixtureStr) -> ResultsBackend: + """ + Provides a concrete test instance of the `ResultsBackend` class for use in tests. + + This fixture dynamically creates a subclass of `ResultsBackend` called `Test` and + overrides its abstract methods, allowing it to be instantiated. The abstract methods + are bypassed by setting the `__abstractmethods__` attribute to an empty frozenset. + The resulting instance is initialized with the provided `results_backend_test_name`. + + Args: + results_backend_test_name (FixtureStr): A fixture that provides the name of the + results backend to be used in the test. + + Returns: + A concrete instance of the `ResultsBackend` class for testing purposes. + """ + + class Test(ResultsBackend): + def __init__(self, backend_name: str): + stores = { + "study": mocker.MagicMock(), + "run": mocker.MagicMock(), + "logical_worker": mocker.MagicMock(), + "physical_worker": mocker.MagicMock(), + } + super().__init__(backend_name, stores) + + Test.__abstractmethods__ = frozenset() + return Test(results_backend_test_name) + + +class TestResultsBackend: + """ + Test suite for the `ResultsBackend` class. + + This class contains unit tests to validate the behavior of the `ResultsBackend` class, which serves as an + abstract base class for implementing backends that manage entity storage and retrieval in the Merlin framework. + + The tests are divided into two categories: + 1. **Concrete Method Tests**: + - Validates the behavior of implemented methods, such as `get_name`, ensuring they return the expected values. + + 2. **Abstract Method Tests**: + - Ensures that abstract methods (`get_version`, `get_connection_string`, `flush_database`, `save`, `retrieve`, + `retrieve_all`, and `delete`) raise `NotImplementedError` when invoked without being implemented in a subclass. + + Fixtures are used to provide test instances of the `ResultsBackend` class and mock objects where necessary. + This ensures the tests focus on the correctness of the abstract base class and its contract for subclasses. + + These tests ensure the integrity of the `ResultsBackend` class as a foundational component for entity storage + and retrieval in the framework. + """ + + ######################### + # Concrete Method Tests # + ######################### + + def test_get_name(self, results_backend_test_instance: ResultsBackend, results_backend_test_name: FixtureStr): + """ + Test the `get_name` method to ensure it returns the correct value. + + Args: + results_backend_test_instance (ResultsBackend): A fixture that provides a test instance + of the `ResultsBackend` class. + results_backend_test_name (FixtureStr): A fixture that provides the name of the + results backend to be used in the test. + """ + assert ( + results_backend_test_instance.get_name() == results_backend_test_name + ), f"get_name should return {results_backend_test_name}" + + ######################### + # Abstract Method Tests # + ######################### + + def test_get_version_raises_exception_if_not_implemented(self, results_backend_test_instance: ResultsBackend): + """ + Test that the `get_version` abstract method raises an exception if it's not implemented. + + Args: + results_backend_test_instance (ResultsBackend): A fixture that provides a test instance + of the `ResultsBackend` class. + """ + with pytest.raises(NotImplementedError): + results_backend_test_instance.get_version() + + def test_get_connection_string(self, mocker: MockerFixture, results_backend_test_instance: ResultsBackend): + """ + Test that the `get_connection_string` method makes the appropriate call. + + Args: + mocker (MockerFixture): A built-in fixture from the pytest-mock library to create a Mock object + results_backend_test_instance (ResultsBackend): A fixture that provides a test instance + of the `ResultsBackend` class. + """ + mock_get_conn_str = mocker.patch("merlin.config.results_backend.get_connection_string") + results_backend_test_instance.get_connection_string() + mock_get_conn_str.assert_called_once() + + def test_flush_database_raises_exception_if_not_implemented(self, results_backend_test_instance: ResultsBackend): + """ + Test that the `flush_database` abstract method raises an exception if it's not implemented. + + Args: + results_backend_test_instance (ResultsBackend): A fixture that provides a test instance + of the `ResultsBackend` class. + """ + with pytest.raises(NotImplementedError): + results_backend_test_instance.flush_database() + + @pytest.mark.parametrize( + "db_model, model_type", + [ + (StudyModel, "study"), + (RunModel, "run"), + (LogicalWorkerModel, "logical_worker"), + (PhysicalWorkerModel, "physical_worker"), + ], + ) + def test_save_valid_entity( + self, + mocker: MockerFixture, + results_backend_test_instance: ResultsBackend, + db_model: BaseDataModel, + model_type: str, + ): + """ + Test saving a valid entity to the Redis store. + + This is a parametrized test that ensures saving to the Redis store works for all valid model types. + Currently this includes: + - studies + - runs + - logical workers + - physical workers + + Args: + mocker (MockerFixture): A built-in fixture from the pytest-mock library to create a Mock object. + results_backend_test_instance (ResultsBackend): A fixture representing a `ResultsBackend` instance with + mocked Redis client and stores. + db_model (BaseDataModel): The database model class representing the entity type being tested. + model_type (str): A string identifier for the type of entity being tested. This corresponds to + the key used in the `ResultsBackend.stores` dictionary. + """ + entity = mocker.MagicMock(spec=db_model) + results_backend_test_instance.save(entity) + results_backend_test_instance.stores[model_type].save.assert_called_once_with(entity) + + def test_save_invalid_entity(self, mocker: MockerFixture, results_backend_test_instance: ResultsBackend): + """ + Test saving an invalid entity raises UnsupportedDataModelError. + + Args: + mocker (MockerFixture): A built-in fixture from the pytest-mock library to create a Mock object. + results_backend_test_instance (ResultsBackend): A fixture representing a `ResultsBackend` instance with + mocked Redis client and stores. + """ + invalid_entity = mocker.MagicMock(spec=object) # Not a subclass of BaseDataModel + with pytest.raises(UnsupportedDataModelError, match="Unsupported data model of type"): + results_backend_test_instance.save(invalid_entity) + + @pytest.mark.parametrize( + "db_model, model_type", + [ + (StudyModel, "study"), + (RunModel, "run"), + (LogicalWorkerModel, "logical_worker"), + (PhysicalWorkerModel, "physical_worker"), + ], + ) + def test_retrieve_valid_entity_by_id( + self, + mocker: MockerFixture, + results_backend_test_instance: ResultsBackend, + db_model: BaseDataModel, + model_type: str, + ): + """ + Test retrieving a valid entity from the Redis store by ID. + + Args: + mocker (MockerFixture): A built-in fixture from the pytest-mock library to create a Mock object. + results_backend_test_instance (ResultsBackend): A fixture representing a `ResultsBackend` instance with + mocked Redis client and stores. + db_model (BaseDataModel): The database model class representing the entity type being tested. + model_type (str): A string identifier for the type of entity being tested. This corresponds to + the key used in the `ResultsBackend.stores` dictionary. + """ + test_id = str(uuid.uuid4()) + results_backend_test_instance.stores[model_type].retrieve.return_value = mocker.MagicMock(spec=db_model) + entity = results_backend_test_instance.retrieve(test_id, model_type) + results_backend_test_instance.stores[model_type].retrieve.assert_called_once_with(test_id) + assert isinstance(entity, db_model), f"Retrieved entity should be of type {type(db_model)}." + + @pytest.mark.parametrize( + "db_model, model_type", + [(StudyModel, "study"), (PhysicalWorkerModel, "physical_worker")], + ) + def test_retrieve_valid_entity_by_name( + self, + mocker: MockerFixture, + results_backend_test_instance: ResultsBackend, + db_model: BaseDataModel, + model_type: str, + ): + """ + Test retrieving a valid entity from the Redis store by name. This test only applies to study and + physical worker entities. + + Args: + mocker (MockerFixture): A built-in fixture from the pytest-mock library to create a Mock object. + results_backend_test_instance (ResultsBackend): A fixture representing a `ResultsBackend` instance with + mocked Redis client and stores. + db_model (BaseDataModel): The database model class representing the entity type being tested. + model_type (str): A string identifier for the type of entity being tested. This corresponds to + the key used in the `ResultsBackend.stores` dictionary. + """ + test_name = "entity_name" + results_backend_test_instance.stores[model_type].retrieve.return_value = mocker.MagicMock(spec=db_model) + entity = results_backend_test_instance.retrieve(test_name, model_type) + results_backend_test_instance.stores[model_type].retrieve.assert_called_once_with(test_name, by_name=True) + assert isinstance(entity, db_model), f"Retrieved entity should be of type {type(db_model)}." + + def test_retrieve_invalid_store_type(self, results_backend_test_instance: ResultsBackend): + """ + Test retrieving from an invalid store type raises ValueError. + + Args: + results_backend_test_instance (ResultsBackend): A fixture representing a `ResultsBackend` instance with + mocked Redis client and stores. + """ + invalid_store_type = "invalid_store" + with pytest.raises(ValueError, match=f"Invalid store type '{invalid_store_type}'."): + results_backend_test_instance.retrieve(str(uuid.uuid4()), invalid_store_type) + + @pytest.mark.parametrize( + "db_model, model_type", + [ + (StudyModel, "study"), + (RunModel, "run"), + (LogicalWorkerModel, "logical_worker"), + (PhysicalWorkerModel, "physical_worker"), + ], + ) + def test_retrieve_all( + self, + mocker: MockerFixture, + results_backend_test_instance: ResultsBackend, + db_model: BaseDataModel, + model_type: str, + ): + """ + Test retrieving all entities from a valid store. + + Args: + mocker (MockerFixture): A built-in fixture from the pytest-mock library to create a Mock object. + results_backend_test_instance (ResultsBackend): A fixture representing a `ResultsBackend` instance with + mocked Redis client and stores. + db_model (BaseDataModel): The database model class representing the entity type being tested. + model_type (str): A string identifier for the type of entity being tested. This corresponds to + the key used in the `ResultsBackend.stores` dictionary. + """ + results_backend_test_instance.stores[model_type].retrieve_all.return_value = [mocker.MagicMock(spec=db_model)] + entities = results_backend_test_instance.retrieve_all(model_type) + results_backend_test_instance.stores[model_type].retrieve_all.assert_called_once() + assert len(entities) == 1, "Should retrieve one entity." + assert isinstance(entities[0], db_model), f"Retrieved entity should be of type {type(db_model)}." + + @pytest.mark.parametrize("model_type", ["study", "run", "logical_worker", "physical_worker"]) + def test_delete_valid_entity_by_id(self, results_backend_test_instance: ResultsBackend, model_type: str): + """ + Test deleting a valid entity from the Redis store by ID. + + Args: + results_backend_test_instance (ResultsBackend): A fixture representing a `ResultsBackend` instance with + mocked Redis client and stores. + model_type (str): A string identifier for the type of entity being tested. This corresponds to + the key used in the `ResultsBackend.stores` dictionary. + """ + test_id = str(uuid.uuid4()) + results_backend_test_instance.delete(test_id, model_type) + results_backend_test_instance.stores[model_type].delete.assert_called_once_with(test_id) + + @pytest.mark.parametrize("model_type", ["study", "physical_worker"]) + def test_delete_valid_entity_by_name(self, results_backend_test_instance: ResultsBackend, model_type: str): + """ + Test deleting a valid entity from the Redis store by name. + + Args: + results_backend_test_instance (ResultsBackend): A fixture representing a `ResultsBackend` instance with + mocked Redis client and stores. + model_type (str): A string identifier for the type of entity being tested. This corresponds to + the key used in the `ResultsBackend.stores` dictionary. + """ + test_name = "entity_name" + results_backend_test_instance.delete(test_name, model_type) + results_backend_test_instance.stores[model_type].delete.assert_called_once_with(test_name, by_name=True) + + def test_delete_invalid_store_type(self, results_backend_test_instance: ResultsBackend): + """ + Test deleting from an invalid store type raises ValueError. + + Args: + results_backend_test_instance (ResultsBackend): A fixture representing a `ResultsBackend` instance with + mocked Redis client and stores. + """ + invalid_store_type = "invalid_store" + with pytest.raises(ValueError, match=f"Invalid store type '{invalid_store_type}'."): + results_backend_test_instance.delete(str(uuid.uuid4()), invalid_store_type) diff --git a/tests/unit/backends/test_store_base.py b/tests/unit/backends/test_store_base.py new file mode 100644 index 000000000..732fe5d85 --- /dev/null +++ b/tests/unit/backends/test_store_base.py @@ -0,0 +1,66 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `store_base.py` module. +""" + +from typing import List + +import pytest + +from merlin.backends.store_base import StoreBase +from merlin.db_scripts.data_models import BaseDataModel + + +class DummyModel(BaseDataModel): + """A minimal model stub for testing.""" + + def __init__(self, id: str): + self.id = id + + +def test_store_base_is_abstract(): + """ + Test that StoreBase cannot be instantiated directly. + """ + with pytest.raises(TypeError): + StoreBase() # type: ignore + + +def test_incomplete_subclass_raises_type_error(): + """ + Test that a subclass that does not implement all abstract methods raises TypeError. + """ + + class IncompleteStore(StoreBase[DummyModel]): + def save(self, entity: DummyModel): + pass # only implements one of four required methods + + with pytest.raises(TypeError): + IncompleteStore() # type: ignore + + +def test_complete_subclass_can_be_instantiated(): + """ + Test that a subclass implementing all abstract methods can be instantiated. + """ + + class CompleteStore(StoreBase[DummyModel]): + def save(self, entity: DummyModel): + self._store = {entity.id: entity} + + def retrieve(self, entity_id: str) -> DummyModel: + return self._store[entity_id] + + def retrieve_all(self) -> List[DummyModel]: + return list(self._store.values()) + + def delete(self, entity_id: str): + del self._store[entity_id] + + store = CompleteStore() + assert isinstance(store, StoreBase) diff --git a/tests/unit/backends/test_utils.py b/tests/unit/backends/test_utils.py new file mode 100644 index 000000000..56f7ab037 --- /dev/null +++ b/tests/unit/backends/test_utils.py @@ -0,0 +1,47 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `backends/utils.py` module. +""" + +import pytest + +from merlin.backends.utils import get_not_found_error_class +from merlin.db_scripts.data_models import BaseDataModel, LogicalWorkerModel, PhysicalWorkerModel, RunModel, StudyModel +from merlin.exceptions import RunNotFoundError, StudyNotFoundError, WorkerNotFoundError + + +@pytest.mark.parametrize( + "db_model, expected_error", + [ + (StudyModel, StudyNotFoundError), + (RunModel, RunNotFoundError), + (LogicalWorkerModel, WorkerNotFoundError), + (PhysicalWorkerModel, WorkerNotFoundError), + ], +) +def test_get_not_found_error_class_valid_model(db_model: BaseDataModel, expected_error: Exception): + """ + Test the `get_not_found_error_class` function returns the correct error class. + + Args: + db_model: The type of model to get the error for. + error_type: The error that's supposed to be raised. + """ + assert get_not_found_error_class(db_model) is expected_error + + +def test_get_not_found_error_class_unknown_model(): + """ + Test the `get_not_found_error_class` function returns the broad Exception for + an unknown data model. + """ + + class UnknownModel: + pass + + assert get_not_found_error_class(UnknownModel) is Exception diff --git a/tests/unit/common/test_dumper.py b/tests/unit/common/test_dumper.py index c52e9fe90..7ef8beaa2 100644 --- a/tests/unit/common/test_dumper.py +++ b/tests/unit/common/test_dumper.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the `dumper.py` file. """ diff --git a/tests/unit/common/test_encryption.py b/tests/unit/common/test_encryption.py index 3e37cef84..30aad87a5 100644 --- a/tests/unit/common/test_encryption.py +++ b/tests/unit/common/test_encryption.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the `encrypt.py` and `encrypt_backend_traffic.py` files. """ @@ -17,34 +23,34 @@ class TestEncryption: This class will house all tests necessary for our encryption modules. """ - def test_encrypt(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_encrypt(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test that our encryption function is encrypting the bytes that we're passing to it. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ str_to_encrypt = b"super secret string shhh" encrypted_str = encrypt(str_to_encrypt) for word in str_to_encrypt.decode("utf-8").split(" "): assert word not in encrypted_str.decode("utf-8") - def test_decrypt(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_decrypt(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test that our decryption function is decrypting the bytes that we're passing to it. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ # This is the output of the bytes from the encrypt test str_to_decrypt = b"gAAAAABld6k-jEncgCW5AePgrwn-C30dhr7dzGVhqzcqskPqFyA2Hdg3VWmo0qQnLklccaUYzAGlB4PMxyp4T-1gAYlAOf_7sC_bJOEcYOIkhZFoH6cX4Uw=" decrypted_str = decrypt(str_to_decrypt) assert decrypted_str == b"super secret string shhh" - def test_get_key_path(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_get_key_path(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test the `_get_key_path` function. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ # Test the default behavior (`_get_key_path` will pull from CONFIG.results_backend which # will be set to the temporary output path for our tests in the `use_fake_encrypt_data_key` fixture) @@ -89,14 +95,17 @@ def test_gen_key(self, temp_output_dir: str): assert key_gen_contents != "" def test_get_key( - self, merlin_server_dir: str, test_encryption_key: bytes, redis_results_backend_config: "fixture" # noqa: F821 + self, + merlin_server_dir: str, + test_encryption_key: bytes, + redis_results_backend_config_function: "fixture", # noqa: F821 ): """ Test the `_get_key` function. :param merlin_server_dir: The directory to the merlin test server configuration :param test_encryption_key: A fixture to establish a fixed encryption key for testing - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ # Test the default functionality actual_default = _get_key() diff --git a/tests/unit/common/test_sample_index.py b/tests/unit/common/test_sample_index.py index c9cd108ee..18279af81 100644 --- a/tests/unit/common/test_sample_index.py +++ b/tests/unit/common/test_sample_index.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the `sample_index.py` and `sample_index_factory.py` files. """ diff --git a/tests/unit/common/test_util_sampling.py b/tests/unit/common/test_util_sampling.py index b4cc252d5..5c9608dc1 100644 --- a/tests/unit/common/test_util_sampling.py +++ b/tests/unit/common/test_util_sampling.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the `util_sampling.py` file. """ diff --git a/tests/unit/config/__init__.py b/tests/unit/config/__init__.py index e69de29bb..3232b50b9 100644 --- a/tests/unit/config/__init__.py +++ b/tests/unit/config/__init__.py @@ -0,0 +1,5 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## diff --git a/tests/unit/config/test_broker.py b/tests/unit/config/test_broker.py index 581b19488..8eb1aab00 100644 --- a/tests/unit/config/test_broker.py +++ b/tests/unit/config/test_broker.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the `broker.py` file. """ @@ -36,37 +42,37 @@ def test_read_file(merlin_server_dir: str): assert actual == SERVER_PASS -def test_get_connection_string_invalid_broker(redis_broker_config: "fixture"): # noqa: F821 +def test_get_connection_string_invalid_broker(redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_connection_string` function with an invalid broker (a broker that isn't one of: ["rabbitmq", "redis", "rediss", "redis+socket", "amqps", "amqp"]). - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ CONFIG.broker.name = "invalid_broker" with pytest.raises(ValueError): get_connection_string() -def test_get_connection_string_no_broker(redis_broker_config: "fixture"): # noqa: F821 +def test_get_connection_string_no_broker(redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_connection_string` function without a broker name value in the CONFIG object. This should raise a ValueError just like the `test_get_connection_string_invalid_broker` does. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ del CONFIG.broker.name with pytest.raises(ValueError): get_connection_string() -def test_get_connection_string_simple(redis_broker_config: "fixture"): # noqa: F821 +def test_get_connection_string_simple(redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_connection_string` function in the simplest way that we can. This function will automatically check for a broker url and if it finds one in the CONFIG object it will just return the value it finds. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ test_url = "test_url" CONFIG.broker.url = test_url @@ -74,11 +80,11 @@ def test_get_connection_string_simple(redis_broker_config: "fixture"): # noqa: assert actual == test_url -def test_get_ssl_config_no_broker(redis_broker_config: "fixture"): # noqa: F821 +def test_get_ssl_config_no_broker(redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_ssl_config` function without a broker. This should return False. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ del CONFIG.broker.name assert not get_ssl_config() @@ -274,11 +280,11 @@ def run_get_redissock_connection(self, expected_vals: Dict[str, str]): actual = get_redissock_connection() assert actual == expected - def test_get_redissock_connection(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_redissock_connection(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_redissock_connection` function with both a db_num and a broker path set. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ # Create and store a fake path and db_num for testing test_path = "/fake/path/to/broker" @@ -290,12 +296,12 @@ def test_get_redissock_connection(self, redis_broker_config: "fixture"): # noqa expected_vals = {"db_num": test_db_num, "path": test_path} self.run_get_redissock_connection(expected_vals) - def test_get_redissock_connection_no_db(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_redissock_connection_no_db(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_redissock_connection` function with a broker path set but no db num. This should default the db_num to 0. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ # Create and store a fake path for testing test_path = "/fake/path/to/broker" @@ -305,25 +311,25 @@ def test_get_redissock_connection_no_db(self, redis_broker_config: "fixture"): expected_vals = {"db_num": 0, "path": test_path} self.run_get_redissock_connection(expected_vals) - def test_get_redissock_connection_no_path(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_redissock_connection_no_path(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_redissock_connection` function with a db num set but no broker path. This should raise an AttributeError since there will be no path value to read from in `CONFIG.broker`. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ CONFIG.broker.db_num = "45" with pytest.raises(AttributeError): get_redissock_connection() - def test_get_redissock_connection_no_path_nor_db(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_redissock_connection_no_path_nor_db(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_redissock_connection` function with neither a broker path nor a db num set. This should raise an AttributeError since there will be no path value to read from in `CONFIG.broker`. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ with pytest.raises(AttributeError): get_redissock_connection() @@ -341,11 +347,11 @@ def run_get_redis_connection(self, expected_vals: Dict[str, Any], include_passwo actual = get_redis_connection(include_password=include_password, use_ssl=use_ssl) assert expected == actual - def test_get_redis_connection(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_redis_connection(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_redis_connection` function with default functionality (including password and not using ssl). - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ expected_vals = { "urlbase": "redis", @@ -356,13 +362,13 @@ def test_get_redis_connection(self, redis_broker_config: "fixture"): # noqa: F8 } self.run_get_redis_connection(expected_vals=expected_vals, include_password=True, use_ssl=False) - def test_get_redis_connection_no_port(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_redis_connection_no_port(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_redis_connection` function with default functionality (including password and not using ssl). We'll run this after deleting the port setting from the CONFIG object. This should still run and give us port = 6379. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ del CONFIG.broker.port expected_vals = { @@ -374,12 +380,12 @@ def test_get_redis_connection_no_port(self, redis_broker_config: "fixture"): # } self.run_get_redis_connection(expected_vals=expected_vals, include_password=True, use_ssl=False) - def test_get_redis_connection_with_db(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_redis_connection_with_db(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_redis_connection` function with default functionality (including password and not using ssl). We'll run this after adding the db_num setting to the CONFIG object. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ test_db_num = "45" CONFIG.broker.db_num = test_db_num @@ -392,25 +398,25 @@ def test_get_redis_connection_with_db(self, redis_broker_config: "fixture"): # } self.run_get_redis_connection(expected_vals=expected_vals, include_password=True, use_ssl=False) - def test_get_redis_connection_no_username(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_redis_connection_no_username(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_redis_connection` function with default functionality (including password and not using ssl). We'll run this after deleting the username setting from the CONFIG object. This should still run and give us username = ''. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ del CONFIG.broker.username expected_vals = {"urlbase": "redis", "spass": ":merlin-test-server@", "server": "127.0.0.1", "port": 6379, "db_num": 0} self.run_get_redis_connection(expected_vals=expected_vals, include_password=True, use_ssl=False) - def test_get_redis_connection_invalid_pass_file(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_redis_connection_invalid_pass_file(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_redis_connection` function with default functionality (including password and not using ssl). We'll run this after changing the permissions of the password file so it can't be opened. This should still run and give us password = CONFIG.broker.password. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ # Capture the initial permissions of the password file so we can reset them orig_file_permissions = os.stat(CONFIG.broker.password).st_mode @@ -435,21 +441,21 @@ def test_get_redis_connection_invalid_pass_file(self, redis_broker_config: "fixt os.chmod(CONFIG.broker.password, orig_file_permissions) - def test_get_redis_connection_dont_include_password(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_redis_connection_dont_include_password(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_redis_connection` function without including the password. This should place 6 *s where the password would normally be placed in spass. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ expected_vals = {"urlbase": "redis", "spass": "default:******@", "server": "127.0.0.1", "port": 6379, "db_num": 0} self.run_get_redis_connection(expected_vals=expected_vals, include_password=False, use_ssl=False) - def test_get_redis_connection_use_ssl(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_redis_connection_use_ssl(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_redis_connection` function with using ssl. This should change the urlbase to rediss (with two 's'). - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ expected_vals = { "urlbase": "rediss", @@ -460,24 +466,24 @@ def test_get_redis_connection_use_ssl(self, redis_broker_config: "fixture"): # } self.run_get_redis_connection(expected_vals=expected_vals, include_password=True, use_ssl=True) - def test_get_redis_connection_no_password(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_redis_connection_no_password(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_redis_connection` function with default functionality (including password and not using ssl). We'll run this after deleting the password setting from the CONFIG object. This should still run and give us spass = ''. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ del CONFIG.broker.password expected_vals = {"urlbase": "redis", "spass": "", "server": "127.0.0.1", "port": 6379, "db_num": 0} self.run_get_redis_connection(expected_vals=expected_vals, include_password=True, use_ssl=False) - def test_get_connection_string_redis(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_connection_string_redis(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_connection_string` function with redis as the broker (this is what our CONFIG - is set to by default with the redis_broker_config fixture). + is set to by default with the redis_broker_config_function fixture). - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ expected_vals = { "urlbase": "redis", @@ -490,11 +496,11 @@ def test_get_connection_string_redis(self, redis_broker_config: "fixture"): # n actual = get_connection_string() assert expected == actual - def test_get_connection_string_rediss(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_connection_string_rediss(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_connection_string` function with rediss (with two 's') as the broker. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ CONFIG.broker.name = "rediss" expected_vals = { @@ -508,11 +514,11 @@ def test_get_connection_string_rediss(self, redis_broker_config: "fixture"): # actual = get_connection_string() assert expected == actual - def test_get_connection_string_redis_socket(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_connection_string_redis_socket(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_connection_string` function with redis+socket as the broker. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ # Change our broker CONFIG.broker.name = "redis+socket" @@ -529,21 +535,21 @@ def test_get_connection_string_redis_socket(self, redis_broker_config: "fixture" actual = get_connection_string() assert actual == expected - def test_get_ssl_config_redis(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_ssl_config_redis(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_ssl_config` function with redis as the broker (this is the default in our tests). This should return False. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ assert not get_ssl_config() - def test_get_ssl_config_rediss(self, redis_broker_config: "fixture"): # noqa: F821 + def test_get_ssl_config_rediss(self, redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_ssl_config` function with rediss (with two 's') as the broker. This should return a dict of cert reqs with ssl.CERT_NONE as the value. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ CONFIG.broker.name = "rediss" expected = {"ssl_cert_reqs": CERT_NONE} diff --git a/tests/unit/config/test_config_object.py b/tests/unit/config/test_config_object.py index 64e56b7d9..8506412be 100644 --- a/tests/unit/config/test_config_object.py +++ b/tests/unit/config/test_config_object.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Test the functionality of the Config object. """ diff --git a/tests/unit/config/test_configfile.py b/tests/unit/config/test_configfile.py index 975e19ee4..b8de35557 100644 --- a/tests/unit/config/test_configfile.py +++ b/tests/unit/config/test_configfile.py @@ -1,15 +1,25 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the configfile.py module. """ import getpass +import logging import os import shutil import ssl from copy import copy, deepcopy +from unittest.mock import MagicMock, patch import pytest import yaml +from _pytest.capture import CaptureFixture +from pytest_mock import MockerFixture from merlin.config.configfile import ( CONFIG, @@ -17,22 +27,90 @@ find_config_file, get_cert_file, get_config, + get_default_config, get_ssl_entries, + initialize_config, is_debug, + is_local_mode, load_config, load_default_celery, - load_default_user_names, load_defaults, merge_sslmap, process_ssl_map, + set_local_mode, + set_username_and_vhost, ) from tests.constants import CERT_FILES +from tests.fixture_types import FixtureCallable, FixtureStr from tests.utils import create_dir -CONFIGFILE_DIR = "{temp_output_dir}/test_configfile" COPIED_APP_FILENAME = "app_copy.yaml" -DUMMY_APP_FILEPATH = f"{os.path.dirname(__file__)}/dummy_app.yaml" +DUMMY_APP_FILEPATH = os.path.join(os.path.dirname(__file__), "dummy_app.yaml") + + +@pytest.fixture(scope="session") +def configfile_testing_dir(create_testing_dir: FixtureCallable, config_testing_dir: FixtureStr) -> FixtureStr: + """ + Fixture to create a temporary output directory for tests related to testing the + `config` directory. + + Args: + create_testing_dir: A fixture which returns a function that creates the testing directory. + config_testing_dir: The path to the temporary ouptut directory for config tests. + + Returns: + The path to the temporary testing directory for tests of the `configfile.py` module + """ + return create_testing_dir(config_testing_dir, "configfile_tests") + + +@pytest.fixture(scope="session") +def demo_app_yaml(configfile_testing_dir: FixtureStr) -> FixtureStr: + """ + Fixture that creates an empty `app.yaml` file in the specified testing directory. + + Args: + configfile_testing_dir (FixtureStr): The directory used for testing configurations. + + Returns: + The path to the newly created `app.yaml` file. + """ + app_yaml_path = os.path.join(configfile_testing_dir, "app.yaml") + with open(app_yaml_path, "w"): + pass + return app_yaml_path + + +@pytest.fixture(scope="session") +def config_path(configfile_testing_dir: FixtureStr, demo_app_yaml: FixtureStr) -> FixtureStr: + """ + Fixture that creates a `config_path.txt` file containing the path to `app.yaml`. + + Args: + configfile_testing_dir (FixtureStr): The directory used for testing configurations. + demo_app_yaml (FixtureStr): The path to the `app.yaml` file created by the `demo_app_yaml` fixture. + + Returns: + The path to the newly created `config_path.txt` file. + """ + config_path_file = os.path.join(configfile_testing_dir, "config_path.txt") + with open(config_path_file, "w") as cpf: + cpf.write(demo_app_yaml) + return config_path_file + + +@pytest.fixture(autouse=True) +def reset_globals(): + """ + Reset IS_LOCAL_MODE before each test. + + This is done automatically without having to manually use this fixture in each test + with the use of `autouse=True`. + """ + set_local_mode(False) + yield + set_local_mode(False) def create_app_yaml(app_yaml_filepath: str): @@ -46,20 +124,66 @@ def create_app_yaml(app_yaml_filepath: str): shutil.copy(DUMMY_APP_FILEPATH, full_app_yaml_filepath) -def test_load_config(temp_output_dir: str): +def test_local_mode_toggle_and_logging(caplog: CaptureFixture): + """ + Test enabling/disabling local mode and logging behavior. + + Args: + caplog: A built-in fixture from the pytest library to capture logs. + """ + caplog.set_level(logging.INFO) + # Test default state + assert is_local_mode() is False + + # Test enabling (default parameter and explicit True) + set_local_mode() # Default True + assert is_local_mode() is True + assert "Running Merlin in local mode (no configuration file required)" in caplog.text + + # Test disabling + set_local_mode(False) + assert is_local_mode() is False + + +def test_default_config_structure_and_values(): + """ + Test default config structure and values. + """ + config = get_default_config() + + # Verify structure and key values + expected_config = { + "broker": { + "username": "user", + "vhost": "vhost", + "server": "localhost", + "name": "rabbitmq", + "port": 5672, + "protocol": "amqp", + }, + "celery": {"omit_queue_tag": False, "queue_tag": "[merlin]_", "override": None}, + "results_backend": {"name": "sqlite", "port": 1234}, + } + + assert config == expected_config + assert isinstance(config, dict) + + +def test_load_config(configfile_testing_dir: FixtureStr): """ Test the `load_config` function. - :param temp_output_dir: The path to the temporary output directory we'll be using for this test run + Args: + configfile_testing_dir (FixtureStr): The directory used for testing configurations. """ - configfile_dir = CONFIGFILE_DIR.format(temp_output_dir=temp_output_dir) + configfile_dir = os.path.join(configfile_testing_dir, "test_load_config") create_dir(configfile_dir) create_app_yaml(configfile_dir) with open(DUMMY_APP_FILEPATH, "r") as dummy_app_file: expected = yaml.load(dummy_app_file, yaml.Loader) - actual = load_config(f"{configfile_dir}/app.yaml") + actual = load_config(os.path.join(configfile_dir, "app.yaml")) assert actual == expected @@ -70,126 +194,117 @@ def test_load_config_invalid_file(): assert load_config("invalid/filepath") is None -def test_find_config_file_valid_path(temp_output_dir: str): +def test_find_config_file_config_path_file_exists_and_is_valid( + mocker: MockerFixture, demo_app_yaml: FixtureStr, config_path: FixtureStr +): """ - Test the `find_config_file` function with passing a valid path in. + Test that `find_config_file` correctly returns the path to the configuration file + when `CONFIG_PATH_FILE` exists and points to a valid `app.yaml` file. - :param temp_output_dir: The path to the temporary output directory we'll be using for this test run + Args: + mocker (MockerFixture): Pytest mocker fixture for mocking functionality. + demo_app_yaml (FixtureStr): Path to the valid `app.yaml` file. + config_path (FixtureStr): Path to the `CONFIG_PATH_FILE` containing the valid configuration path. """ - configfile_dir = CONFIGFILE_DIR.format(temp_output_dir=temp_output_dir) - create_dir(configfile_dir) - create_app_yaml(configfile_dir) - - assert find_config_file(configfile_dir) == f"{configfile_dir}/app.yaml" + mocker.patch("merlin.config.configfile.CONFIG_PATH_FILE", config_path) + result = find_config_file() + assert result == demo_app_yaml -def test_find_config_file_invalid_path(): +def test_find_config_file_config_path_file_exists_but_invalid(mocker: MockerFixture, configfile_testing_dir: FixtureStr): """ - Test the `find_config_file` function with passing an invalid path in. + Test that `find_config_file` returns `None` when `CONFIG_PATH_FILE` exists but points to an invalid path. + + Args: + mocker (MockerFixture): Pytest mocker fixture for mocking functionality. + configfile_testing_dir (FixtureStr): Directory used for testing invalid configuration paths. """ - assert find_config_file("invalid/path") is None + config_file_path = os.path.join(configfile_testing_dir, "config_path_app_yaml_doesnt_exist.txt") + with open(config_file_path, "w") as cfp: + cfp.write("invalid_app.yaml") + mocker.patch("merlin.config.configfile.CONFIG_PATH_FILE", config_file_path) + mocker.patch("merlin.config.configfile.MERLIN_HOME", os.path.join(configfile_testing_dir, ".merlin")) + result = find_config_file() + assert result is None -def test_find_config_file_local_path(temp_output_dir: str): +def test_find_config_file_local_app_yaml_exists(mocker: MockerFixture, configfile_testing_dir: FixtureStr): """ - Test the `find_config_file` function by having it find a local (in our cwd) app.yaml file. - We'll use the `temp_output_dir` fixture so that our current working directory is in a temp - location. + Test that `find_config_file` correctly returns the path to `app.yaml` when it exists in the current working directory. - :param temp_output_dir: The path to the temporary output directory we'll be using for this test run + Args: + mocker (MockerFixture): Pytest mocker fixture for mocking functionality. + configfile_testing_dir (FixtureStr): Directory used for testing. """ + mocker.patch("merlin.config.configfile.CONFIG_PATH_FILE", os.path.join(configfile_testing_dir, "invalid_config_path.txt")) + mocker.patch("os.getcwd", return_value=configfile_testing_dir) + local_app_yaml = os.path.join(configfile_testing_dir, "app.yaml") + mocker.patch("os.path.isfile", side_effect=lambda x: x == local_app_yaml) - # Create the configfile directory and put an app.yaml file there - configfile_dir = CONFIGFILE_DIR.format(temp_output_dir=temp_output_dir) - create_dir(configfile_dir) - create_app_yaml(configfile_dir) - - # Move into the configfile directory and run the test - os.chdir(configfile_dir) - try: - assert find_config_file() == f"{os.getcwd()}/app.yaml" - except AssertionError as exc: - # Move back to the temp output directory even if the test fails - os.chdir(temp_output_dir) - raise AssertionError from exc + result = find_config_file() + assert result == local_app_yaml - # Move back to the temp output directory - os.chdir(temp_output_dir) - -def test_find_config_file_merlin_home_path(temp_output_dir: str): +def test_find_config_file_merlin_home_app_yaml_exists(mocker: MockerFixture, configfile_testing_dir: FixtureStr): """ - Test the `find_config_file` function by having it find an app.yaml file in our merlin directory. - We'll use the `temp_output_dir` fixture so that our current working directory is in a temp - location. + Test that `find_config_file` correctly returns the path to `app.yaml` when it exists in the `MERLIN_HOME` directory. - :param temp_output_dir: The path to the temporary output directory we'll be using for this test run + Args: + mocker (MockerFixture): Pytest mocker fixture for mocking functionality. + configfile_testing_dir (FixtureStr): Directory used for testing. """ - merlin_home = os.path.expanduser("~/.merlin") - if not os.path.exists(merlin_home): - os.mkdir(merlin_home) - create_app_yaml(merlin_home) - assert find_config_file() == f"{merlin_home}/app.yaml" + mocker.patch("merlin.config.configfile.CONFIG_PATH_FILE", os.path.join(configfile_testing_dir, "invalid_config_path.txt")) + mocker.patch("merlin.config.configfile.MERLIN_HOME", configfile_testing_dir) + merlin_home_app_yaml = os.path.join(configfile_testing_dir, "app.yaml") + mocker.patch("os.path.isfile", side_effect=lambda x: x == merlin_home_app_yaml) + result = find_config_file() + assert result == merlin_home_app_yaml -def check_for_and_move_app_yaml(dir_to_check: str) -> bool: + +def test_find_config_file_no_app_yaml_found(mocker: MockerFixture): """ - Check for any app.yaml files in `dir_to_check`. If one is found, rename it. - Return True if an app.yaml was found, false otherwise. + Test that `find_config_file` returns `None` when no `app.yaml` file is found in any location. - :param dir_to_check: The directory to search for an app.yaml in - :returns: True if an app.yaml was found. False otherwise. + Args: + mocker (MockerFixture): Pytest mocker fixture for mocking functionality. """ - for filename in os.listdir(dir_to_check): - full_path = os.path.join(dir_to_check, filename) - if os.path.isfile(full_path) and filename == "app.yaml": - os.rename(full_path, f"{dir_to_check}/{COPIED_APP_FILENAME}") - return True - return False + mocker.patch("os.path.isfile", return_value=False) + result = find_config_file() + assert result is None -def test_find_config_file_no_path(temp_output_dir: str): +def test_find_config_file_path_provided_app_yaml_exists(mocker: MockerFixture, configfile_testing_dir: FixtureStr): """ - Test the `find_config_file` function by making it unable to find any app.yaml path. - We'll use the `temp_output_dir` fixture so that our current working directory is in a temp - location. + Test that `find_config_file` correctly returns the path to `app.yaml` when a valid directory path is provided. - :param temp_output_dir: The path to the temporary output directory we'll be using for this test run + Args: + mocker (MockerFixture): Pytest mocker fixture for mocking functionality. + configfile_testing_dir (FixtureStr): Directory used for testing. """ + mocker.patch("merlin.config.configfile.CONFIG_PATH_FILE", os.path.join(configfile_testing_dir, "invalid_config_path.txt")) + mocker.patch("os.path.isfile", side_effect=lambda x: x == "/mock/provided/path/app.yaml") + mocker.patch("os.path.exists", return_value=True) + result = find_config_file("/mock/provided/path") + assert result == "/mock/provided/path/app.yaml" - # Rename any app.yaml in the cwd - cwd_path = os.getcwd() - cwd_had_app_yaml = check_for_and_move_app_yaml(cwd_path) - # Rename any app.yaml in the merlin home directory - merlin_home_dir = os.path.expanduser("~/.merlin") - merlin_home_had_app_yaml = check_for_and_move_app_yaml(merlin_home_dir) - - try: - assert find_config_file() is None - except AssertionError as exc: - # Reset the cwd app.yaml even if the test fails - if cwd_had_app_yaml: - os.rename(f"{cwd_path}/{COPIED_APP_FILENAME}", f"{cwd_path}/app.yaml") - - # Reset the merlin home app.yaml even if the test fails - if merlin_home_had_app_yaml: - os.rename(f"{merlin_home_dir}/{COPIED_APP_FILENAME}", f"{merlin_home_dir}/app.yaml") - - raise AssertionError from exc - - # Reset the cwd app.yaml - if cwd_had_app_yaml: - os.rename(f"{cwd_path}/{COPIED_APP_FILENAME}", f"{cwd_path}/app.yaml") +def test_find_config_file_path_provided_app_yaml_does_not_exist(mocker: MockerFixture): + """ + Test that `find_config_file` returns `None` when a directory path is provided but `app.yaml` does not exist in that directory. - # Reset the merlin home app.yaml - if merlin_home_had_app_yaml: - os.rename(f"{merlin_home_dir}/{COPIED_APP_FILENAME}", f"{merlin_home_dir}/app.yaml") + Args: + mocker (MockerFixture): Pytest mocker fixture for mocking functionality. + """ + mocker.patch("os.path.isfile", return_value=False) + mocker.patch("os.path.exists", return_value=False) + result = find_config_file("/mock/provided/path") + assert result is None -def test_load_default_user_names_nothing_to_load(): +def test_set_username_and_vhost_nothing_to_load(): """ - Test the `load_default_user_names` function with nothing to load. In other words, in this + Test the `set_username_and_vhost` function with nothing to load. In other words, in this test the config dict will have a username and vhost already set for the broker. We'll create the dict then make a copy of it to test against after calling the function. """ @@ -197,35 +312,35 @@ def test_load_default_user_names_nothing_to_load(): expected_config = deepcopy(actual_config) assert actual_config is not expected_config - load_default_user_names(actual_config) + set_username_and_vhost(actual_config) - # Ensure that nothing was modified after our call to load_default_user_names + # Ensure that nothing was modified after our call to set_username_and_vhost assert actual_config == expected_config -def test_load_default_user_names_no_username(): +def test_set_username_and_vhost_no_username(): """ - Test the `load_default_user_names` function with no username. In other words, in this + Test the `set_username_and_vhost` function with no username. In other words, in this test the config dict will have vhost already set for the broker but not a username. """ expected_config = {"broker": {"username": getpass.getuser(), "vhost": "host4testing"}} actual_config = {"broker": {"vhost": "host4testing"}} - load_default_user_names(actual_config) + set_username_and_vhost(actual_config) - # Ensure that the username was set in the call to load_default_user_names + # Ensure that the username was set in the call to set_username_and_vhost assert actual_config == expected_config -def test_load_default_user_names_no_vhost(): +def test_set_username_and_vhost_no_vhost(): """ - Test the `load_default_user_names` function with no vhost. In other words, in this + Test the `set_username_and_vhost` function with no vhost. In other words, in this test the config dict will have username already set for the broker but not a vhost. """ expected_config = {"broker": {"username": "default", "vhost": getpass.getuser()}} actual_config = {"broker": {"username": "default"}} - load_default_user_names(actual_config) + set_username_and_vhost(actual_config) - # Ensure that the vhost was set in the call to load_default_user_names + # Ensure that the vhost was set in the call to set_username_and_vhost assert actual_config == expected_config @@ -313,15 +428,16 @@ def test_load_defaults(): assert actual_config == expected_config -def test_get_config(temp_output_dir: str): +def test_get_config(configfile_testing_dir: FixtureStr): """ Test the `get_config` function. - :param temp_output_dir: The path to the temporary output directory we'll be using for this test run + Args: + configfile_testing_dir (FixtureStr): The directory used for testing configurations. """ # Create the configfile directory and put an app.yaml file there - configfile_dir = CONFIGFILE_DIR.format(temp_output_dir=temp_output_dir) + configfile_dir = os.path.join(configfile_testing_dir, "test_get_config") create_dir(configfile_dir) create_app_yaml(configfile_dir) @@ -405,60 +521,32 @@ def test_is_debug_with_merlin_debug(): os.environ["MERLIN_DEBUG"] = debug_val -def test_default_config_info(temp_output_dir: str): +def test_default_config_info(mocker: MockerFixture): """ Test the `default_config_info` function. - :param temp_output_dir: The path to the temporary output directory we'll be using for this test run + Args: + mocker (MockerFixture): Pytest mocker fixture for mocking functionality. """ + # Mock the necessary functions/variables + config_file_path = "/mock/config/app.yaml" + debug_mode = False + merlin_home = "/mock/merlin_home/" + find_config_file_mock = mocker.patch("merlin.config.configfile.find_config_file", return_value=config_file_path) + is_debug_mock = mocker.patch("merlin.config.configfile.is_debug", return_value=debug_mode) + mocker.patch("merlin.config.configfile.MERLIN_HOME", merlin_home) - # Create the configfile directory and put an app.yaml file there - configfile_dir = CONFIGFILE_DIR.format(temp_output_dir=temp_output_dir) - create_dir(configfile_dir) - create_app_yaml(configfile_dir) - cwd = os.getcwd() - os.chdir(configfile_dir) - - # Delete the current val of MERLIN_DEBUG and store it (if there is one) - reset_merlin_debug = False - debug_val = None - if "MERLIN_DEBUG" in os.environ: - debug_val = copy(os.environ["MERLIN_DEBUG"]) - del os.environ["MERLIN_DEBUG"] - reset_merlin_debug = True - - # Create the merlin home directory if it doesn't already exist - merlin_home = f"{os.path.expanduser('~')}/.merlin" - remove_merlin_home = False - if not os.path.exists(merlin_home): - os.mkdir(merlin_home) - remove_merlin_home = True - - # Run the test - try: - expected = { - "config_file": f"{configfile_dir}/app.yaml", - "is_debug": False, - "merlin_home": merlin_home, - "merlin_home_exists": True, - } - actual = default_config_info() - assert actual == expected - except AssertionError as exc: - # Make sure to reset values even if the test fails - if reset_merlin_debug: - os.environ["MERLIN_DEBUG"] = debug_val - if remove_merlin_home: - os.rmdir(merlin_home) - raise AssertionError from exc - - # Reset values if necessary - if reset_merlin_debug: - os.environ["MERLIN_DEBUG"] = debug_val - if remove_merlin_home: - os.rmdir(merlin_home) - - os.chdir(cwd) + # Run the test and verify the output + expected = { + "config_file": config_file_path, + "is_debug": debug_mode, + "merlin_home": merlin_home, + "merlin_home_exists": False, + } + actual = default_config_info() + find_config_file_mock.assert_called_once() + is_debug_mock.assert_called_once() + assert actual == expected def test_get_cert_file_all_valid_args(mysql_results_backend_config: "fixture", merlin_server_dir: str): # noqa: F821 @@ -694,3 +782,229 @@ def test_merge_sslmap_some_keys_present(): } actual = merge_sslmap(test_server_ssl, test_ssl_map) assert actual == expected + + +@patch("merlin.config.configfile.CONFIG", return_value=MagicMock) +@patch("merlin.config.configfile.get_default_config") +@patch("merlin.config.configfile.Config") +@patch("merlin.config.configfile.get_config") +def test_initialize_config_default_parameters( + mock_get_config: MagicMock, mock_config_class: MagicMock, mock_get_default_config: MagicMock, mock_global_config: MagicMock +): + """ + Test initialize_config with default parameters. + + Args: + mock_get_config: A mocked `get_config` function. + mock_config_class: A mocked `Config` object. + mock_get_default_config: A mocked `get_default_config` function. + mock_global_config: A mocked `CONFIG` variable. + """ + mock_app_config = {"test": "config"} + mock_get_config.return_value = mock_app_config + mock_config_instance = MagicMock() + mock_config_class.return_value = mock_config_instance + + initialize_config() + + mock_get_config.assert_called_once_with(None) + mock_config_class.assert_called_once_with(mock_app_config) + assert is_local_mode() is False + + +@patch("merlin.config.configfile.CONFIG", return_value=MagicMock) +@patch("merlin.config.configfile.get_default_config") +@patch("merlin.config.configfile.Config") +@patch("merlin.config.configfile.get_config") +def test_initialize_config_with_path( + mock_get_config: MagicMock, mock_config_class: MagicMock, mock_get_default_config: MagicMock, mock_global_config: MagicMock +): + """ + Test initialize_config with custom path. + + Args: + mock_get_config: A mocked `get_config` function. + mock_config_class: A mocked `Config` object. + mock_get_default_config: A mocked `get_default_config` function. + mock_global_config: A mocked `CONFIG` variable. + """ + mock_app_config = {"test": "config"} + mock_get_config.return_value = mock_app_config + mock_config_instance = MagicMock() + mock_config_class.return_value = mock_config_instance + + test_path = "/path/to/config" + initialize_config(path=test_path) + + mock_get_config.assert_called_once_with(test_path) + mock_config_class.assert_called_once_with(mock_app_config) + assert is_local_mode() is False + + +@patch("merlin.config.configfile.CONFIG", return_value=MagicMock) +@patch("merlin.config.configfile.get_default_config") +@patch("merlin.config.configfile.Config") +@patch("merlin.config.configfile.get_config") +def test_initialize_config_with_local_mode_true( + mock_get_config: MagicMock, mock_config_class: MagicMock, mock_get_default_config: MagicMock, mock_global_config: MagicMock +): + """ + Test initialize_config with local_mode=True. + + Args: + mock_get_config: A mocked `get_config` function. + mock_config_class: A mocked `Config` object. + mock_get_default_config: A mocked `get_default_config` function. + mock_global_config: A mocked `CONFIG` variable. + """ + mock_app_config = {"test": "config"} + mock_get_config.return_value = mock_app_config + mock_config_instance = MagicMock() + mock_config_class.return_value = mock_config_instance + + with patch("merlin.config.configfile.LOG"): + initialize_config(local_mode=True) + + mock_get_config.assert_called_once_with(None) + mock_config_class.assert_called_once_with(mock_app_config) + assert is_local_mode() is True + + +@patch("merlin.config.configfile.CONFIG", return_value=MagicMock) +@patch("merlin.config.configfile.get_default_config") +@patch("merlin.config.configfile.Config") +@patch("merlin.config.configfile.get_config") +def test_initialize_config_with_path_and_local_mode( + mock_get_config: MagicMock, mock_config_class: MagicMock, mock_get_default_config: MagicMock, mock_global_config: MagicMock +): + """ + Test initialize_config with both path and local_mode. + + Args: + mock_get_config: A mocked `get_config` function. + mock_config_class: A mocked `Config` object. + mock_get_default_config: A mocked `get_default_config` function. + mock_global_config: A mocked `CONFIG` variable. + """ + mock_app_config = {"test": "config"} + mock_get_config.return_value = mock_app_config + mock_config_instance = MagicMock() + mock_config_class.return_value = mock_config_instance + + test_path = "/custom/path" + with patch("merlin.config.configfile.LOG"): + initialize_config(path=test_path, local_mode=True) + + mock_get_config.assert_called_once_with(test_path) + mock_config_class.assert_called_once_with(mock_app_config) + assert is_local_mode() is True + + +@patch("merlin.config.configfile.CONFIG", return_value=MagicMock) +@patch("merlin.config.configfile.get_default_config") +@patch("merlin.config.configfile.Config") +@patch("merlin.config.configfile.get_config") +def test_initialize_config_with_local_mode_false( + mock_get_config: MagicMock, mock_config_class: MagicMock, mock_get_default_config: MagicMock, mock_global_config: MagicMock +): + """ + Test initialize_config with explicit local_mode=False. + + Args: + mock_get_config: A mocked `get_config` function. + mock_config_class: A mocked `Config` object. + mock_get_default_config: A mocked `get_default_config` function. + mock_global_config: A mocked `CONFIG` variable. + """ + mock_app_config = {"test": "config"} + mock_get_config.return_value = mock_app_config + mock_config_instance = MagicMock() + mock_config_class.return_value = mock_config_instance + + initialize_config(local_mode=False) + + mock_get_config.assert_called_once_with(None) + mock_config_class.assert_called_once_with(mock_app_config) + assert is_local_mode() is False + + +@patch("merlin.config.configfile.CONFIG", return_value=MagicMock) +@patch("merlin.config.configfile.LOG") +@patch("merlin.config.configfile.get_default_config") +@patch("merlin.config.configfile.Config") +@patch("merlin.config.configfile.get_config") +def test_initialize_config_fallback_to_default( + mock_get_config: MagicMock, + mock_config_class: MagicMock, + mock_get_default_config: MagicMock, + mock_log: MagicMock, + mock_global_config: MagicMock, +): + """ + Test initialize_config falls back to default config when get_config raises ValueError. + + Args: + mock_get_config: A mocked `get_config` function. + mock_config_class: A mocked `Config` object. + mock_get_default_config: A mocked `get_default_config` function. + mock_log: A mocked logger. + mock_global_config: A mocked `CONFIG` variable. + """ + # Setup mocks + mock_get_config.side_effect = ValueError("Config file not found") + mock_default_config = {"default": "config"} + mock_get_default_config.return_value = mock_default_config + mock_config_instance = MagicMock() + mock_config_class.return_value = mock_config_instance + + initialize_config() + + # Verify behavior + mock_get_config.assert_called_once_with(None) + mock_get_default_config.assert_called_once() + mock_config_class.assert_called_once_with(mock_default_config) + mock_log.warning.assert_called_once_with( + "Error loading configuration: Config file not found. Falling back to default configuration." + ) + assert is_local_mode() is False + + +@patch("merlin.config.configfile.CONFIG", return_value=MagicMock) +@patch("merlin.config.configfile.LOG") +@patch("merlin.config.configfile.get_default_config") +@patch("merlin.config.configfile.Config") +@patch("merlin.config.configfile.get_config") +def test_initialize_config_fallback_with_local_mode( + mock_get_config: MagicMock, + mock_config_class: MagicMock, + mock_get_default_config: MagicMock, + mock_log: MagicMock, + mock_global_config: MagicMock, +): + """ + Test initialize_config falls back to default config when get_config raises ValueError with local_mode=True. + + Args: + mock_get_config: A mocked `get_config` function. + mock_config_class: A mocked `Config` object. + mock_get_default_config: A mocked `get_default_config` function. + mock_log: A mocked logger. + mock_global_config: A mocked `CONFIG` variable. + """ + # Setup mocks + mock_get_config.side_effect = ValueError("Config file not found") + mock_default_config = {"default": "config"} + mock_get_default_config.return_value = mock_default_config + mock_config_instance = MagicMock() + mock_config_class.return_value = mock_config_instance + + initialize_config(local_mode=True) + + # Verify behavior + mock_get_config.assert_called_once_with(None) + mock_get_default_config.assert_called_once() + mock_config_class.assert_called_once_with(mock_default_config) + mock_log.warning.assert_called_once_with( + "Error loading configuration: Config file not found. Falling back to default configuration." + ) + assert is_local_mode() is True diff --git a/tests/unit/config/test_merlin_config_manager.py b/tests/unit/config/test_merlin_config_manager.py new file mode 100644 index 000000000..63bd81c31 --- /dev/null +++ b/tests/unit/config/test_merlin_config_manager.py @@ -0,0 +1,208 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `merlin_config_manager.py` module. +""" + +import os +from argparse import Namespace + +import pytest +import yaml +from pytest_mock import MockerFixture + +from merlin.config.merlin_config_manager import MerlinConfigManager +from tests.fixture_types import FixtureCallable, FixtureStr + + +@pytest.fixture(scope="session") +def merlin_config_manager_testing_dir(create_testing_dir: FixtureCallable, config_testing_dir: FixtureStr) -> FixtureStr: + """ + Fixture to create a temporary output directory for tests related to testing the + `config` directory. + + Args: + create_testing_dir: A fixture which returns a function that creates the testing directory. + config_testing_dir: The path to the temporary ouptut directory for config tests + + Returns: + The path to the temporary testing directory for tests of the `merlin_config_manager.py` module + """ + return create_testing_dir(config_testing_dir, "merlin_config_manager_tests") + + +@pytest.fixture +def args() -> Namespace: + """ + Fixture for creating a mock `Namespace` object with default arguments. + + Returns: + A mock object containing default configuration arguments for testing. + """ + return Namespace( + config_file=None, + task_server="celery", + broker="redis", + type="redis", + password_file="fake_pass.txt", + server="localhost", + port=6379, + db_num=0, + cert_reqs="none", + username="", + password="", + vhost="/", + test=True, + ) + + +def test_create_template_config_creates_file( + mocker: MockerFixture, args: Namespace, merlin_config_manager_testing_dir: FixtureStr +): + """ + Test that `create_template_config` creates the expected configuration file. + + Args: + mocker (MockerFixture): The mocker fixture for mocking dependencies. + args (Namespace): The mock configuration arguments fixture. + merlin_config_manager_testing_dir (str): The directory used for testing configurations. + """ + output_dir = os.path.join(merlin_config_manager_testing_dir, "test_create_template_config_creates_file") + mock_encrypt = mocker.patch("merlin.common.security.encrypt.init_key") + args.config_file = os.path.join(output_dir, "app.yaml") + + config_manager = MerlinConfigManager(args) + config_manager.create_template_config() + + mock_encrypt.assert_called_once() + assert os.path.exists(args.config_file) + + +def test_save_config_path(mocker: MockerFixture, args: Namespace, merlin_config_manager_testing_dir: FixtureStr): + """ + Test that `save_config_path` writes the correct config file path. + + Args: + mocker (pytest_mock.MockerFixture): The mocker fixture for mocking dependencies. + args (Namespace): The mock configuration arguments fixture. + merlin_config_manager_testing_dir (str): The directory used for testing configurations. + """ + # Mock CONFIG_PATH_FILE and set a config file path + output_dir = os.path.join(merlin_config_manager_testing_dir, "test_save_config_path") + mocked_config_path_file = os.path.join(output_dir, "config_path.txt") + mocker.patch("merlin.config.merlin_config_manager.CONFIG_PATH_FILE", mocked_config_path_file) + args.config_file = os.path.join(output_dir, "app.yaml") + + # Run the test + config_manager = MerlinConfigManager(args) + config_manager.create_template_config() + config_manager.save_config_path() + + # Check that the output is correct + assert os.path.exists(mocked_config_path_file) + with open(mocked_config_path_file) as f: + path = f.read().strip() + assert path == args.config_file + + +def test_update_redis_broker_config(args: Namespace, merlin_config_manager_testing_dir: FixtureStr): + """ + Test that `update_backend` correctly updates the Redis broker configuration. + + Args: + args (Namespace): The mock configuration arguments fixture. + merlin_config_manager_testing_dir (str): The directory used for testing configurations. + """ + output_dir = os.path.join(merlin_config_manager_testing_dir, "test_update_redis_broker_config") + args.config_file = os.path.join(output_dir, "app.yaml") + args.port = 1234 + args.db_num = 1 + config_manager = MerlinConfigManager(args) + config_manager.create_template_config() + + config = {"broker": {}} + with open(args.config_file, "w") as f: + yaml.dump(config, f) + + config_manager.update_broker() + + with open(args.config_file) as f: + updated_config = yaml.safe_load(f) + + assert updated_config["broker"]["name"] == "rediss" # from `args` fixture + assert updated_config["broker"]["server"] == "localhost" # from `args` fixture + assert updated_config["broker"]["port"] == 1234 + assert updated_config["broker"]["db_num"] == 1 + + +def test_update_redis_backend_config(args: Namespace, merlin_config_manager_testing_dir: FixtureStr): + """ + Test that `update_backend` correctly updates the Redis backend configuration. + + Args: + args (Namespace): The mock configuration arguments fixture. + merlin_config_manager_testing_dir (str): The directory used for testing configurations. + """ + output_dir = os.path.join(merlin_config_manager_testing_dir, "test_update_redis_broker_config") + args.config_file = os.path.join(output_dir, "app.yaml") + args.port = 1234 + args.db_num = 1 + config_manager = MerlinConfigManager(args) + config_manager.create_template_config() + + config = {"results_backend": {}} + with open(args.config_file, "w") as f: + yaml.dump(config, f) + + config_manager.update_backend() + + with open(args.config_file) as f: + updated_config = yaml.safe_load(f) + + assert updated_config["results_backend"]["name"] == "rediss" # from `args` fixture + assert updated_config["results_backend"]["server"] == "localhost" # from `args` fixture + assert updated_config["results_backend"]["port"] == 1234 + assert updated_config["results_backend"]["db_num"] == 1 + + +def test_update_rabbitmq_config(args: Namespace): + """ + Test that `update_rabbitmq_config` correctly updates the RabbitMQ configuration. + + Args: + args (Namespace): The mock configuration arguments fixture. + """ + args.type = "rabbitmq" + args.password_file = "pw.txt" + args.username = "user" + args.vhost = "vhost" + args.server = "server" + args.port = 5672 + args.cert_reqs = "CERT_REQUIRED" + args.config_file = "dummy.yaml" + + config_manager = MerlinConfigManager(args) + config = {} + config_manager.update_rabbitmq_config(config) + assert config["name"] == "rabbitmq" + assert config["username"] == "user" + assert config["vhost"] == "vhost" + + +def test_update_config_generic(args: Namespace): + """ + Test that `update_config` correctly updates the configuration with generic fields. + + Args: + args (Namespace): The mock configuration arguments fixture. + """ + args.test_field = "test_value" + args.config_file = "dummy.yaml" + config_manager = MerlinConfigManager(args) + config = {} + config_manager.update_config(config, ["test_field"]) + assert config["test_field"] == "test_value" diff --git a/tests/unit/config/test_results_backend.py b/tests/unit/config/test_results_backend.py index f49e3e897..56b9a7d9f 100644 --- a/tests/unit/config/test_results_backend.py +++ b/tests/unit/config/test_results_backend.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the `results_backend.py` file. """ @@ -102,7 +108,7 @@ def test_get_backend_password_using_certs_path(temp_output_dir: str): assert get_backend_password(pass_filename, certs_path=test_dir) == SERVER_PASS -def test_get_ssl_config_no_results_backend(config: "fixture"): # noqa: F821 +def test_get_ssl_config_no_results_backend(config_function: "fixture"): # noqa: F821 """ Test the `get_ssl_config` function with no results_backend set. This should return False. NOTE: we're using the config fixture here to make sure values are reset after this test finishes. @@ -114,7 +120,7 @@ def test_get_ssl_config_no_results_backend(config: "fixture"): # noqa: F821 assert get_ssl_config() is False -def test_get_connection_string_no_results_backend(config: "fixture"): # noqa: F821 +def test_get_connection_string_no_results_backend(config_function: "fixture"): # noqa: F821 """ Test the `get_connection_string` function with no results_backend set. This should raise a ValueError. @@ -160,11 +166,11 @@ def run_get_redis( actual = get_redis(certs_path=certs_path, include_password=include_password, ssl=ssl) assert actual == expected - def test_get_redis(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_get_redis(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test the `get_redis` function with default functionality. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ expected_vals = { "urlbase": "redis", @@ -175,11 +181,11 @@ def test_get_redis(self, redis_results_backend_config: "fixture"): # noqa: F821 } self.run_get_redis(expected_vals=expected_vals, certs_path=None, include_password=True, ssl=False) - def test_get_redis_dont_include_password(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_get_redis_dont_include_password(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test the `get_redis` function with the password hidden. This should * out the password. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ expected_vals = { "urlbase": "redis", @@ -190,11 +196,11 @@ def test_get_redis_dont_include_password(self, redis_results_backend_config: "fi } self.run_get_redis(expected_vals=expected_vals, certs_path=None, include_password=False, ssl=False) - def test_get_redis_using_ssl(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_get_redis_using_ssl(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test the `get_redis` function with ssl enabled. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ expected_vals = { "urlbase": "rediss", @@ -205,11 +211,11 @@ def test_get_redis_using_ssl(self, redis_results_backend_config: "fixture"): # } self.run_get_redis(expected_vals=expected_vals, certs_path=None, include_password=True, ssl=True) - def test_get_redis_no_port(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_get_redis_no_port(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test the `get_redis` function with no port in our CONFIG object. This should default to port=6379. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ del CONFIG.results_backend.port expected_vals = { @@ -221,11 +227,11 @@ def test_get_redis_no_port(self, redis_results_backend_config: "fixture"): # no } self.run_get_redis(expected_vals=expected_vals, certs_path=None, include_password=True, ssl=False) - def test_get_redis_no_db_num(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_get_redis_no_db_num(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test the `get_redis` function with no db_num in our CONFIG object. This should default to db_num=0. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ del CONFIG.results_backend.db_num expected_vals = { @@ -237,11 +243,11 @@ def test_get_redis_no_db_num(self, redis_results_backend_config: "fixture"): # } self.run_get_redis(expected_vals=expected_vals, certs_path=None, include_password=True, ssl=False) - def test_get_redis_no_username(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_get_redis_no_username(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test the `get_redis` function with no username in our CONFIG object. This should default to username=''. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ del CONFIG.results_backend.username expected_vals = { @@ -253,11 +259,11 @@ def test_get_redis_no_username(self, redis_results_backend_config: "fixture"): } self.run_get_redis(expected_vals=expected_vals, certs_path=None, include_password=True, ssl=False) - def test_get_redis_no_password_file(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_get_redis_no_password_file(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test the `get_redis` function with no password filepath in our CONFIG object. This should default to spass=''. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ del CONFIG.results_backend.password expected_vals = { @@ -269,12 +275,12 @@ def test_get_redis_no_password_file(self, redis_results_backend_config: "fixture } self.run_get_redis(expected_vals=expected_vals, certs_path=None, include_password=True, ssl=False) - def test_get_redis_invalid_pass_file(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_get_redis_invalid_pass_file(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test the `get_redis` function. We'll run this after changing the permissions of the password file so it can't be opened. This should still run and give us password=CONFIG.results_backend.password. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ # Capture the initial permissions of the password file so we can reset them @@ -299,41 +305,41 @@ def test_get_redis_invalid_pass_file(self, redis_results_backend_config: "fixtur os.chmod(CONFIG.results_backend.password, orig_file_permissions) raise AssertionError from exc - def test_get_ssl_config_redis(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_get_ssl_config_redis(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test the `get_ssl_config` function with redis as the results_backend. This should return False since ssl requires using rediss (with two 's'). - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ assert get_ssl_config() is False - def test_get_ssl_config_rediss(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_get_ssl_config_rediss(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test the `get_ssl_config` function with rediss as the results_backend. This should return a dict of cert reqs with ssl.CERT_NONE as the value. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ CONFIG.results_backend.name = "rediss" assert get_ssl_config() == {"ssl_cert_reqs": CERT_NONE} - def test_get_ssl_config_rediss_no_cert_reqs(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_get_ssl_config_rediss_no_cert_reqs(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test the `get_ssl_config` function with rediss as the results_backend and no cert_reqs set. This should return True. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ del CONFIG.results_backend.cert_reqs CONFIG.results_backend.name = "rediss" assert get_ssl_config() is True - def test_get_connection_string_redis(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_get_connection_string_redis(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test the `get_connection_string` function with redis as the results_backend. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ expected_vals = { "urlbase": "redis", @@ -346,11 +352,11 @@ def test_get_connection_string_redis(self, redis_results_backend_config: "fixtur actual = get_connection_string() assert actual == expected - def test_get_connection_string_rediss(self, redis_results_backend_config: "fixture"): # noqa: F821 + def test_get_connection_string_rediss(self, redis_results_backend_config_function: "fixture"): # noqa: F821 """ Test the `get_connection_string` function with rediss as the results_backend. - :param redis_results_backend_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_results_backend_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ CONFIG.results_backend.name = "rediss" expected_vals = { diff --git a/tests/unit/config/test_utils.py b/tests/unit/config/test_utils.py index 9d64c10c7..710998014 100644 --- a/tests/unit/config/test_utils.py +++ b/tests/unit/config/test_utils.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the merlin/config/utils.py module. """ @@ -47,12 +53,12 @@ def test_get_priority_rabbit_broker(rabbit_broker_config: "fixture"): # noqa: F assert get_priority(Priority.RETRY) == 10 -def test_get_priority_redis_broker(redis_broker_config: "fixture"): # noqa: F821 +def test_get_priority_redis_broker(redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_priority` function with redis as the broker. Low priority for redis is 10 and high is 2. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ assert get_priority(Priority.LOW) == 10 assert get_priority(Priority.MID) == 5 @@ -60,12 +66,12 @@ def test_get_priority_redis_broker(redis_broker_config: "fixture"): # noqa: F82 assert get_priority(Priority.RETRY) == 1 -def test_get_priority_invalid_broker(redis_broker_config: "fixture"): # noqa: F821 +def test_get_priority_invalid_broker(redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_priority` function with an invalid broker. This should raise a ValueError. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ CONFIG.broker.name = "invalid" with pytest.raises(ValueError) as excinfo: @@ -73,12 +79,12 @@ def test_get_priority_invalid_broker(redis_broker_config: "fixture"): # noqa: F assert "Unsupported broker name: invalid" in str(excinfo.value) -def test_get_priority_invalid_priority(redis_broker_config: "fixture"): # noqa: F821 +def test_get_priority_invalid_priority(redis_broker_config_function: "fixture"): # noqa: F821 """ Test the `get_priority` function with an invalid priority. This should raise a TypeError. - :param redis_broker_config: A fixture to set the CONFIG object to a test configuration that we'll use here + :param redis_broker_config_function: A fixture to set the CONFIG object to a test configuration that we'll use here """ with pytest.raises(ValueError) as excinfo: get_priority("invalid_priority") diff --git a/tests/unit/db_scripts/__init__.py b/tests/unit/db_scripts/__init__.py new file mode 100644 index 000000000..3232b50b9 --- /dev/null +++ b/tests/unit/db_scripts/__init__.py @@ -0,0 +1,5 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## diff --git a/tests/unit/db_scripts/entities/__init__.py b/tests/unit/db_scripts/entities/__init__.py new file mode 100644 index 000000000..3232b50b9 --- /dev/null +++ b/tests/unit/db_scripts/entities/__init__.py @@ -0,0 +1,5 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## diff --git a/tests/unit/db_scripts/entities/test_db_entity.py b/tests/unit/db_scripts/entities/test_db_entity.py new file mode 100644 index 000000000..a558be175 --- /dev/null +++ b/tests/unit/db_scripts/entities/test_db_entity.py @@ -0,0 +1,351 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `db_entity.py` module. +""" + +from typing import Dict, List +from unittest.mock import Mock + +import pytest +from pytest_mock import MockerFixture + +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.data_models import BaseDataModel +from merlin.db_scripts.entities.db_entity import DatabaseEntity +from merlin.exceptions import EntityNotFoundError, RunNotFoundError, StudyNotFoundError, WorkerNotFoundError + + +class MockEntityInfo(BaseDataModel): + """ + A mock implementation of `BaseDataModel` used for testing `DatabaseEntity`. + + Attributes: + id (str): The ID of the mock entity. + additional_data (Dict, optional): Any extra data associated with the entity. + """ + + def __init__(self, id: str, additional_data: Dict = None): + """ + Constructor method. + + Args: + id (str): The ID of the mock entity. + additional_data (Dict, optional): Any extra data associated with the entity. + """ + self.id = id + self.additional_data = additional_data or {} + + @property + def fields_allowed_to_be_updated(self) -> List: + """ + Return the list of fields allowed to be updated. + + Returns: + An empty list, as updates are not allowed for this mock. + """ + return [] + + +class TestEntity(DatabaseEntity[MockEntityInfo]): + """ + A concrete implementation of `DatabaseEntity` for testing purposes. + """ + + def __repr__(self) -> str: + """Return the official string representation of the entity.""" + return f"TestEntity(id={self.get_id()})" + + def __str__(self) -> str: + """Return a user-friendly string representation of the entity.""" + return f"Test Entity {self.get_id()}" + + @classmethod + def _get_entity_type(cls) -> str: + """ + Return the type of entity used for backend operations. + + Returns: + The string `"test_entity"`. + """ + return "test_entity" + + +class DefaultTypeEntity(DatabaseEntity[MockEntityInfo]): + """ + A DatabaseEntity subclass that does not override `_get_entity_type`, + used to test default entity type inference. + """ + + def __repr__(self) -> str: + """Return the official string representation of the entity.""" + return f"DefaultTypeEntity(id={self.get_id()})" + + def __str__(self) -> str: + """Return a user-friendly string representation of the entity.""" + return f"Default Type Entity {self.get_id()}" + + +class TestDatabaseEntity: + """Test suite for the `DatabaseEntity` base class.""" + + @pytest.fixture + def mock_backend(self, mocker: MockerFixture) -> Mock: + """ + Fixture to provide a mocked `ResultsBackend`. + + Args: + mocker (MockerFixture): A pytest-mock fixture for mocking. + + Returns: + A mock `ResultsBackend` instance. + """ + return mocker.Mock(spec=ResultsBackend) + + @pytest.fixture + def entity_info(self) -> MockEntityInfo: + """ + Fixture to provide a sample `MockEntityInfo` instance. + + Returns: + A mock data model for testing. + """ + return MockEntityInfo(id="test-123", additional_data={"key": "value"}) + + @pytest.fixture + def entity(self, entity_info: MockEntityInfo, mock_backend: Mock) -> TestEntity: + """ + Fixture to provide a `TestEntity` instance. + + Args: + entity_info (MockEntityInfo): The mock entity data. + mock_backend (Mock): The mock backend. + + Returns: + The entity under test. + """ + return TestEntity(entity_info, mock_backend) + + def test_init(self, entity: TestEntity, entity_info: MockEntityInfo, mock_backend: Mock): + """ + Test that the entity is initialized with the correct info and backend. + + Args: + entity: A fixture that returns an entity object for testing. + entity_info: A fixture that returns a mocked data model instance. + mock_backend: A fixture that returns a mocked results backend instance. + """ + assert entity.entity_info == entity_info + assert entity.backend == mock_backend + + def test_get_id(self, entity: TestEntity): + """ + Test that get_id returns the correct entity ID. + + Args: + entity: A fixture that returns an entity object for testing. + """ + assert entity.get_id() == "test-123" + + def test_repr(self, entity: TestEntity): + """ + Test the __repr__ output. + + Args: + entity: A fixture that returns an entity object for testing. + """ + assert repr(entity) == "TestEntity(id=test-123)" + + def test_str(self, entity: TestEntity): + """ + Test the __str__ output. + + Args: + entity: A fixture that returns an entity object for testing. + """ + assert str(entity) == "Test Entity test-123" + + def test_entity_type_custom(self, entity: TestEntity): + """ + Test that a custom entity type is correctly returned. + + Args: + entity: A fixture that returns an entity object for testing. + """ + assert entity.entity_type == "test_entity" + + def test_entity_type_default(self, entity_info: MockEntityInfo, mock_backend: Mock): + """ + Test that the default entity type is inferred from the class name. + + Should return 'defaulttype' for `DefaultTypeEntity`. + + Args: + entity_info: A fixture that returns a mocked data model instance. + mock_backend: A fixture that returns a mocked results backend instance. + """ + default_entity = DefaultTypeEntity(entity_info, mock_backend) + # Should use class name without "entity" suffix, lowercase + assert default_entity.entity_type == "defaulttype" + + def test_reload_data_success(self, entity: TestEntity, mock_backend: Mock): + """ + Test successful reload of entity data from the backend. + + Args: + entity: A fixture that returns an entity object for testing. + mock_backend: A fixture that returns a mocked results backend instance. + """ + new_entity_info = MockEntityInfo(id="test-123", additional_data={"key": "updated"}) + mock_backend.retrieve.return_value = new_entity_info + + entity.reload_data() + + mock_backend.retrieve.assert_called_once_with("test-123", "test_entity") + assert entity.entity_info == new_entity_info + + def test_reload_data_not_found(self, entity: TestEntity, mock_backend: Mock): + """ + Test that `EntityNotFoundError` is raised when the entity is not found. + + Args: + entity: A fixture that returns an entity object for testing. + mock_backend: A fixture that returns a mocked results backend instance. + """ + mock_backend.retrieve.return_value = None + + with pytest.raises(EntityNotFoundError) as excinfo: + entity.reload_data() + + assert "Test_entity with ID test-123 not found" in str(excinfo.value) + mock_backend.retrieve.assert_called_once_with("test-123", "test_entity") + + def test_get_additional_data(self, entity: TestEntity, mock_backend: Mock): + """ + Test that additional data can be retrieved correctly. + + Args: + entity: A fixture that returns an entity object for testing. + mock_backend: A fixture that returns a mocked results backend instance. + """ + new_entity_info = MockEntityInfo(id="test-123", additional_data={"key": "updated"}) + mock_backend.retrieve.return_value = new_entity_info + + result = entity.get_additional_data() + + assert result == {"key": "updated"} + mock_backend.retrieve.assert_called_once_with("test-123", "test_entity") + + def test_save(self, mocker: MockerFixture, entity: TestEntity, mock_backend: Mock): + """ + Test that the entity is saved correctly and `_post_save_hook` is called. + + Args: + mocker: A pytest-mock fixture for mocking. + entity: A fixture that returns an entity object for testing. + mock_backend: A fixture that returns a mocked results backend instance. + """ + # Mock _post_save_hook to verify it's called + entity._post_save_hook = mocker.Mock() + + entity.save() + + mock_backend.save.assert_called_once_with(entity.entity_info) + entity._post_save_hook.assert_called_once() + + def test_post_save_hook(self, entity: TestEntity): + """ + Test that the default `_post_save_hook` implementation runs without error. + + Args: + entity: A fixture that returns an entity object for testing. + """ + # Default implementation should do nothing, but be callable + entity._post_save_hook() # Should not raise any exception + + def test_load_success(self, mock_backend: Mock): + """ + Test successful loading of an entity from the backend. + + Args: + mock_backend: A fixture that returns a mocked results backend instance. + """ + entity_info = MockEntityInfo(id="test-123") + mock_backend.retrieve.return_value = entity_info + + entity = TestEntity.load("test-123", mock_backend) + + assert isinstance(entity, TestEntity) + assert entity.get_id() == "test-123" + mock_backend.retrieve.assert_called_once_with("test-123", "test_entity") + + def test_load_not_found(self, mock_backend: Mock): + """ + Test that loading a non-existent entity raises `EntityNotFoundError`. + + Args: + mock_backend: A fixture that returns a mocked results backend instance. + """ + mock_backend.retrieve.return_value = None + + with pytest.raises(EntityNotFoundError) as excinfo: + TestEntity.load("test-123", mock_backend) + + assert "Test_entity with ID test-123 not found" in str(excinfo.value) + mock_backend.retrieve.assert_called_once_with("test-123", "test_entity") + + def test_delete(self, mock_backend: Mock): + """ + Test that the `delete` method calls the backend with the correct arguments. + + Args: + mock_backend: A fixture that returns a mocked results backend instance. + """ + TestEntity.delete("test-123", mock_backend) + + mock_backend.delete.assert_called_once_with("test-123", "test_entity") + + def test_error_classes_mapping(self): + """ + Test that error class mappings for known entity types are correct. + """ + # Test the error class mapping for different entity types + assert TestEntity._error_classes["run"] == RunNotFoundError + assert TestEntity._error_classes["study"] == StudyNotFoundError + assert TestEntity._error_classes["logical_worker"] == WorkerNotFoundError + assert TestEntity._error_classes["physical_worker"] == WorkerNotFoundError + + def test_specific_error_classes(self, entity_info: MockEntityInfo, mock_backend: Mock): + """ + Test that appropriate error classes are raised for specific entity types. + + Args: + entity_info: A fixture that returns a mocked data model instance. + mock_backend: A fixture that returns a mocked results backend instance. + """ + + # Create entities with types that map to specific error classes + class RunEntity(TestEntity): + @classmethod + def _get_entity_type(cls) -> str: + return "run" + + class StudyEntity(TestEntity): + @classmethod + def _get_entity_type(cls) -> str: + return "study" + + # Test run entity not found error + mock_backend.retrieve.return_value = None + with pytest.raises(RunNotFoundError): + run_entity = RunEntity(entity_info, mock_backend) + run_entity.reload_data() + + # Test study entity not found error + with pytest.raises(StudyNotFoundError): + study_entity = StudyEntity(entity_info, mock_backend) + study_entity.reload_data() diff --git a/tests/unit/db_scripts/entities/test_logical_worker_entity.py b/tests/unit/db_scripts/entities/test_logical_worker_entity.py new file mode 100644 index 000000000..d21205bbd --- /dev/null +++ b/tests/unit/db_scripts/entities/test_logical_worker_entity.py @@ -0,0 +1,223 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `logical_worker_entity.py` module. +""" + +from unittest.mock import MagicMock, patch + +import pytest + +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.data_models import LogicalWorkerModel +from merlin.db_scripts.entities.logical_worker_entity import LogicalWorkerEntity +from merlin.db_scripts.entities.physical_worker_entity import PhysicalWorkerEntity + + +class TestLogicalWorkerEntity: + """Tests for the `LogicalWorkerEntity` class.""" + + @pytest.fixture + def mock_model(self) -> MagicMock: + """ + Create a mock `LogicalWorkerModel` for testing. + + Returns: + A mocked `LogicalWorkerModel` instance. + """ + model = MagicMock(spec=LogicalWorkerModel) + model.id = "logical_123" + model.name = "test_logical_worker" + model.runs = ["run_1", "run_2"] + model.queues = ["queue_1", "queue_2"] + model.physical_workers = ["physical_1", "physical_2"] + model.additional_data = {"key": "value"} + return model + + @pytest.fixture + def mock_backend(self, mock_model: MagicMock) -> MagicMock: + """ + Create a mock `ResultsBackend` for testing. + + Args: + mock_model: A mocked `LogicalWorkerModel` instance. + + Returns: + A mocked `ResultsBackend` instance. + """ + backend = MagicMock(spec=ResultsBackend) + backend.get_name.return_value = "mock_backend" + # Mock the retrieve return so it always returns our test model + backend.retrieve.return_value = mock_model + return backend + + @pytest.fixture + def logical_worker_entity(self, mock_model: MagicMock, mock_backend: MagicMock): + """ + Create a `LogicalWorkerEntity` instance for testing. + + Args: + mock_model: A mocked `LogicalWorkerEntity` instance. + mock_backend: A mocked `ResultsBackend` instance. + + Returns: + A `LogicalWorkerEntity` instance. + """ + return LogicalWorkerEntity(mock_model, mock_backend) + + @pytest.fixture + def mock_physical_worker(self) -> PhysicalWorkerEntity: + """ + Mock `PhysicalWorkerEntity` for testing. + + Returns: + A mocked `PhysicalWorkerEntity` instance. + """ + worker = MagicMock(spec=PhysicalWorkerEntity) + worker.get_id.return_value = "physical_1" + worker.get_name.return_value = "test_physical_worker" + return worker + + def test_get_entity_type(self): + """Test that _get_entity_type returns the correct value.""" + assert LogicalWorkerEntity._get_entity_type() == "logical_worker" + + def test_repr(self, logical_worker_entity: LogicalWorkerEntity): + """ + Test the __repr__ method. + + Args: + logical_worker_entity: A fixture that returns a `LogicalWorkerEntity` instance. + """ + repr_str = repr(logical_worker_entity) + assert "LogicalWorkerEntity" in repr_str + assert f"id={logical_worker_entity.get_id()}" in repr_str + assert f"name={logical_worker_entity.get_name()}" in repr_str + + def test_str(self, logical_worker_entity: LogicalWorkerEntity, mock_physical_worker: PhysicalWorkerEntity): + """ + Test the __str__ method. + + Args: + logical_worker_entity: A fixture that returns a `LogicalWorkerEntity` instance. + mock_physical_worker: A fixture that returns a `PhysicalWorkerEntity` instance. + """ + with patch.object(PhysicalWorkerEntity, "load", return_value=mock_physical_worker): + str_output = str(logical_worker_entity) + assert f"Logical Worker with ID {logical_worker_entity.get_id()}" in str_output + assert f"Name: {logical_worker_entity.get_name()}" in str_output + assert "Physical Workers:" in str_output + + def test_get_physical_workers(self, logical_worker_entity: LogicalWorkerEntity, mock_model: MagicMock): + """ + Test get_physical_workers returns the correct value. + + Args: + logical_worker_entity: A fixture that returns a `LogicalWorkerEntity` instance. + mock_model: A fixture that returns a mocked `LogicalWorkerModel` instance. + """ + assert logical_worker_entity.get_physical_workers() == mock_model.physical_workers + + def test_add_physical_worker(self, logical_worker_entity: LogicalWorkerEntity, mock_backend: MagicMock): + """ + Test add_physical_worker adds the worker and saves the model. + + Args: + logical_worker_entity: A fixture that returns a `LogicalWorkerEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + new_worker_id = "physical_3" + logical_worker_entity.add_physical_worker(new_worker_id) + assert new_worker_id in logical_worker_entity.entity_info.physical_workers + mock_backend.save.assert_called_once() + + def test_remove_physical_worker(self, logical_worker_entity: LogicalWorkerEntity, mock_backend: MagicMock): + """ + Test remove_physical_worker removes the worker and saves the model. + + Args: + logical_worker_entity: A fixture that returns a `LogicalWorkerEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + worker_id = "physical_1" + logical_worker_entity.remove_physical_worker(worker_id) + assert worker_id not in logical_worker_entity.entity_info.physical_workers + mock_backend.save.assert_called_once() + + def test_get_runs(self, logical_worker_entity: LogicalWorkerEntity, mock_model: MagicMock): + """ + Test get_runs returns the correct value. + + Args: + logical_worker_entity: A fixture that returns a `LogicalWorkerEntity` instance. + mock_model: A fixture that returns a mocked `LogicalWorkerModel` instance. + """ + assert logical_worker_entity.get_runs() == mock_model.runs + + def test_add_run(self, logical_worker_entity: LogicalWorkerEntity, mock_backend: MagicMock): + """ + Test add_run adds the run and saves the model. + + Args: + logical_worker_entity: A fixture that returns a `LogicalWorkerEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + new_run_id = "run_3" + logical_worker_entity.add_run(new_run_id) + assert new_run_id in logical_worker_entity.entity_info.runs + mock_backend.save.assert_called_once() + + def test_remove_run(self, logical_worker_entity: LogicalWorkerEntity, mock_backend: MagicMock): + """ + Test remove_run removes the run and saves the model. + + Args: + logical_worker_entity: A fixture that returns a `LogicalWorkerEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + run_id = "run_1" + logical_worker_entity.remove_run(run_id) + assert run_id not in logical_worker_entity.entity_info.runs + mock_backend.save.assert_called_once() + + def test_get_queues(self, logical_worker_entity: LogicalWorkerEntity, mock_model: MagicMock): + """ + Test get_queues returns the correct value. + + Args: + logical_worker_entity: A fixture that returns a `LogicalWorkerEntity` instance. + mock_model: A fixture that returns a mocked `LogicalWorkerModel` instance. + """ + assert logical_worker_entity.get_queues() == mock_model.queues + + @patch.object(LogicalWorkerEntity, "load") + def test_load(self, mock_load: MagicMock, mock_backend: MagicMock): + """ + Test the load class method. + + Args: + mock_load: A mocked version of the `load` method of `LogicalWorkerEntity`. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + entity_id = "logical_123" + mock_load.return_value = "loaded_entity" + result = LogicalWorkerEntity.load(entity_id, mock_backend) + mock_load.assert_called_once_with(entity_id, mock_backend) + assert result == "loaded_entity" + + @patch.object(LogicalWorkerEntity, "delete") + def test_delete(self, mock_delete: MagicMock, mock_backend: MagicMock): + """ + Test the delete class method. + + Args: + mock_delete: A mocked version of the `delete` method of `LogicalWorkerEntity`. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + entity_id = "logical_123" + LogicalWorkerEntity.delete(entity_id, mock_backend) + mock_delete.assert_called_once_with(entity_id, mock_backend) diff --git a/tests/unit/db_scripts/entities/test_mixins.py b/tests/unit/db_scripts/entities/test_mixins.py new file mode 100644 index 000000000..b9fb1b120 --- /dev/null +++ b/tests/unit/db_scripts/entities/test_mixins.py @@ -0,0 +1,216 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for database-related mixin classes. +""" + +from dataclasses import dataclass +from typing import List +from unittest.mock import MagicMock, patch + +import pytest + +from merlin.db_scripts.entities.mixins.name import NameMixin +from merlin.db_scripts.entities.mixins.queue_management import QueueManagementMixin +from merlin.db_scripts.entities.mixins.run_management import RunManagementMixin + + +# Test fixtures and helper classes + + +@dataclass +class EntityInfo: + """Simple data class to mimic `entity_info` objects""" + + name: str = "" + queues: List[str] = None + runs: List[str] = None + + def __post_init__(self): + if self.queues is None: + self.queues = [] + if self.runs is None: + self.runs = [] + + +class TestClass(NameMixin, QueueManagementMixin, RunManagementMixin): + """A test class that uses all the mixins""" + + def __init__(self): + self.entity_info = EntityInfo() + self.backend = "mock_backend" + self.reload_data_called = 0 + self.save_called = 0 + + def reload_data(self): + self.reload_data_called += 1 + + def save(self): + self.save_called += 1 + + +class TestNameMixin: + """Tests for the `NameMixin` class.""" + + def test_get_name(self): + """Test that `get_name` returns the correct name""" + test_obj = TestClass() + test_obj.entity_info.name = "Test Entity" + + assert test_obj.get_name() == "Test Entity" + + def test_get_name_empty(self): + """Test that `get_name` returns an empty string when name is empty""" + test_obj = TestClass() + test_obj.entity_info.name = "" + + assert test_obj.get_name() == "" + + +class TestQueueManagementMixin: + """Tests for the `QueueManagementMixin` class.""" + + def test_get_queues_empty(self): + """Test that `get_queues` returns an empty list when no queues are assigned""" + test_obj = TestClass() + + assert test_obj.get_queues() == [] + + def test_get_queues_with_values(self): + """Test that `get_queues` returns the correct list of queues""" + test_obj = TestClass() + test_obj.entity_info.queues = ["queue1", "queue2", "queue3"] + + queues = test_obj.get_queues() + assert len(queues) == 3 + assert "queue1" in queues + assert "queue2" in queues + assert "queue3" in queues + + +class TestRunManagementMixin: + """Tests for the `RunManagementMixin` class.""" + + def test_get_runs_empty(self): + """Test that `get_runs` returns an empty list when no runs are present""" + test_obj = TestClass() + + runs = test_obj.get_runs() + assert test_obj.reload_data_called == 1 + assert runs == [] + + def test_get_runs_with_values(self): + """Test that `get_runs` returns the correct list of runs""" + test_obj = TestClass() + test_obj.entity_info.runs = ["run1", "run2", "run3"] + + runs = test_obj.get_runs() + assert test_obj.reload_data_called == 1 + assert len(runs) == 3 + assert "run1" in runs + assert "run2" in runs + assert "run3" in runs + + def test_add_run(self): + """Test that `add_run` correctly adds a run ID to the list""" + test_obj = TestClass() + test_obj.entity_info.runs = ["run1"] + + test_obj.add_run("run2") + assert test_obj.save_called == 1 + assert len(test_obj.entity_info.runs) == 2 + assert "run1" in test_obj.entity_info.runs + assert "run2" in test_obj.entity_info.runs + + def test_remove_run(self): + """Test that `remove_run` correctly removes a run ID from the list""" + test_obj = TestClass() + test_obj.entity_info.runs = ["run1", "run2", "run3"] + + test_obj.remove_run("run2") + assert test_obj.reload_data_called == 1 + assert test_obj.save_called == 1 + assert len(test_obj.entity_info.runs) == 2 + assert "run1" in test_obj.entity_info.runs + assert "run3" in test_obj.entity_info.runs + assert "run2" not in test_obj.entity_info.runs + + def test_remove_run_not_in_list(self): + """Test that `remove_run` raises `ValueError` when run ID is not in the list""" + test_obj = TestClass() + test_obj.entity_info.runs = ["run1", "run3"] + + with pytest.raises(ValueError): + test_obj.remove_run("run2") + + def test_construct_run_string_empty(self): + """Test `construct_run_string` when no runs are present""" + test_obj = TestClass() + test_obj.entity_info.runs = [] + + run_str = test_obj.construct_run_string() + assert run_str == " No runs found.\n" + + @patch("merlin.db_scripts.entities.run_entity.RunEntity") + def test_construct_run_string_with_runs(self, mock_run_entity: MagicMock): + """ + Test `construct_run_string` with runs. + + Args: + mock_run_entity: A mocked version of the `RunEntity` class. + """ + # Setup the mock + mock_run1 = MagicMock() + mock_run1.get_id.return_value = "run1" + mock_run1.get_workspace.return_value = "workspace-run1" + + mock_run2 = MagicMock() + mock_run2.get_id.return_value = "run2" + mock_run2.get_workspace.return_value = "workspace-run2" + + mock_run_entity.load.side_effect = lambda run_id, backend: {"run1": mock_run1, "run2": mock_run2}[run_id] + + # Test the function + test_obj = TestClass() + test_obj.entity_info.runs = ["run1", "run2"] + + expected_str = " - ID: run1\n Workspace: workspace-run1\n" + " - ID: run2\n Workspace: workspace-run2\n" + + with patch("merlin.db_scripts.entities.run_entity.RunEntity", mock_run_entity): + run_str = test_obj.construct_run_string() + assert run_str == expected_str + + @patch("merlin.db_scripts.entities.run_entity.RunEntity") + def test_construct_run_string_with_error(self, mock_run_entity: MagicMock): + """ + Test `construct_run_string` when an error occurs loading a run + + Args: + mock_run_entity: A mocked version of the `RunEntity` class. + """ + # Setup the mock + mock_run1 = MagicMock() + mock_run1.get_id.return_value = "run1" + mock_run1.get_workspace.return_value = "workspace-run1" + + def mock_load(run_id, backend): + if run_id == "run1": + return mock_run1 + else: + raise Exception("Error loading run") + + mock_run_entity.load.side_effect = mock_load + + # Test the function + test_obj = TestClass() + test_obj.entity_info.runs = ["run1", "run2"] + + expected_str = " - ID: run1\n Workspace: workspace-run1\n" + " - ID: run2 (Error loading run)\n" + + with patch("merlin.db_scripts.entities.run_entity.RunEntity", mock_run_entity): + run_str = test_obj.construct_run_string() + assert run_str == expected_str diff --git a/tests/unit/db_scripts/entities/test_physical_worker_entity.py b/tests/unit/db_scripts/entities/test_physical_worker_entity.py new file mode 100644 index 000000000..f5095a0de --- /dev/null +++ b/tests/unit/db_scripts/entities/test_physical_worker_entity.py @@ -0,0 +1,324 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `physical_worker_entity.py` module. +""" + +from datetime import datetime +from unittest.mock import MagicMock, patch + +import pytest + +from merlin.backends.results_backend import ResultsBackend +from merlin.common.enums import WorkerStatus +from merlin.db_scripts.data_models import PhysicalWorkerModel +from merlin.db_scripts.entities.physical_worker_entity import PhysicalWorkerEntity + + +class TestPhysicalWorkerEntity: + """Tests for the `PhysicalWorkerEntity` class.""" + + @pytest.fixture + def mock_model(self) -> MagicMock: + """ + Create a mock `PhysicalWorkerModel` for testing. + + Returns: + A mocked `PhysicalWorkerModel` instance. + """ + model = MagicMock(spec=PhysicalWorkerModel) + model.id = "test_id" + model.name = "test_worker" + model.logical_worker_id = "logical_worker_123" + model.launch_cmd = "python worker.py" + model.args = {"arg1": "value1"} + model.pid = "12345" + model.status = WorkerStatus.RUNNING + model.heartbeat_timestamp = datetime.now().isoformat() + model.latest_start_time = datetime.now() + model.host = "test_host" + model.restart_count = 0 + model.additional_data = {"key": "value"} + return model + + @pytest.fixture + def mock_backend(self, mock_model: MagicMock) -> MagicMock: + """ + Create a mock `ResultsBackend` for testing. + + Args: + mock_model: A mocked `PhysicalWorkerModel` instance. + + Returns: + A mocked `ResultsBackend` instance. + """ + backend = MagicMock(spec=ResultsBackend) + backend.get_name.return_value = "mock_backend" + # Mock the retrieve return so it always returns our test model + backend.retrieve.return_value = mock_model + return backend + + @pytest.fixture + def worker_entity(self, mock_model: MagicMock, mock_backend: MagicMock) -> PhysicalWorkerEntity: + """ + Create a `PhysicalWorkerEntity` instance for testing. + + Args: + mock_model: A mocked `PhysicalWorkerModel` instance. + mock_backend: A mocked `ResultsBackend` instance. + + Returns: + A `PhysicalWorkerEntity` instance. + """ + return PhysicalWorkerEntity(mock_model, mock_backend) + + def test_get_entity_type(self): + """Test that _get_entity_type returns the correct value.""" + assert PhysicalWorkerEntity._get_entity_type() == "physical_worker" + + def test_repr(self, worker_entity: PhysicalWorkerEntity): + """ + Test the __repr__ method. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + """ + repr_str = repr(worker_entity) + assert "PhysicalWorkerEntity" in repr_str + assert f"id={worker_entity.get_id()}" in repr_str + assert f"name={worker_entity.get_name()}" in repr_str + + def test_str(self, worker_entity: PhysicalWorkerEntity): + """ + Test the __str__ method. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + """ + str_output = str(worker_entity) + assert f"Physical Worker with ID {worker_entity.get_id()}" in str_output + assert f"Name: {worker_entity.get_name()}" in str_output + + def test_get_logical_worker_id(self, worker_entity: PhysicalWorkerEntity, mock_model: MagicMock): + """ + Test get_logical_worker_id returns the correct value. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_model: A fixture that returns a mocked `PhysicalWorkerModel` instance. + """ + assert worker_entity.get_logical_worker_id() == mock_model.logical_worker_id + + def test_get_launch_cmd(self, worker_entity: PhysicalWorkerEntity, mock_model: MagicMock): + """ + Test get_launch_cmd returns the correct value. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_model: A fixture that returns a mocked `PhysicalWorkerModel` instance. + """ + assert worker_entity.get_launch_cmd() == mock_model.launch_cmd + + def test_set_launch_cmd(self, worker_entity: PhysicalWorkerEntity, mock_backend: MagicMock): + """ + Test set_launch_cmd updates the model and saves it. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + new_cmd = "python new_worker.py" + worker_entity.set_launch_cmd(new_cmd) + assert worker_entity.entity_info.launch_cmd == new_cmd + mock_backend.save.assert_called_once() + + def test_get_args(self, worker_entity: PhysicalWorkerEntity, mock_model: MagicMock): + """ + Test get_args returns the correct value. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_model: A fixture that returns a mocked `PhysicalWorkerModel` instance. + """ + assert worker_entity.get_args() == mock_model.args + + def test_set_args(self, worker_entity: PhysicalWorkerEntity, mock_backend: MagicMock): + """ + Test set_args updates the model and saves it. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + new_args = {"arg2": "value2"} + worker_entity.set_args(new_args) + assert worker_entity.entity_info.args == new_args + mock_backend.save.assert_called_once() + + def test_get_pid(self, worker_entity: PhysicalWorkerEntity, mock_model: MagicMock): + """ + Test get_pid returns the correct value. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_model: A fixture that returns a mocked `PhysicalWorkerModel` instance. + """ + assert worker_entity.get_pid() == int(mock_model.pid) + + def test_get_pid_none(self, worker_entity: PhysicalWorkerEntity, mock_model: MagicMock): + """ + Test get_pid returns None when pid is not set. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_model: A fixture that returns a mocked `PhysicalWorkerModel` instance. + """ + mock_model.pid = None + assert worker_entity.get_pid() is None + + def test_set_pid(self, worker_entity: PhysicalWorkerEntity, mock_backend: MagicMock): + """ + Test set_pid updates the model and saves it. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + new_pid = "54321" + worker_entity.set_pid(new_pid) + assert worker_entity.entity_info.pid == new_pid + mock_backend.save.assert_called_once() + + def test_get_status(self, worker_entity: PhysicalWorkerEntity, mock_model: MagicMock): + """ + Test get_status returns the correct value. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_model: A fixture that returns a mocked `PhysicalWorkerModel` instance. + """ + assert worker_entity.get_status() == mock_model.status + + def test_set_status(self, worker_entity: PhysicalWorkerEntity, mock_backend: MagicMock): + """ + Test set_status updates the model and saves it. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + new_status = WorkerStatus.STOPPED + worker_entity.set_status(new_status) + assert worker_entity.entity_info.status == new_status + mock_backend.save.assert_called_once() + + def test_get_heartbeat_timestamp(self, worker_entity: PhysicalWorkerEntity, mock_model: MagicMock): + """ + Test get_heartbeat_timestamp returns the correct value. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_model: A fixture that returns a mocked `PhysicalWorkerModel` instance. + """ + assert worker_entity.get_heartbeat_timestamp() == mock_model.heartbeat_timestamp + + def test_set_heartbeat_timestamp(self, worker_entity: PhysicalWorkerEntity, mock_backend: MagicMock): + """ + Test set_heartbeat_timestamp updates the model and saves it. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + new_timestamp = datetime.now() + worker_entity.set_heartbeat_timestamp(new_timestamp) + assert worker_entity.entity_info.heartbeat_timestamp == new_timestamp + mock_backend.save.assert_called_once() + + def test_get_latest_start_time(self, worker_entity: PhysicalWorkerEntity, mock_model: MagicMock): + """ + Test get_latest_start_time returns the correct value. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_model: A fixture that returns a mocked `PhysicalWorkerModel` instance. + """ + assert worker_entity.get_latest_start_time() == mock_model.latest_start_time + + def test_set_latest_start_time(self, worker_entity: PhysicalWorkerEntity, mock_backend: MagicMock): + """ + Test set_latest_start_time updates the model and saves it. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + new_start_time = datetime.now() + worker_entity.set_latest_start_time(new_start_time) + assert worker_entity.entity_info.latest_start_time == new_start_time + mock_backend.save.assert_called_once() + + def test_get_host(self, worker_entity: PhysicalWorkerEntity, mock_model: MagicMock): + """ + Test get_host returns the correct value. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_model: A fixture that returns a mocked `PhysicalWorkerModel` instance. + """ + assert worker_entity.get_host() == mock_model.host + + def test_get_restart_count(self, worker_entity: PhysicalWorkerEntity, mock_model: MagicMock): + """ + Test get_restart_count returns the correct value. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_model: A fixture that returns a mocked `PhysicalWorkerModel` instance. + """ + assert worker_entity.get_restart_count() == mock_model.restart_count + + def test_increment_restart_count(self, worker_entity: PhysicalWorkerEntity, mock_backend: MagicMock): + """ + Test increment_restart_count increments the count and saves the model. + + Args: + worker_entity: A fixture that returns a `PhysicalWorkerEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + initial_count = worker_entity.get_restart_count() + worker_entity.increment_restart_count() + assert worker_entity.entity_info.restart_count == initial_count + 1 + mock_backend.save.assert_called_once() + + @patch.object(PhysicalWorkerEntity, "load") + def test_load(self, mock_load: MagicMock, mock_backend: MagicMock): + """ + Test the load class method. + + Args: + mock_load: A mocked version of the `load` method of `PhysicalWorkerEntity`. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + entity_id = "worker_123" + mock_load.return_value = "loaded_entity" + result = PhysicalWorkerEntity.load(entity_id, mock_backend) + mock_load.assert_called_once_with(entity_id, mock_backend) + assert result == "loaded_entity" + + @patch.object(PhysicalWorkerEntity, "delete") + def test_delete(self, mock_delete: MagicMock, mock_backend: MagicMock): + """ + Test the delete class method. + + Args: + mock_delete: A mocked version of the `delete` method of `PhysicalWorkerEntity`. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + entity_id = "worker_123" + PhysicalWorkerEntity.delete(entity_id, mock_backend) + mock_delete.assert_called_once_with(entity_id, mock_backend) diff --git a/tests/unit/db_scripts/entities/test_run_entity.py b/tests/unit/db_scripts/entities/test_run_entity.py new file mode 100644 index 000000000..13ee3d98f --- /dev/null +++ b/tests/unit/db_scripts/entities/test_run_entity.py @@ -0,0 +1,333 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `run_entity.py` module. +""" + +from unittest.mock import MagicMock, patch + +import pytest + +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.data_models import RunModel +from merlin.db_scripts.entities.run_entity import RunEntity +from merlin.db_scripts.entities.study_entity import StudyEntity + + +class TestRunEntity: + """Tests for the `RunEntity` class.""" + + @pytest.fixture + def mock_model(self) -> MagicMock: + """ + Create a mock `RunModel` for testing. + + Returns: + A mocked `RunModel` instance. + """ + model = MagicMock(spec=RunModel) + model.id = "run_123" + model.study_id = "study_123" + model.workspace = "/test/workspace" + model.queues = ["queue_1", "queue_2"] + model.workers = ["worker_1", "worker_2"] + model.parent = "parent_run" + model.child = "child_run" + model.run_complete = False + model.additional_data = {"key": "value"} + return model + + @pytest.fixture + def mock_backend(self, mock_model: MagicMock) -> MagicMock: + """ + Create a mock `ResultsBackend` for testing. + + Args: + mock_model: A mocked `RunModel` instance. + + Returns: + A mocked `ResultsBackend` instance. + """ + backend = MagicMock(spec=ResultsBackend) + backend.get_name.return_value = "mock_backend" + backend.retrieve.return_value = mock_model + return backend + + @pytest.fixture + def run_entity(self, mock_model: MagicMock, mock_backend: MagicMock) -> RunEntity: + """ + Create a `RunEntity` instance for testing. + + Args: + mock_model: A mocked `RunModel` instance. + mock_backend: A mocked `ResultsBackend` instance. + + Returns: + A `RunEntity` instance. + """ + with patch("os.path.join", return_value="/test/workspace/merlin_info/run_metadata.json"): + return RunEntity(mock_model, mock_backend) + + @pytest.fixture + def mock_study(self) -> MagicMock: + """ + Mock `StudyEntity` for testing. + + Returns: + A mocked `StudyEntity` instance. + """ + study = MagicMock(spec=StudyEntity) + study.get_id.return_value = "study_123" + study.get_name.return_value = "test_study" + return study + + def test_get_entity_type(self): + """ + Test that `_get_entity_type` returns the correct value. + """ + assert RunEntity._get_entity_type() == "run" + + def test_init(self, mock_model: MagicMock, mock_backend: MagicMock): + """ + Test that initialization sets the `_metadata_file` correctly. + + Args: + mock_model: A mocked `RunModel` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + with patch("os.path.join", return_value="/test/workspace/merlin_info/run_metadata.json"): + run_entity = RunEntity(mock_model, mock_backend) + assert run_entity._metadata_file == "/test/workspace/merlin_info/run_metadata.json" + + def test_repr(self, run_entity: RunEntity): + """ + Test the `__repr__` method. + + Args: + run_entity: A fixture that returns a `RunEntity` instance. + """ + repr_str = repr(run_entity) + assert "RunEntity" in repr_str + assert f"id={run_entity.get_id()}" in repr_str + assert f"study_id={run_entity.get_study_id()}" in repr_str + + def test_str(self, run_entity: RunEntity, mock_study: MagicMock): + """ + Test the `__str__` method. + + Args: + run_entity: A fixture that returns a `RunEntity` instance. + mock_study: A fixture that returns a mocked `StudyEntity` instance. + """ + with patch.object(StudyEntity, "load", return_value=mock_study): + str_output = str(run_entity) + assert f"Run with ID {run_entity.get_id()}" in str_output + assert f"Workspace: {run_entity.get_workspace()}" in str_output + assert "Study:" in str_output + + def test_run_complete_property(self, run_entity: RunEntity, mock_model: MagicMock): + """ + Test the `run_complete` property getter and setter. + + Args: + run_entity: A fixture that returns a `RunEntity` instance. + mock_model: A fixture that returns a mocked `RunModel` instance. + """ + assert run_entity.run_complete == mock_model.run_complete + run_entity.run_complete = True + assert run_entity.entity_info.run_complete is True + + def test_get_metadata_file(self, run_entity: RunEntity): + """ + Test `get_metadata_file` returns the correct value. + + Args: + run_entity: A fixture that returns a `RunEntity` instance. + """ + assert run_entity.get_metadata_file() == "/test/workspace/merlin_info/run_metadata.json" + + def test_get_metadata_filepath(self): + """ + Test `get_metadata_filepath` class method returns the correct path. + """ + with patch("os.path.join", return_value="/test/workspace/merlin_info/run_metadata.json"): + result = RunEntity.get_metadata_filepath("/test/workspace") + assert result == "/test/workspace/merlin_info/run_metadata.json" + + def test_get_study_id(self, run_entity: RunEntity, mock_model: MagicMock): + """ + Test `get_study_id` returns the correct value. + + Args: + run_entity: A fixture that returns a `RunEntity` instance. + mock_model: A fixture that returns a mocked `RunModel` instance. + """ + assert run_entity.get_study_id() == mock_model.study_id + + def test_get_workspace(self, run_entity: RunEntity, mock_model: MagicMock): + """ + Test `get_workspace` returns the correct value. + + Args: + run_entity: A fixture that returns a `RunEntity` instance. + mock_model: A fixture that returns a mocked `RunModel` instance. + """ + assert run_entity.get_workspace() == mock_model.workspace + + def test_get_workers(self, run_entity: RunEntity, mock_model: MagicMock): + """ + Test `get_workers` returns the correct value. + + Args: + run_entity: A fixture that returns a `RunEntity` instance. + mock_model: A fixture that returns a mocked `RunModel` instance. + """ + assert run_entity.get_workers() == mock_model.workers + + def test_add_worker(self, run_entity: RunEntity, mock_backend: MagicMock): + """ + Test `add_worker` adds the worker and saves the model. + + Args: + run_entity: A fixture that returns a `RunEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + new_worker_id = "worker_3" + run_entity.add_worker(new_worker_id) + assert new_worker_id in run_entity.entity_info.workers + mock_backend.save.assert_called_once() + + def test_remove_worker(self, run_entity: RunEntity, mock_backend: MagicMock): + """ + Test `remove_worker` removes the worker and saves the model. + + Args: + run_entity: A fixture that returns a `RunEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + worker_id = "worker_1" + run_entity.remove_worker(worker_id) + assert worker_id not in run_entity.entity_info.workers + mock_backend.save.assert_called_once() + + def test_get_parent(self, run_entity: RunEntity, mock_model: MagicMock): + """ + Test `get_parent` returns the correct value. + + Args: + run_entity: A fixture that returns a `RunEntity` instance. + mock_model: A fixture that returns a mocked `RunModel` instance. + """ + assert run_entity.get_parent() == mock_model.parent + + def test_get_child(self, run_entity: RunEntity, mock_model: MagicMock): + """ + Test `get_child` returns the correct value. + + Args: + run_entity: A fixture that returns a `RunEntity` instance. + mock_model: A fixture that returns a mocked `RunModel` instance. + """ + assert run_entity.get_child() == mock_model.child + + def test_get_queues(self, run_entity: RunEntity, mock_model: MagicMock): + """ + Test `get_queues` returns the correct value. + + Args: + run_entity: A fixture that returns a `RunEntity` instance. + mock_model: A fixture that returns a mocked `RunModel` instance. + """ + assert run_entity.get_queues() == mock_model.queues + + def test_post_save_hook(self, run_entity: RunEntity): + """ + Test `_post_save_hook` calls dump_metadata. + + Args: + run_entity: A fixture that returns a `RunEntity` instance. + """ + with patch.object(run_entity, "dump_metadata") as mock_dump: + run_entity._post_save_hook() + mock_dump.assert_called_once() + + def test_dump_metadata(self, run_entity: RunEntity): + """ + Test `dump_metadata` calls dump_to_json_file on the model. + + Args: + run_entity: A fixture that returns a `RunEntity` instance. + """ + run_entity.dump_metadata() + run_entity.entity_info.dump_to_json_file.assert_called_once_with(run_entity.get_metadata_file()) + + @patch("os.path.isdir", return_value=True) + @patch("os.path.exists", return_value=True) + @patch("merlin.db_scripts.data_models.RunModel.load_from_json_file") + def test_load_from_workspace( + self, + mock_load_json: MagicMock, + mock_exists: MagicMock, + mock_isdir: MagicMock, + mock_model: MagicMock, + mock_backend: MagicMock, + ): + """ + Test `load` method when loading from workspace. + + Args: + mock_load_json: A mocked version of the `load_from_json_file` method of `RunModel`. + mock_exists: A mocked version of the `os.path.exists` function. + mock_isdir: A mocked version of the `os.path.isdir` function. + mock_model: A fixture that returns a mocked `RunModel` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + workspace = "/test/workspace" + mock_load_json.return_value = mock_model + + with patch( + "merlin.db_scripts.entities.run_entity.RunEntity.get_metadata_filepath", + return_value="/test/workspace/merlin_info/run_metadata.json", + ): + run = RunEntity.load(workspace, mock_backend) + mock_load_json.assert_called_once_with("/test/workspace/merlin_info/run_metadata.json") + assert run.entity_info == mock_model + + @patch("os.path.isdir", return_value=False) + def test_load_from_id(self, mock_isdir: MagicMock, mock_model: MagicMock, mock_backend: MagicMock): + """ + Test `load` method when loading from ID. + + Args: + mock_isdir: A mocked version of the `os.path.isdir` function. + mock_model: A fixture that returns a mocked `RunModel` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + run_id = "run_123" + mock_backend.retrieve.return_value = mock_model + + run = RunEntity.load(run_id, mock_backend) + mock_backend.retrieve.assert_called_once_with(run_id, "run") + assert run.entity_info == mock_model + + @patch("merlin.db_scripts.entities.run_entity.RunEntity.load") + def test_delete(self, mock_load: MagicMock, mock_backend: MagicMock): + """ + Test `delete` method. + + Args: + mock_load: A mocked version of the `load` method of `StudyEntity`. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + run_id = "run_123" + mock_run = MagicMock() + mock_run.get_id.return_value = run_id + mock_load.return_value = mock_run + + RunEntity.delete(run_id, mock_backend) + mock_load.assert_called_once_with(run_id, mock_backend) + mock_backend.delete.assert_called_once_with(run_id, "run") diff --git a/tests/unit/db_scripts/entities/test_study_entity.py b/tests/unit/db_scripts/entities/test_study_entity.py new file mode 100644 index 000000000..06ec10e34 --- /dev/null +++ b/tests/unit/db_scripts/entities/test_study_entity.py @@ -0,0 +1,171 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `study_entity.py` module. +""" + +from unittest.mock import MagicMock, patch + +import pytest + +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.data_models import StudyModel +from merlin.db_scripts.entities.study_entity import StudyEntity + + +class TestStudyEntity: + """Tests for the `StudyEntity` class.""" + + @pytest.fixture + def mock_model(self) -> MagicMock: + """ + Create a mock `StudyModel` for testing. + + Returns: + A mocked `StudyModel` instance. + """ + model = MagicMock(spec=StudyModel) + model.id = "study_123" + model.name = "test_study" + model.runs = ["run_1", "run_2"] + model.additional_data = {"key": "value"} + return model + + @pytest.fixture + def mock_backend(self, mock_model: MagicMock) -> MagicMock: + """ + Create a mock `ResultsBackend` for testing. + + Args: + mock_model: A mocked `StudyModel` instance. + + Returns: + A mocked `ResultsBackend` instance. + """ + backend = MagicMock(spec=ResultsBackend) + backend.get_name.return_value = "mock_backend" + backend.retrieve.return_value = mock_model + return backend + + @pytest.fixture + def study_entity(self, mock_model: MagicMock, mock_backend: MagicMock) -> StudyEntity: + """ + Create a `StudyEntity` instance for testing. + + Args: + mock_model: A mocked `StudyModel` instance. + mock_backend: A mocked `ResultsBackend` instance. + + Returns: + A `StudyEntity` instance. + """ + return StudyEntity(mock_model, mock_backend) + + def test_get_entity_type(self): + """ + Test that `_get_entity_type` returns the correct value. + """ + assert StudyEntity._get_entity_type() == "study" + + def test_repr(self, study_entity: StudyEntity): + """ + Test the `__repr__` method. + + Args: + study_entity: A fixture that returns a `StudyEntity` instance. + """ + repr_str = repr(study_entity) + assert "StudyEntity" in repr_str + assert f"id={study_entity.get_id()}" in repr_str + assert f"name={study_entity.get_name()}" in repr_str + assert "runs=" in repr_str + + def test_str(self, study_entity: StudyEntity): + """ + Test the `__str__` method. + + Args: + study_entity: A fixture that returns a `StudyEntity` instance. + """ + str_output = str(study_entity) + assert f"Study with ID {study_entity.get_id()}" in str_output + assert f"Name: {study_entity.get_name()}" in str_output + assert "Runs:" in str_output + + def test_get_runs(self, study_entity: StudyEntity, mock_model: MagicMock): + """ + Test `get_runs` returns the correct value. + + Args: + study_entity: A fixture that returns a `StudyEntity` instance. + mock_model: A fixture that returns a mocked `StudyModel` instance. + """ + assert study_entity.get_runs() == mock_model.runs + + def test_add_run(self, study_entity: StudyEntity, mock_backend: MagicMock): + """ + Test `add_run` adds the run and saves the model. + + Args: + study_entity: A fixture that returns a `StudyEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + new_run_id = "run_3" + study_entity.add_run(new_run_id) + assert new_run_id in study_entity.entity_info.runs + mock_backend.save.assert_called_once() + + def test_remove_run(self, study_entity: StudyEntity, mock_backend: MagicMock): + """ + Test `remove_run` removes the run and saves the model. + + Args: + study_entity: A fixture that returns a `StudyEntity` instance. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + run_id = "run_1" + study_entity.remove_run(run_id) + assert run_id not in study_entity.entity_info.runs + mock_backend.save.assert_called_once() + + def test_get_name(self, study_entity: StudyEntity, mock_model: MagicMock): + """ + Test `get_name` returns the correct value. + + Args: + study_entity: A fixture that returns a `StudyEntity` instance. + mock_model: A fixture that returns a mocked `StudyModel` instance. + """ + assert study_entity.get_name() == mock_model.name + + @patch.object(StudyEntity, "load") + def test_load(self, mock_load: MagicMock, mock_backend: MagicMock): + """ + Test the `load` class method. + + Args: + mock_load: A mocked version of the `load` method of `StudyEntity`. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + entity_id = "study_123" + mock_load.return_value = "loaded_entity" + result = StudyEntity.load(entity_id, mock_backend) + mock_load.assert_called_once_with(entity_id, mock_backend) + assert result == "loaded_entity" + + @patch.object(StudyEntity, "delete") + def test_delete(self, mock_delete: MagicMock, mock_backend: MagicMock): + """ + Test the `delete` class method. + + Args: + mock_delete: A mocked version of the `delete` method of `StudyEntity`. + mock_backend: A fixture that returns a mocked `ResultsBackend` instance. + """ + entity_id = "study_123" + StudyEntity.delete(entity_id, mock_backend) + mock_delete.assert_called_once_with(entity_id, mock_backend) diff --git a/tests/unit/db_scripts/entity_managers/__init__.py b/tests/unit/db_scripts/entity_managers/__init__.py new file mode 100644 index 000000000..3232b50b9 --- /dev/null +++ b/tests/unit/db_scripts/entity_managers/__init__.py @@ -0,0 +1,5 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## diff --git a/tests/unit/db_scripts/entity_managers/test_entity_manager.py b/tests/unit/db_scripts/entity_managers/test_entity_manager.py new file mode 100644 index 000000000..fe123b225 --- /dev/null +++ b/tests/unit/db_scripts/entity_managers/test_entity_manager.py @@ -0,0 +1,409 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `entity_manager.py` module. +""" + +from dataclasses import dataclass +from typing import Any, List +from unittest.mock import MagicMock, call + +import pytest +from _pytest.capture import CaptureFixture + +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.data_models import BaseDataModel +from merlin.db_scripts.entities.db_entity import DatabaseEntity +from merlin.db_scripts.entity_managers.entity_manager import EntityManager +from merlin.exceptions import RunNotFoundError, StudyNotFoundError, WorkerNotFoundError + + +@dataclass +class TestDataModel(BaseDataModel): + """A concrete subclass of `BaseDataModel` for testing.""" + + id: str = "test_id" + name: str = "default" + attr1: str = "val" + + @property + def fields_allowed_to_be_updated(self): + return ["name", "attr1"] + + +class TestEntity(DatabaseEntity): + """A concrete test implementation of a database entity.""" + + def __init__(self, entity_info: TestDataModel, backend: ResultsBackend): + self.entity_info = entity_info + self.backend = backend + + def __repr__(self): + """Return the official string representation of the entity.""" + return f"TestEntity(id={self.get_id()})" + + def __str__(self) -> str: + """Return a user-friendly string representation of the entity.""" + return f"Test Entity {self.get_id()}" + + @classmethod + def load(cls, identifier: str, backend: ResultsBackend) -> "TestEntity": + try: + entity_info = backend.retrieve(identifier, "test_entity") + return cls(entity_info, backend) + except KeyError: + raise WorkerNotFoundError(f"Test entity {identifier} not found") + + @classmethod + def delete(cls, identifier: str, backend: ResultsBackend): + backend.delete(identifier, "test_entity") + + +class TestEntityManager(EntityManager[TestEntity, TestDataModel]): + """A concrete test implementation of an entity manager.""" + + def create(self, name: str, **kwargs: Any) -> TestEntity: + return self._create_entity_if_not_exists( + TestEntity, + TestDataModel, + name, + f"Test entity {name} already exists, loading it.", + f"Creating new test entity {name}", + name=name, + **kwargs, + ) + + def get(self, identifier: str) -> TestEntity: + return self._get_entity(TestEntity, identifier) + + def get_all(self) -> List[TestEntity]: + return self._get_all_entities(TestEntity, "test_entity") + + def delete(self, identifier: str, **kwargs: Any): + cleanup_fn = kwargs.get("cleanup_fn") + self._delete_entity(TestEntity, identifier, cleanup_fn=cleanup_fn) + + def delete_all(self, **kwargs: Any): + self._delete_all_by_type(self.get_all, self.delete, "test entities", **kwargs) + + +@pytest.fixture +def mock_backend() -> MagicMock: + """ + Create a mock backend instance. + + Returns: + A mocked `ResultsBackend` instance. + """ + backend = MagicMock(spec=ResultsBackend) + return backend + + +@pytest.fixture +def entity_manager(mock_backend: MagicMock) -> TestEntityManager: + """ + Create an instance of the base `EntityManager` for testing. + + Args: + mock_backend: A mocked `ResultsBackend` instance. + + Returns: + An instance of the base `EntityManager` for testing. + """ + return TestEntityManager(mock_backend) + + +class TestEntityManagerBase: + def test_init(self, entity_manager: TestEntityManager, mock_backend: MagicMock): + """ + Test initialization of `EntityManager`. + + Args: + entity_manager: An instance of the base `EntityManager` for testing. + mock_backend: A mocked `ResultsBackend` instance. + """ + assert entity_manager.backend == mock_backend + assert entity_manager.db is None + + def test_create_entity_if_not_exists_new(self, entity_manager: TestEntityManager, mock_backend: MagicMock): + """ + Test the `_create_entity_if_not_exists` method when the entity is new (does not yet + exist in the database). + + Args: + entity_manager: An instance of the base `EntityManager` for testing. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup backend to raise an error, simulating entity not found + mock_backend.retrieve.side_effect = KeyError() + + # Create a new entity + entity = entity_manager.create("test1", attr1="value1") + + # Verify the entity was created with correct attributes + assert isinstance(entity, TestEntity) + assert entity.entity_info.name == "test1" + assert entity.entity_info.attr1 == "value1" + + # Verify backend interactions + mock_backend.retrieve.assert_called_once_with("test1", "test_entity") + mock_backend.save.assert_called_once() + + def test_create_entity_if_not_exists_existing(self, entity_manager: TestEntityManager, mock_backend: MagicMock): + """ + Test the `_create_entity_if_not_exists` method when the entity is already exists in the database. + + Args: + entity_manager: An instance of the base `EntityManager` for testing. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup backend to return an existing entity + existing_model = TestDataModel(name="test1", attr1="value1") + mock_backend.retrieve.return_value = existing_model + + # Create/get the entity + entity = entity_manager.create("test1", attr2="value2") + + # Verify the existing entity was returned + assert isinstance(entity, TestEntity) + assert entity.entity_info.name == "test1" + assert entity.entity_info.attr1 == "value1" + assert not hasattr(entity.entity_info, "attr2") # Should not have the new attribute + + # Verify backend interactions + mock_backend.retrieve.assert_called_once_with("test1", "test_entity") + mock_backend.save.assert_not_called() + + def test_get_entity(self, entity_manager: TestEntityManager, mock_backend: MagicMock): + """ + Test the `get_entity` method with an existing entity. + + Args: + entity_manager: An instance of the base `EntityManager` for testing. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup backend to return an entity + existing_model = TestDataModel(name="test1", attr1="value1") + mock_backend.retrieve.return_value = existing_model + + # Get the entity + entity = entity_manager.get("test1") + + # Verify the entity was returned correctly + assert isinstance(entity, TestEntity) + assert entity.entity_info.name == "test1" + assert entity.entity_info.attr1 == "value1" + + # Verify backend interactions + mock_backend.retrieve.assert_called_once_with("test1", "test_entity") + + def test_get_entity_not_found(self, entity_manager: TestEntityManager, mock_backend: MagicMock): + """ + Test the `get_entity` method with a non-existing entity. + + Args: + entity_manager: An instance of the base `EntityManager` for testing. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup backend to raise an error + mock_backend.retrieve.side_effect = KeyError() + + # Attempt to get a non-existent entity + with pytest.raises(WorkerNotFoundError): + entity_manager.get("nonexistent") + + # Verify backend interactions + mock_backend.retrieve.assert_called_once_with("nonexistent", "test_entity") + + def test_get_all_entities_empty(self, entity_manager: TestEntityManager, mock_backend: MagicMock): + """ + Test the `get_all_entities` method with no entities in the database. + + Args: + entity_manager: An instance of the base `EntityManager` for testing. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup backend to return no entities + mock_backend.retrieve_all.return_value = [] + + # Get all entities + entities = entity_manager.get_all() + + # Verify result is empty + assert entities == [] + + # Verify backend interactions + mock_backend.retrieve_all.assert_called_once_with("test_entity") + + def test_get_all_entities(self, entity_manager: TestEntityManager, mock_backend: MagicMock): + """ + Test the `get_all_entities` method with entities in the database. + + Args: + entity_manager: An instance of the base `EntityManager` for testing. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup backend to return multiple entities + model1 = TestDataModel(name="test1", attr1="value1") + model2 = TestDataModel(name="test2", attr1="value2") + mock_backend.retrieve_all.return_value = [model1, model2] + + # Get all entities + entities = entity_manager.get_all() + + # Verify correct entities were returned + assert len(entities) == 2 + assert all(isinstance(entity, TestEntity) for entity in entities) + assert entities[0].entity_info.name == "test1" + assert entities[1].entity_info.name == "test2" + + # Verify backend interactions + mock_backend.retrieve_all.assert_called_once_with("test_entity") + + def test_delete_entity(self, entity_manager: TestEntityManager, mock_backend: MagicMock): + """ + Test the `delete_entity` method with an existing entity. + + Args: + entity_manager: An instance of the base `EntityManager` for testing. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup backend to return an entity for load + existing_model = TestDataModel(name="test1", attr1="value1") + mock_backend.retrieve.return_value = existing_model + + # Delete the entity + entity_manager.delete("test1") + + # Verify backend interactions + mock_backend.retrieve.assert_called_once_with("test1", "test_entity") + mock_backend.delete.assert_called_once_with("test1", "test_entity") + + def test_delete_entity_with_cleanup(self, entity_manager: TestEntityManager, mock_backend: MagicMock): + """ + Test the `delete_entity` method with a cleanup function. + + Args: + entity_manager: An instance of the base `EntityManager` for testing. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup backend to return an entity for load + existing_model = TestDataModel(name="test1", attr1="value1") + mock_backend.retrieve.return_value = existing_model + + # Create a cleanup function + cleanup_fn = MagicMock() + + # Delete the entity with cleanup + entity_manager.delete("test1", cleanup_fn=cleanup_fn) + + # Verify cleanup function was called with entity + cleanup_fn.assert_called_once() + called_entity = cleanup_fn.call_args[0][0] + assert isinstance(called_entity, TestEntity) + assert called_entity.entity_info.name == "test1" + + # Verify backend interactions + mock_backend.retrieve.assert_called_once_with("test1", "test_entity") + mock_backend.delete.assert_called_once_with("test1", "test_entity") + + def test_delete_all_empty(self, caplog: CaptureFixture, entity_manager: TestEntityManager, mock_backend: MagicMock): + """ + Test the `delete_all` method with a no existing entities in the database. + + Args: + caplog: A built-in fixture from the pytest library to capture logs. + entity_manager: An instance of the base `EntityManager` for testing. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup backend to return no entities + mock_backend.retrieve_all.return_value = [] + + # Delete all entities + entity_manager.delete_all() + + # Verify log warning was issued + assert "No test entities found in the database." in caplog.text + + # Verify backend interactions + mock_backend.retrieve_all.assert_called_once_with("test_entity") + mock_backend.delete.assert_not_called() + + def test_delete_all(self, entity_manager: TestEntityManager, mock_backend: MagicMock): + """ + Test the `delete_all` method with a existing entities. + + Args: + entity_manager: An instance of the base `EntityManager` for testing. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup backend to return multiple entities + model1 = TestDataModel(name="test1", attr1="value1") + model2 = TestDataModel(name="test2", attr1="value2") + mock_backend.retrieve_all.return_value = [model1, model2] + + # Delete all entities + entity_manager.delete_all() + + # Verify backend interactions + mock_backend.retrieve_all.assert_called_once_with("test_entity") + assert mock_backend.delete.call_count == 2 + mock_backend.delete.assert_has_calls([call("test_id", "test_entity"), call("test_id", "test_entity")], any_order=True) + + def test_delete_all_with_cleanup(self, entity_manager: TestEntityManager, mock_backend: MagicMock): + """ + Test the `delete_all` method with a cleanup function. + + Args: + entity_manager: An instance of the base `EntityManager` for testing. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup backend to return multiple entities + model1 = TestDataModel(name="test1", attr1="value1") + model2 = TestDataModel(name="test2", attr1="value2") + mock_backend.retrieve_all.return_value = [model1, model2] + + # Create a cleanup function + cleanup_fn = MagicMock() + + # Delete all entities with cleanup + entity_manager.delete_all(cleanup_fn=cleanup_fn) + + # Verify backend interactions + mock_backend.retrieve_all.assert_called_once_with("test_entity") + assert mock_backend.delete.call_count == 2 + mock_backend.delete.assert_has_calls([call("test_id", "test_entity"), call("test_id", "test_entity")], any_order=True) + + # Verify cleanup function was called for each entity + assert cleanup_fn.call_count == 2 + + @pytest.mark.parametrize("exception_type", [WorkerNotFoundError, StudyNotFoundError, RunNotFoundError]) + def test_create_entity_handles_various_not_found_errors( + self, entity_manager: TestEntityManager, mock_backend: MagicMock, exception_type: Exception + ): + """ + Test the `create` method to make sure it handles all of the exception types that could be raised. + + Args: + entity_manager: An instance of the base `EntityManager` for testing. + mock_backend: A mocked `ResultsBackend` instance. + exception_type: The type of exception that's being tested. + """ + # Setup backend to raise different types of not found errors + mock_backend.retrieve.side_effect = exception_type("Entity not found") + + # Create a new entity + entity = entity_manager.create("test1", attr1="value1") + + # Verify the entity was created + assert isinstance(entity, TestEntity) + assert entity.entity_info.name == "test1" + assert entity.entity_info.attr1 == "value1" + + # Verify backend interactions + mock_backend.retrieve.assert_called_once() + mock_backend.save.assert_called_once() diff --git a/tests/unit/db_scripts/entity_managers/test_logical_worker_manager.py b/tests/unit/db_scripts/entity_managers/test_logical_worker_manager.py new file mode 100644 index 000000000..4688289c4 --- /dev/null +++ b/tests/unit/db_scripts/entity_managers/test_logical_worker_manager.py @@ -0,0 +1,256 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `logical_worker_manager.py` module. +""" + +from unittest.mock import MagicMock, call, patch + +import pytest + +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.data_models import LogicalWorkerModel +from merlin.db_scripts.entities.logical_worker_entity import LogicalWorkerEntity +from merlin.db_scripts.entity_managers.logical_worker_manager import LogicalWorkerManager +from merlin.db_scripts.merlin_db import MerlinDatabase +from merlin.exceptions import RunNotFoundError + + +class TestLogicalWorkerManager: + """Tests for the `LogicalWorkerManager` class.""" + + @pytest.fixture + def mock_backend(self) -> MagicMock: + """ + Create a mock `ResultsBackend` for testing. + + Returns: + A mocked `ResultsBackend` instance. + """ + return MagicMock(spec=ResultsBackend) + + @pytest.fixture + def mock_db(self) -> MagicMock: + """ + Create a mock `MerlinDatabase` for testing. + + Returns: + A mocked `MerlinDatabase` instance. + """ + db = MagicMock(spec=MerlinDatabase) + db.runs = MagicMock() + return db + + @pytest.fixture + def worker_manager(self, mock_backend: MagicMock, mock_db: MagicMock) -> LogicalWorkerManager: + """ + Create a `LogicalWorkerManager` instance for testing. + + Args: + mock_backend: A mocked `ResultsBackend` instance. + mock_db: A mocked `MerlinDatabase` instance. + + Returns: + A `LogicalWorkerManager` instance. + """ + manager = LogicalWorkerManager(mock_backend) + manager.set_db_reference(mock_db) + return manager + + def test_create_worker(self, worker_manager: LogicalWorkerManager, mock_backend: MagicMock): + """ + Test creating a new logical worker. + + Args: + worker_manager: A `LogicalWorkerManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + worker_name = "test_worker" + queues = ["queue1", "queue2"] + worker_id = LogicalWorkerModel.generate_id(worker_name, queues) + + # Mock the backend get call to simulate worker doesn't exist + mock_backend.retrieve.return_value = None + + # Execute + worker = worker_manager.create(worker_name, queues) + + # Assert + assert isinstance(worker, LogicalWorkerEntity) + mock_backend.retrieve.assert_called_once_with(worker_id, "logical_worker") + mock_backend.save.assert_called_once() + saved_model = mock_backend.save.call_args[0][0] + assert saved_model.id == worker_id + assert saved_model.name == worker_name + assert saved_model.queues == queues + + def test_create_existing_worker(self, worker_manager: LogicalWorkerManager, mock_backend: MagicMock): + """ + Test creating a worker that already exists returns the existing worker. + + Args: + worker_manager: A `LogicalWorkerManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + worker_name = "existing_worker" + queues = ["queue1"] + worker_id = LogicalWorkerModel.generate_id(worker_name, queues) + + existing_model = LogicalWorkerModel(name=worker_name, queues=queues) + mock_backend.retrieve.return_value = existing_model + + # Execute + worker = worker_manager.create(worker_name, queues) + + # Assert + assert isinstance(worker, LogicalWorkerEntity) + mock_backend.retrieve.assert_called_once_with(worker_id, "logical_worker") + mock_backend.save.assert_not_called() + + def test_get_worker_by_id(self, worker_manager: LogicalWorkerManager, mock_backend: MagicMock): + """ + Test retrieving a worker by ID. + + Args: + worker_manager: A `LogicalWorkerManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + worker_id = "worker_id_123" + mock_model = LogicalWorkerModel(id=worker_id, name="test", queues=["q1"]) + mock_backend.retrieve.return_value = mock_model + + # Execute + worker = worker_manager.get(worker_id=worker_id) + + # Assert + assert isinstance(worker, LogicalWorkerEntity) + mock_backend.retrieve.assert_called_once_with(worker_id, "logical_worker") + + def test_get_worker_by_name_queues(self, worker_manager: LogicalWorkerManager, mock_backend: MagicMock): + """ + Test retrieving a worker by name and queues. + + Args: + worker_manager: A `LogicalWorkerManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + worker_name = "test_worker" + queues = ["queue1", "queue2"] + worker_id = LogicalWorkerModel.generate_id(worker_name, queues) + + mock_model = LogicalWorkerModel(id=worker_id, name=worker_name, queues=queues) + mock_backend.retrieve.return_value = mock_model + + # Execute + worker = worker_manager.get(worker_name=worker_name, queues=queues) + + # Assert + assert isinstance(worker, LogicalWorkerEntity) + mock_backend.retrieve.assert_called_once_with(worker_id, "logical_worker") + + def test_get_worker_invalid_args(self, worker_manager: LogicalWorkerManager): + """ + Test that `get` raises ValueError with invalid arguments. + + Args: + worker_manager: A `LogicalWorkerManager` instance. + """ + # Missing both worker_id and (worker_name, queues) + with pytest.raises(ValueError): + worker_manager.get() + + # Missing queues when worker_name is provided + with pytest.raises(ValueError): + worker_manager.get(worker_name="test") + + # Missing worker_name when queues is provided + with pytest.raises(ValueError): + worker_manager.get(queues=["q1"]) + + # Providing both worker_id and (worker_name, queues) + with pytest.raises(ValueError): + worker_manager.get(worker_id="id", worker_name="test", queues=["q1"]) + + def test_get_all_workers(self, worker_manager: LogicalWorkerManager, mock_backend: MagicMock): + """ + Test retrieving all workers. + + Args: + worker_manager: A `LogicalWorkerManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + mock_models = [LogicalWorkerModel(name="worker1", queues=["q1"]), LogicalWorkerModel(name="worker2", queues=["q2"])] + mock_backend.retrieve_all.return_value = mock_models + + # Execute + workers = worker_manager.get_all() + + # Assert + assert len(workers) == 2 + assert all(isinstance(w, LogicalWorkerEntity) for w in workers) + mock_backend.retrieve_all.assert_called_once_with("logical_worker") + + def test_delete_worker(self, worker_manager: LogicalWorkerManager, mock_backend: MagicMock, mock_db: MagicMock): + """ + Test deleting a worker and cleanup of associated runs. + + Args: + worker_manager: A `LogicalWorkerManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + mock_db: A mocked `MerlinDatabase` instance. + """ + # Setup worker entity + worker_id = "worker_id_123" + mock_worker = MagicMock(spec=LogicalWorkerEntity) + mock_worker.get_id.return_value = worker_id + mock_worker.get_runs.return_value = ["run1", "run2"] + + # Mock the get methods to return our mock worker + worker_manager.get = MagicMock(return_value=mock_worker) + worker_manager._get_entity = MagicMock(return_value=mock_worker) + + # Setup mock runs + mock_run1 = MagicMock() + mock_db.runs.get.side_effect = [mock_run1, RunNotFoundError] + + with patch.object(LogicalWorkerEntity, "delete") as mock_delete: + # Execute + worker_manager.delete(worker_id=worker_id) + + # Assert + worker_manager.get.assert_called_once_with(worker_id=worker_id, worker_name=None, queues=None) + worker_manager._get_entity.assert_called_once_with(LogicalWorkerEntity, worker_id) + mock_db.runs.get.assert_has_calls([call("run1"), call("run2")]) + mock_run1.remove_worker.assert_called_once_with(worker_id) + mock_delete.assert_called_once_with(worker_id, mock_backend) + + def test_delete_all_workers(self, worker_manager: LogicalWorkerManager): + """ + Test deleting all workers. + + Args: + worker_manager: A `LogicalWorkerManager` instance. + """ + # Setup + mock_workers = [MagicMock(spec=LogicalWorkerEntity), MagicMock(spec=LogicalWorkerEntity)] + mock_workers[0].get_id.return_value = "worker1" + mock_workers[1].get_id.return_value = "worker2" + + worker_manager.get_all = MagicMock(return_value=mock_workers) + worker_manager.delete = MagicMock() + + # Execute + worker_manager.delete_all() + + # Assert + worker_manager.get_all.assert_called_once() + worker_manager.delete.assert_has_calls([call("worker1"), call("worker2")]) diff --git a/tests/unit/db_scripts/entity_managers/test_physical_worker_manager.py b/tests/unit/db_scripts/entity_managers/test_physical_worker_manager.py new file mode 100644 index 000000000..8dfca4c8f --- /dev/null +++ b/tests/unit/db_scripts/entity_managers/test_physical_worker_manager.py @@ -0,0 +1,259 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `physical_worker_manager.py` module. +""" + +from unittest.mock import MagicMock, call, patch + +import pytest + +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.data_models import PhysicalWorkerModel +from merlin.db_scripts.entities.physical_worker_entity import PhysicalWorkerEntity +from merlin.db_scripts.entity_managers.physical_worker_manager import PhysicalWorkerManager +from merlin.db_scripts.merlin_db import MerlinDatabase +from merlin.exceptions import WorkerNotFoundError + + +class TestPhysicalWorkerManager: + """Tests for the `PhysicalWorkerManager` class.""" + + @pytest.fixture + def mock_backend(self) -> MagicMock: + """ + Create a mock `ResultsBackend` for testing. + + Returns: + A mocked `ResultsBackend` instance. + """ + return MagicMock(spec=ResultsBackend) + + @pytest.fixture + def mock_db(self) -> MagicMock: + """ + Create a mock `MerlinDatabase` for testing. + + Returns: + A mocked `MerlinDatabase` instance. + """ + db = MagicMock(spec=MerlinDatabase) + db.logical_workers = MagicMock() + return db + + @pytest.fixture + def worker_manager(self, mock_backend: MagicMock, mock_db: MagicMock) -> PhysicalWorkerManager: + """ + Create a `PhysicalWorkerManager` instance for testing. + + Args: + mock_backend: A mocked `ResultsBackend` instance. + mock_db: A mocked `MerlinDatabase` instance. + + Returns: + A `PhysicalWorkerManager` instance. + """ + manager = PhysicalWorkerManager(mock_backend) + manager.set_db_reference(mock_db) + return manager + + def test_create_worker(self, worker_manager: PhysicalWorkerManager, mock_backend: MagicMock): + """ + Test creating a new physical worker. + + Args: + worker_manager: A `PhysicalWorkerManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + worker_name = "test_worker" + + # Mock the backend get call to simulate worker doesn't exist + mock_backend.retrieve.return_value = None + + # Execute + worker = worker_manager.create(worker_name) + + # Assert + assert isinstance(worker, PhysicalWorkerEntity) + mock_backend.retrieve.assert_called_once_with(worker_name, "physical_worker") + mock_backend.save.assert_called_once() + saved_model = mock_backend.save.call_args[0][0] + assert saved_model.name == worker_name + + def test_create_worker_with_additional_attrs(self, worker_manager: PhysicalWorkerManager, mock_backend: MagicMock): + """ + Test creating a worker with additional attributes. + + Args: + worker_manager: A `PhysicalWorkerManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + worker_name = "test_worker" + attrs = {"host": "localhost", "pid": 1234} + + # Mock the backend get call to simulate worker doesn't exist + mock_backend.retrieve.return_value = None + + # Execute + worker_manager.create(worker_name, **attrs) + + # Assert + saved_model = mock_backend.save.call_args[0][0] + assert saved_model.name == worker_name + assert saved_model.host == "localhost" + assert saved_model.pid == 1234 + + def test_create_existing_worker(self, worker_manager: PhysicalWorkerManager, mock_backend: MagicMock): + """ + Test creating a worker that already exists returns the existing worker. + + Args: + worker_manager: A `PhysicalWorkerManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + worker_name = "existing_worker" + + existing_model = PhysicalWorkerModel(name=worker_name) + mock_backend.retrieve.return_value = existing_model + + # Execute + worker = worker_manager.create(worker_name) + + # Assert + assert isinstance(worker, PhysicalWorkerEntity) + mock_backend.retrieve.assert_called_once_with(worker_name, "physical_worker") + mock_backend.save.assert_not_called() + + def test_get_worker(self, worker_manager: PhysicalWorkerManager, mock_backend: MagicMock): + """ + Test retrieving a worker by ID or name. + + Args: + worker_manager: A `PhysicalWorkerManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + worker_id = "worker_name" # In this case, ID is the same as name + mock_model = PhysicalWorkerModel(name=worker_id) + mock_backend.retrieve.return_value = mock_model + + # Execute + worker = worker_manager.get(worker_id) + + # Assert + assert isinstance(worker, PhysicalWorkerEntity) + mock_backend.retrieve.assert_called_once_with(worker_id, "physical_worker") + + def test_get_all_workers(self, worker_manager: PhysicalWorkerManager, mock_backend: MagicMock): + """ + Test retrieving all workers. + + Args: + worker_manager: A `PhysicalWorkerManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + mock_models = [PhysicalWorkerModel(name="worker1"), PhysicalWorkerModel(name="worker2")] + mock_backend.retrieve_all.return_value = mock_models + + # Execute + workers = worker_manager.get_all() + + # Assert + assert len(workers) == 2 + assert all(isinstance(w, PhysicalWorkerEntity) for w in workers) + mock_backend.retrieve_all.assert_called_once_with("physical_worker") + + def test_delete_worker(self, worker_manager: PhysicalWorkerManager, mock_backend: MagicMock, mock_db: MagicMock): + """ + Test deleting a worker and cleanup of associated logical worker. + + Args: + worker_manager: A `PhysicalWorkerManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + mock_db: A mocked `MerlinDatabase` instance. + """ + # Setup + worker_id = "worker_name" + + mock_worker = MagicMock(spec=PhysicalWorkerEntity) + mock_worker.get_id.return_value = worker_id + mock_worker.get_logical_worker_id.return_value = "logical_worker_id" + + # Mock the get method to return our mock worker + worker_manager._get_entity = MagicMock(return_value=mock_worker) + + # Setup mock logical worker + mock_logical_worker = MagicMock() + mock_db.logical_workers.get.return_value = mock_logical_worker + + with patch.object(PhysicalWorkerEntity, "delete") as mock_delete: + # Execute + worker_manager.delete(worker_id) + + # Assert + worker_manager._get_entity.assert_called_once_with(PhysicalWorkerEntity, worker_id) + mock_db.logical_workers.get.assert_called_once_with(worker_id="logical_worker_id") + mock_logical_worker.remove_physical_worker.assert_called_once_with(worker_id) + mock_delete.assert_called_once_with(worker_id, mock_backend) + + def test_delete_worker_with_missing_logical_worker( + self, worker_manager: PhysicalWorkerManager, mock_backend: MagicMock, mock_db: MagicMock + ): + """ + Test deleting a worker when the associated logical worker isn't found. + + Args: + worker_manager: A `PhysicalWorkerManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + mock_db: A mocked `MerlinDatabase` instance. + """ + # Setup + worker_id = "worker_name" + + mock_worker = MagicMock(spec=PhysicalWorkerEntity) + mock_worker.get_id.return_value = worker_id + mock_worker.get_logical_worker_id.return_value = "missing_logical_worker" + + # Mock the get method to return our mock worker + worker_manager._get_entity = MagicMock(return_value=mock_worker) + + # Setup logical worker not found + mock_db.logical_workers.get.side_effect = WorkerNotFoundError + + with patch.object(PhysicalWorkerEntity, "delete") as mock_delete: + # Execute + worker_manager.delete(worker_id) + + # Assert + mock_db.logical_workers.get.assert_called_once_with(worker_id="missing_logical_worker") + mock_delete.assert_called_once_with(worker_id, mock_backend) + + def test_delete_all_workers(self, worker_manager: PhysicalWorkerManager): + """ + Test deleting all workers. + + Args: + worker_manager: A `PhysicalWorkerManager` instance. + """ + # Setup + mock_workers = [MagicMock(spec=PhysicalWorkerEntity), MagicMock(spec=PhysicalWorkerEntity)] + mock_workers[0].get_id.return_value = "worker1" + mock_workers[1].get_id.return_value = "worker2" + + worker_manager.get_all = MagicMock(return_value=mock_workers) + worker_manager.delete = MagicMock() + + # Execute + worker_manager.delete_all() + + # Assert + worker_manager.get_all.assert_called_once() + worker_manager.delete.assert_has_calls([call("worker1"), call("worker2")]) diff --git a/tests/unit/db_scripts/entity_managers/test_run_manager.py b/tests/unit/db_scripts/entity_managers/test_run_manager.py new file mode 100644 index 000000000..5b2ad2fc4 --- /dev/null +++ b/tests/unit/db_scripts/entity_managers/test_run_manager.py @@ -0,0 +1,271 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `run_manager.py` module. +""" + +from unittest.mock import MagicMock, call, patch + +import pytest + +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.data_models import RunModel +from merlin.db_scripts.entities.run_entity import RunEntity +from merlin.db_scripts.entities.study_entity import StudyEntity +from merlin.db_scripts.entity_managers.run_manager import RunManager +from merlin.db_scripts.merlin_db import MerlinDatabase +from merlin.exceptions import StudyNotFoundError, WorkerNotFoundError + + +class TestRunManager: + """Tests for the `RunManager` class.""" + + @pytest.fixture + def mock_backend(self) -> MagicMock: + """ + Create a mock `ResultsBackend` for testing. + + Returns: + A mocked `ResultsBackend` instance. + """ + return MagicMock(spec=ResultsBackend) + + @pytest.fixture + def mock_db(self) -> MagicMock: + """ + Create a mock `MerlinDatabase` for testing. + + Returns: + A mocked `MerlinDatabase` instance. + """ + db = MagicMock(spec=MerlinDatabase) + db.studies = MagicMock() + db.logical_workers = MagicMock() + return db + + @pytest.fixture + def run_manager(self, mock_backend: MagicMock, mock_db: MagicMock) -> RunManager: + """ + Create a `RunManager` instance for testing. + + Args: + mock_backend: A mocked `ResultsBackend` instance. + mock_db: A mocked `MerlinDatabase` instance. + + Returns: + A `RunManager` instance. + """ + manager = RunManager(mock_backend) + manager.set_db_reference(mock_db) + return manager + + def test_create_run(self, run_manager: RunManager, mock_backend: MagicMock, mock_db: MagicMock): + """ + Test creating a new run. + + Args: + run_manager: A `RunManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + mock_db: A mocked `MerlinDatabase` instance. + """ + # Setup + study_name = "test_study" + workspace = "test_workspace" + queues = ["queue1", "queue2"] + + # Mock study creation + mock_study = MagicMock(spec=StudyEntity) + mock_study.get_id.return_value = "study_id_123" + mock_db.studies.create.return_value = mock_study + + # Execute + run = run_manager.create(study_name, workspace, queues) + + # Assert + assert isinstance(run, RunEntity) + mock_db.studies.create.assert_called_once_with(study_name) + mock_backend.save.assert_called_once() + saved_model = mock_backend.save.call_args[0][0] + assert saved_model.study_id == "study_id_123" + assert saved_model.workspace == workspace + assert saved_model.queues == queues + mock_study.add_run.assert_called_once() + + def test_create_run_with_additional_args(self, run_manager: RunManager, mock_backend: MagicMock, mock_db: MagicMock): + """ + Test creating a run with additional valid and invalid arguments. + + Args: + run_manager: A `RunManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + mock_db: A mocked `MerlinDatabase` instance. + """ + # Setup + study_name = "test_study" + workspace = "test_workspace" + queues = ["queue1"] + + # Valid RunModel field and additional data + valid_field = "run_complete" + valid_value = False + invalid_field = "non_model_field" + invalid_value = "extra_data" + + # Mock study creation + mock_study = MagicMock(spec=StudyEntity) + mock_study.get_id.return_value = "study_id_123" + mock_db.studies.create.return_value = mock_study + + # Execute + run_manager.create(study_name, workspace, queues, **{valid_field: valid_value, invalid_field: invalid_value}) + + # Assert + saved_model = mock_backend.save.call_args[0][0] + assert saved_model.run_complete == valid_value + assert saved_model.additional_data == {invalid_field: invalid_value} + + def test_get_run(self, run_manager: RunManager, mock_backend: MagicMock): + """ + Test retrieving a run by ID or workspace. + + Args: + run_manager: A `RunManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + run_id = "run_id_123" + mock_model = RunModel(id=run_id, study_id="study_1", workspace="test_workspace", queues=["q1"]) + mock_backend.retrieve.return_value = mock_model + + # Execute + run = run_manager.get(run_id) + + # Assert + assert isinstance(run, RunEntity) + mock_backend.retrieve.assert_called_once_with(run_id, "run") + + def test_get_all_runs(self, run_manager: RunManager, mock_backend: MagicMock): + """ + Test retrieving all runs. + + Args: + run_manager: A `RunManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + mock_models = [ + RunModel(study_id="study_1", workspace="workspace1", queues=["q1"]), + RunModel(study_id="study_2", workspace="workspace2", queues=["q2"]), + ] + mock_backend.retrieve_all.return_value = mock_models + + # Execute + runs = run_manager.get_all() + + # Assert + assert len(runs) == 2 + assert all(isinstance(r, RunEntity) for r in runs) + mock_backend.retrieve_all.assert_called_once_with("run") + + def test_delete_run(self, run_manager: RunManager, mock_backend: MagicMock, mock_db: MagicMock): + """ + Test deleting a run and cleanup of associated entities. + + Args: + run_manager: A `RunManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + mock_db: A mocked `MerlinDatabase` instance. + """ + # Setup + run_id = "run_id_123" + + mock_run = MagicMock(spec=RunEntity) + mock_run.get_id.return_value = run_id + mock_run.get_study_id.return_value = "study_id_123" + mock_run.get_workers.return_value = ["worker1", "worker2"] + + # Mock the get method to return our mock run + run_manager._get_entity = MagicMock(return_value=mock_run) + + # Setup mock study and workers + mock_study = MagicMock() + mock_worker1 = MagicMock() + + # Mock study get success, worker1 get success, worker2 not found + mock_db.studies.get.return_value = mock_study + mock_db.logical_workers.get.side_effect = [mock_worker1, WorkerNotFoundError] + + with patch.object(RunEntity, "delete") as mock_delete: + # Execute + run_manager.delete(run_id) + + # Assert + run_manager._get_entity.assert_called_once_with(RunEntity, run_id) + + # Study cleanup + mock_db.studies.get.assert_called_once_with("study_id_123") + mock_study.remove_run.assert_called_once_with(run_id) + + # Worker cleanup + mock_db.logical_workers.get.assert_has_calls([call(worker_id="worker1"), call(worker_id="worker2")]) + mock_worker1.remove_run.assert_called_once_with(run_id) + + # Run deletion + mock_delete.assert_called_once_with(run_id, mock_backend) + + def test_delete_run_with_missing_study(self, run_manager: RunManager, mock_backend: MagicMock, mock_db: MagicMock): + """ + Test deleting a run when the associated study isn't found. + + Args: + run_manager: A `RunManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + mock_db: A mocked `MerlinDatabase` instance. + """ + # Setup + run_id = "run_id_123" + + mock_run = MagicMock(spec=RunEntity) + mock_run.get_id.return_value = run_id + mock_run.get_study_id.return_value = "missing_study" + mock_run.get_workers.return_value = [] + + # Mock the get method to return our mock run + run_manager._get_entity = MagicMock(return_value=mock_run) + + # Setup study not found + mock_db.studies.get.side_effect = StudyNotFoundError + + with patch.object(RunEntity, "delete") as mock_delete: + # Execute + run_manager.delete(run_id) + + # Assert + mock_db.studies.get.assert_called_once_with("missing_study") + mock_delete.assert_called_once_with(run_id, mock_backend) + + def test_delete_all_runs(self, run_manager: RunManager): + """ + Test deleting all runs. + + Args: + run_manager: A `RunManager` instance. + """ + # Setup + mock_runs = [MagicMock(spec=RunEntity), MagicMock(spec=RunEntity)] + mock_runs[0].get_id.return_value = "run1" + mock_runs[1].get_id.return_value = "run2" + + run_manager.get_all = MagicMock(return_value=mock_runs) + run_manager.delete = MagicMock() + + # Execute + run_manager.delete_all() + + # Assert + run_manager.get_all.assert_called_once() + run_manager.delete.assert_has_calls([call("run1"), call("run2")]) diff --git a/tests/unit/db_scripts/entity_managers/test_study_manager.py b/tests/unit/db_scripts/entity_managers/test_study_manager.py new file mode 100644 index 000000000..4712837d8 --- /dev/null +++ b/tests/unit/db_scripts/entity_managers/test_study_manager.py @@ -0,0 +1,226 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `study_manager.py` module. +""" + +from unittest.mock import MagicMock, call, patch + +import pytest + +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.data_models import StudyModel +from merlin.db_scripts.entities.study_entity import StudyEntity +from merlin.db_scripts.entity_managers.study_manager import StudyManager +from merlin.db_scripts.merlin_db import MerlinDatabase + + +class TestStudyManager: + """Tests for the `StudyManager` class.""" + + @pytest.fixture + def mock_backend(self) -> MagicMock: + """ + Create a mock `ResultsBackend` for testing. + + Returns: + A mocked `ResultsBackend` instance. + """ + return MagicMock(spec=ResultsBackend) + + @pytest.fixture + def mock_db(self) -> MagicMock: + """ + Create a mock `MerlinDatabase` for testing. + + Returns: + A mocked `MerlinDatabase` instance. + """ + db = MagicMock(spec=MerlinDatabase) + db.runs = MagicMock() + return db + + @pytest.fixture + def study_manager(self, mock_backend: MagicMock, mock_db: MagicMock) -> StudyManager: + """ + Create a `StudyManager` instance for testing. + + Args: + mock_backend: A mocked `ResultsBackend` instance. + mock_db: A mocked `MerlinDatabase` instance. + + Returns: + A `StudyManager` instance. + """ + manager = StudyManager(mock_backend) + manager.set_db_reference(mock_db) + return manager + + def test_create_study(self, study_manager: StudyManager, mock_backend: MagicMock): + """ + Test creating a new study. + + Args: + study_manager: A `StudyManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + study_name = "test_study" + + # Mock the backend get call to simulate study doesn't exist + mock_backend.retrieve.return_value = None + + # Execute + study = study_manager.create(study_name) + + # Assert + assert isinstance(study, StudyEntity) + mock_backend.retrieve.assert_called_once_with(study_name, "study") + mock_backend.save.assert_called_once() + saved_model = mock_backend.save.call_args[0][0] + assert saved_model.name == study_name + + def test_create_existing_study(self, study_manager: StudyManager, mock_backend: MagicMock): + """ + Test creating a study that already exists returns the existing study. + + Args: + study_manager: A `StudyManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + study_name = "existing_study" + + existing_model = StudyModel(name=study_name) + mock_backend.retrieve.return_value = existing_model + + # Execute + study = study_manager.create(study_name) + + # Assert + assert isinstance(study, StudyEntity) + mock_backend.retrieve.assert_called_once_with(study_name, "study") + mock_backend.save.assert_not_called() + + def test_get_study(self, study_manager: StudyManager, mock_backend: MagicMock): + """ + Test retrieving a study by ID or name. + + Args: + study_manager: A `StudyManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + study_id = "study_name" # In this case, ID is the same as name + mock_model = StudyModel(name=study_id) + mock_backend.retrieve.return_value = mock_model + + # Execute + study = study_manager.get(study_id) + + # Assert + assert isinstance(study, StudyEntity) + mock_backend.retrieve.assert_called_once_with(study_id, "study") + + def test_get_all_studies(self, study_manager: StudyManager, mock_backend: MagicMock): + """ + Test retrieving all studies. + + Args: + study_manager: A `StudyManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + """ + # Setup + mock_models = [StudyModel(name="study1"), StudyModel(name="study2")] + mock_backend.retrieve_all.return_value = mock_models + + # Execute + studies = study_manager.get_all() + + # Assert + assert len(studies) == 2 + assert all(isinstance(s, StudyEntity) for s in studies) + mock_backend.retrieve_all.assert_called_once_with("study") + + def test_delete_study_with_runs(self, study_manager: StudyManager, mock_backend: MagicMock, mock_db: MagicMock): + """ + Test deleting a study with associated runs. + + Args: + study_manager: A `StudyManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + mock_db: A mocked `MerlinDatabase` instance. + """ + # Setup + study_id = "study_name" + + mock_study = MagicMock(spec=StudyEntity) + mock_study.get_id.return_value = study_id + mock_study.get_runs.return_value = ["run1", "run2"] + + # Mock the get method to return our mock study + study_manager._get_entity = MagicMock(return_value=mock_study) + + with patch.object(StudyEntity, "delete") as mock_delete: + # Execute + study_manager.delete(study_id, remove_associated_runs=True) + + # Assert + study_manager._get_entity.assert_called_once_with(StudyEntity, study_id) + mock_db.runs.delete.assert_has_calls([call("run1"), call("run2")]) + mock_delete.assert_called_once_with(study_id, mock_backend) + + def test_delete_study_without_runs(self, study_manager: StudyManager, mock_backend: MagicMock, mock_db: MagicMock): + """ + Test deleting a study without removing associated runs. + + Args: + study_manager: A `StudyManager` instance. + mock_backend: A mocked `ResultsBackend` instance. + mock_db: A mocked `MerlinDatabase` instance. + """ + # Setup + study_id = "study_name" + + mock_study = MagicMock(spec=StudyEntity) + mock_study.get_id.return_value = study_id + mock_study.get_runs.return_value = ["run1", "run2"] + + # Mock the get method to return our mock study + study_manager._get_entity = MagicMock(return_value=mock_study) + + with patch.object(StudyEntity, "delete") as mock_delete: + # Execute + study_manager.delete(study_id, remove_associated_runs=False) + + # Assert + mock_db.runs.delete.assert_not_called() + mock_delete.assert_called_once_with(study_id, mock_backend) + + def test_delete_all_studies(self, study_manager: StudyManager): + """ + Test deleting all studies. + + Args: + study_manager: A `StudyManager` instance. + """ + # Setup + mock_studies = [MagicMock(spec=StudyEntity), MagicMock(spec=StudyEntity)] + mock_studies[0].get_id.return_value = "study1" + mock_studies[1].get_id.return_value = "study2" + + study_manager.get_all = MagicMock(return_value=mock_studies) + study_manager.delete = MagicMock() + + # Execute + study_manager.delete_all(remove_associated_runs=True) + + # Assert + study_manager.get_all.assert_called_once() + study_manager.delete.assert_has_calls( + [call("study1", remove_associated_runs=True), call("study2", remove_associated_runs=True)] + ) diff --git a/tests/unit/db_scripts/test_data_models.py b/tests/unit/db_scripts/test_data_models.py new file mode 100644 index 000000000..c92593d22 --- /dev/null +++ b/tests/unit/db_scripts/test_data_models.py @@ -0,0 +1,589 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `data_models.py` module. +""" + +import json +import os +from dataclasses import dataclass +from datetime import datetime +from unittest.mock import patch + +import pytest +from _pytest.capture import CaptureFixture +from pytest_mock import MockerFixture + +from merlin.common.enums import WorkerStatus +from merlin.db_scripts.data_models import BaseDataModel, LogicalWorkerModel, PhysicalWorkerModel, RunModel, StudyModel +from tests.fixture_types import FixtureCallable, FixtureStr + + +# Create a concrete subclass of BaseDataModel for testing +@pytest.fixture +def concrete_base_model_class() -> BaseDataModel: + """ + Creates a concrete subclass of `BaseDataModel` for testing. + + Returns: + A concrete subclass of `BaseDataModel`. + """ + + @dataclass + class ConcreteBaseModel(BaseDataModel): + field1: str = "default" + field2: int = 0 + + @property + def fields_allowed_to_be_updated(self): + return ["field1"] + + return ConcreteBaseModel + + +@pytest.fixture(scope="session") +def db_scripts_testing_dir(create_testing_dir: FixtureCallable, temp_output_dir: FixtureStr) -> FixtureStr: + """ + Fixture to create a temporary output directory for tests of database scripts. + + Args: + create_testing_dir: A fixture which returns a function that creates the testing directory. + temp_output_dir: The path to the temporary ouptut directory we'll be using for this test run. + + Returns: + The path to the temporary testing directory for database script tests. + """ + return create_testing_dir(temp_output_dir, "db_scripts_testing") + + +class TestBaseDataModel: + """Tests for the BaseDataModel abstract base class.""" + + def test_to_dict(self, concrete_base_model_class: BaseDataModel): + """ + Test conversion to dictionary. + + Args: + concrete_base_model_class: A concrete subclass of `BaseDataModel`. + """ + model = concrete_base_model_class(field1="test", field2=42) + model_dict = model.to_dict() + + assert model_dict["field1"] == "test" + assert model_dict["field2"] == 42 + assert "additional_data" in model_dict + + def test_to_json(self, concrete_base_model_class: BaseDataModel): + """ + Test conversion to JSON. + + Args: + concrete_base_model_class: A concrete subclass of `BaseDataModel`. + """ + model = concrete_base_model_class(field1="test", field2=42) + json_str = model.to_json() + + # Parse back to verify the structure + parsed = json.loads(json_str) + assert parsed["field1"] == "test" + assert parsed["field2"] == 42 + + def test_from_dict(self, concrete_base_model_class: BaseDataModel): + """ + Test creation from dictionary. + + Args: + concrete_base_model_class: A concrete subclass of `BaseDataModel`. + """ + data = {"field1": "from_dict", "field2": 99, "additional_data": {"extra": "data"}} + model = concrete_base_model_class.from_dict(data) + + assert model.field1 == "from_dict" + assert model.field2 == 99 + assert model.additional_data == {"extra": "data"} + + def test_from_json(self, concrete_base_model_class: BaseDataModel): + """ + Test creation from JSON string. + + Args: + concrete_base_model_class: A concrete subclass of `BaseDataModel`. + """ + json_str = '{"field1": "from_json", "field2": 77, "additional_data": {"json": "data"}}' + model = concrete_base_model_class.from_json(json_str) + + assert model.field1 == "from_json" + assert model.field2 == 77 + assert model.additional_data == {"json": "data"} + + def test_dump_to_json_file( + self, mocker: MockerFixture, concrete_base_model_class: BaseDataModel, db_scripts_testing_dir: FixtureStr + ): + """ + Test writing to a JSON file. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + concrete_base_model_class: A concrete subclass of `BaseDataModel`. + db_scripts_testing_dir: The path to the temporary testing directory for database script tests. + """ + model = concrete_base_model_class(field1="file_test", field2=123) + filepath = os.path.join(db_scripts_testing_dir, "base_data_model_test_dump_to_json.json") + + # Mock the FileLock + with mocker.patch("merlin.db_scripts.data_models.FileLock"): + model.dump_to_json_file(filepath) + + # Verify the file exists and contains expected content + assert os.path.exists(filepath) + with open(filepath, "r") as f: + data = json.load(f) + assert data["field1"] == "file_test" + assert data["field2"] == 123 + + def test_dump_to_json_file_invalid_path(self, concrete_base_model_class: BaseDataModel): + """ + Test error handling for invalid filepath. + + Args: + concrete_base_model_class: A concrete subclass of `BaseDataModel`. + """ + model = concrete_base_model_class() + + with pytest.raises(ValueError): + model.dump_to_json_file("") + + def test_load_from_json_file(self, concrete_base_model_class: BaseDataModel, db_scripts_testing_dir: FixtureStr): + """ + Test loading from a JSON file. + + Args: + concrete_base_model_class: A concrete subclass of `BaseDataModel`. + db_scripts_testing_dir: The path to the temporary testing directory for database script tests. + """ + # Create a test file + test_data = {"field1": "loaded", "field2": 321, "additional_data": {}} + filepath = os.path.join(db_scripts_testing_dir, "base_data_model_test_load_from_json_file.json") + + with open(filepath, "w") as f: + json.dump(test_data, f) + + # Mock the FileLock + with patch("merlin.db_scripts.data_models.FileLock"): + model = concrete_base_model_class.load_from_json_file(filepath) + + assert model.field1 == "loaded" + assert model.field2 == 321 + + def test_load_from_json_file_not_exists(self, concrete_base_model_class: BaseDataModel): + """ + Test error handling when file does not exist. + + Args: + concrete_base_model_class: A concrete subclass of `BaseDataModel`. + """ + with pytest.raises(ValueError): + concrete_base_model_class.load_from_json_file("/nonexistent/file.json") + + def test_get_instance_fields(self, concrete_base_model_class: BaseDataModel): + """ + Test retrieving instance fields. + + Args: + concrete_base_model_class: A concrete subclass of `BaseDataModel`. + """ + model = concrete_base_model_class() + fields = model.get_instance_fields() + + field_names = [f.name for f in fields] + assert "field1" in field_names + assert "field2" in field_names + assert "additional_data" in field_names + + def test_get_class_fields(self, concrete_base_model_class: BaseDataModel): + """ + Test retrieving class fields. + + Args: + concrete_base_model_class: A concrete subclass of `BaseDataModel`. + """ + fields = concrete_base_model_class.get_class_fields() + + field_names = [f.name for f in fields] + assert "field1" in field_names + assert "field2" in field_names + assert "additional_data" in field_names + + def test_update_fields_allowed(self, concrete_base_model_class: BaseDataModel): + """ + Test updating allowed fields. + + Args: + concrete_base_model_class: A concrete subclass of `BaseDataModel`. + """ + model = concrete_base_model_class(field1="original", field2=100) + + model.update_fields({"field1": "updated"}) + assert model.field1 == "updated" + assert model.field2 == 100 + + def test_update_fields_not_allowed(self, caplog: CaptureFixture, concrete_base_model_class: BaseDataModel): + """ + Test attempting to update fields that are not allowed. + + Args: + caplog: A built-in fixture from the pytest library to capture logs. + concrete_base_model_class: A concrete subclass of `BaseDataModel`. + """ + model = concrete_base_model_class(field1="original", field2=100) + + # Patch the logger to check for warnings + model.update_fields({"field2": 999}) + + # Field should not be updated + assert model.field2 == 100 + + # Warning should be logged + assert "not allowed to be updated" in caplog.text + + def test_update_fields_nonexistent(self, caplog: CaptureFixture, concrete_base_model_class: BaseDataModel): + """ + Test updating fields that don't exist on the model. + + Args: + caplog: A built-in fixture from the pytest library to capture logs. + concrete_base_model_class: A concrete subclass of `BaseDataModel`. + """ + model = concrete_base_model_class() + model.update_fields({"nonexistent_field": "value"}) + + # Should be added to additional_data + assert model.additional_data["nonexistent_field"] == "value" + + # Warning should be logged + assert "does not explicitly exist" in caplog.text + + def test_update_fields_ignore_id(self, concrete_base_model_class: BaseDataModel): + """ + Test that ID field cannot be updated. + + Args: + concrete_base_model_class: A concrete subclass of `BaseDataModel`. + """ + model = concrete_base_model_class() + original_id = model.id if hasattr(model, "id") else None + + model.update_fields({"id": "new_id"}) + + # ID shouldn't be updated, even if it's in allowed fields + if hasattr(model, "id"): + assert model.id == original_id + + +class TestStudyModel: + """Tests for the StudyModel class.""" + + def test_default_initialization(self): + """Test default initialization of StudyModel.""" + study = StudyModel() + + assert isinstance(study.id, str) + assert study.name is None + assert study.runs == [] + assert study.additional_data == {} + + def test_custom_initialization(self): + """Test initialization with custom values.""" + study = StudyModel(id="custom-id", name="Test Study", runs=["run1", "run2"], additional_data={"key": "value"}) + + assert study.id == "custom-id" + assert study.name == "Test Study" + assert study.runs == ["run1", "run2"] + assert study.additional_data == {"key": "value"} + + def test_allowed_fields_update(self, caplog: CaptureFixture): + """ + Test updating allowed fields. + + Args: + caplog: A built-in fixture from the pytest library to capture logs. + """ + study = StudyModel(name="Original Study", runs=["run1"]) + + # Test allowed field + study.update_fields({"runs": ["run1", "run2"]}) + assert study.runs == ["run1", "run2"] + + # Test not allowed field + study.update_fields({"name": "Updated Study"}) + assert study.name == "Original Study" # Should be the same as before + assert "not allowed to be updated" in caplog.text # Warning should be logged + + def test_fields_allowed_to_be_updated(self): + """Test that fields_allowed_to_be_updated returns expected values.""" + study = StudyModel() + assert study.fields_allowed_to_be_updated == ["runs"] + + +class TestRunModel: + """Tests for the RunModel class.""" + + def test_default_initialization(self): + """Test default initialization of RunModel.""" + run = RunModel() + + assert isinstance(run.id, str) + assert run.study_id is None + assert run.workspace is None + assert run.steps == [] + assert run.queues == [] + assert run.workers == [] + assert run.parent is None + assert run.child is None + assert run.run_complete is False + assert run.parameters == {} + assert run.samples == {} + assert run.additional_data == {} + + def test_custom_initialization(self): + """Test initialization with custom values.""" + run = RunModel( + id="run-id", + study_id="study-id", + workspace="/path/to/workspace", + steps=["step1", "step2"], + queues=["queue1", "queue2"], + workers=["worker1"], + parent="parent-run", + child="child-run", + run_complete=True, + parameters={"param1": "value1"}, + samples={"sample1": "data1"}, + additional_data={"meta": "data"}, + ) + + assert run.id == "run-id" + assert run.study_id == "study-id" + assert run.workspace == "/path/to/workspace" + assert run.steps == ["step1", "step2"] + assert run.queues == ["queue1", "queue2"] + assert run.workers == ["worker1"] + assert run.parent == "parent-run" + assert run.child == "child-run" + assert run.run_complete is True + assert run.parameters == {"param1": "value1"} + assert run.samples == {"sample1": "data1"} + assert run.additional_data == {"meta": "data"} + + def test_allowed_fields_update(self, caplog: CaptureFixture): + """ + Test updating allowed fields. + + Args: + caplog: A built-in fixture from the pytest library to capture logs. + """ + run = RunModel(workers=["worker1"], study_id="study-id") + + # Test allowed field + run.update_fields({"workers": ["worker1", "worker2"]}) + assert run.workers == ["worker1", "worker2"] + + # Test not allowed field + run.update_fields({"study_id": "new-study-id"}) + assert run.study_id == "study-id" # Should be the same as before + assert "not allowed to be updated" in caplog.text # Warning should be logged + + def test_fields_allowed_to_be_updated(self): + """Test that fields_allowed_to_be_updated returns expected values.""" + run = RunModel() + assert set(run.fields_allowed_to_be_updated) == {"parent", "child", "run_complete", "additional_data", "workers"} + + +class TestLogicalWorkerModel: + """Tests for the LogicalWorkerModel class.""" + + def test_initialization_requires_name_and_queues(self): + """Test that initialization requires name and queues.""" + # Missing name + with pytest.raises(TypeError): + LogicalWorkerModel(queues={"queue1"}) + + # Missing queues + with pytest.raises(TypeError): + LogicalWorkerModel(name="worker1") + + # Empty queues + with pytest.raises(TypeError): + LogicalWorkerModel(name="worker1", queues={}) + + # Valid initialization + worker = LogicalWorkerModel(name="worker1", queues={"queue1"}) + assert worker.name == "worker1" + assert worker.queues == {"queue1"} + + def test_id_generation(self): + """Test that ID is generated consistently based on name and queues.""" + worker1 = LogicalWorkerModel(name="worker1", queues={"queue1", "queue2"}) + worker2 = LogicalWorkerModel(name="worker1", queues={"queue2", "queue1"}) + + # Should generate the same ID regardless of queue order + assert worker1.id == worker2.id + + # Different name should generate different ID + worker3 = LogicalWorkerModel(name="different", queues={"queue1", "queue2"}) + assert worker1.id != worker3.id + + # Different queue should generate different ID + worker4 = LogicalWorkerModel(name="worker1", queues={"queue1", "queue3"}) + assert worker1.id != worker4.id + + def test_id_overwrite_warning(self, caplog: CaptureFixture): + """ + Test warning when provided ID is overwritten. + + Args: + caplog: A built-in fixture from the pytest library to capture logs. + """ + worker = LogicalWorkerModel(name="worker1", queues={"queue1"}, id="provided-id") + + # ID should be overwritten + assert worker.id != "provided-id" + + # Warning should be logged + assert "will be overwritten" in caplog.text + + def test_allowed_fields_update(self, caplog: CaptureFixture): + """ + Test updating allowed fields. + + Args: + caplog: A built-in fixture from the pytest library to capture logs. + """ + worker = LogicalWorkerModel(name="worker1", queues={"queue1"}) + + # Test allowed field + worker.update_fields({"runs": ["run1", "run2"]}) + assert worker.runs == ["run1", "run2"] + + worker.update_fields({"physical_workers": ["pw1", "pw2"]}) + assert worker.physical_workers == ["pw1", "pw2"] + + # Test not allowed field + worker.update_fields({"name": "new-name"}) + + # Field should not be updated + assert worker.name == "worker1" + + # Warning should be logged + assert "not allowed to be updated" in caplog.text + + def test_generate_id_class_method(self): + """Test the class method for ID generation.""" + id1 = LogicalWorkerModel.generate_id("worker1", ["queue1", "queue2"]) + id2 = LogicalWorkerModel.generate_id("worker1", ["queue2", "queue1"]) + + # Same inputs should produce same ID + assert id1 == id2 + + # Different inputs should produce different IDs + id3 = LogicalWorkerModel.generate_id("worker2", ["queue1", "queue2"]) + assert id1 != id3 + + def test_fields_allowed_to_be_updated(self): + """Test that fields_allowed_to_be_updated returns expected values.""" + worker = LogicalWorkerModel(name="worker1", queues={"queue1"}) + assert worker.fields_allowed_to_be_updated == ["runs", "physical_workers"] + + +class TestPhysicalWorkerModel: + """Tests for the PhysicalWorkerModel class.""" + + def test_default_initialization(self): + """Test default initialization of PhysicalWorkerModel.""" + worker = PhysicalWorkerModel() + + assert isinstance(worker.id, str) + assert worker.logical_worker_id is None + assert worker.name is None + assert worker.launch_cmd is None + assert worker.args == {} + assert worker.pid is None + assert worker.status == WorkerStatus.STOPPED + assert isinstance(worker.heartbeat_timestamp, datetime) + assert isinstance(worker.latest_start_time, datetime) + assert worker.host is None + assert worker.restart_count == 0 + assert worker.additional_data == {} + + def test_custom_initialization(self): + """Test initialization with custom values.""" + current_time = datetime.now() + worker = PhysicalWorkerModel( + id="physical-id", + logical_worker_id="logical-id", + name="celery@worker1.host", + launch_cmd="celery worker", + args={"arg1": "value1"}, + pid="12345", + status=WorkerStatus.RUNNING, + heartbeat_timestamp=current_time, + latest_start_time=current_time, + host="hostname", + restart_count=3, + additional_data={"meta": "data"}, + ) + + assert worker.id == "physical-id" + assert worker.logical_worker_id == "logical-id" + assert worker.name == "celery@worker1.host" + assert worker.launch_cmd == "celery worker" + assert worker.args == {"arg1": "value1"} + assert worker.pid == "12345" + assert worker.status == WorkerStatus.RUNNING + assert worker.heartbeat_timestamp == current_time + assert worker.latest_start_time == current_time + assert worker.host == "hostname" + assert worker.restart_count == 3 + assert worker.additional_data == {"meta": "data"} + + def test_allowed_fields_update(self, caplog: CaptureFixture): + """ + Test updating allowed fields. + + Args: + caplog: A built-in fixture from the pytest library to capture logs. + """ + worker = PhysicalWorkerModel(pid="12345", name="celery@worker1.host") + + # Test allowed field + worker.update_fields({"pid": "67890"}) + assert worker.pid == "67890" + + worker.update_fields({"status": WorkerStatus.RUNNING}) + assert worker.status == WorkerStatus.RUNNING + + # Test not allowed field + worker.update_fields({"name": "celery@new.host"}) + + # Field should not be updated + assert worker.name == "celery@worker1.host" + + # Warning should be logged + assert "not allowed to be updated" in caplog.text + + def test_fields_allowed_to_be_updated(self): + """Test that fields_allowed_to_be_updated returns expected values.""" + worker = PhysicalWorkerModel() + allowed_fields = worker.fields_allowed_to_be_updated + + # Check each expected field is in the list + expected_fields = ["launch_cmd", "args", "pid", "status", "heartbeat_timestamp", "latest_start_time", "restart_count"] + for field in expected_fields: + assert field in allowed_fields + + # Check field count matches + assert len(allowed_fields) == len(expected_fields) diff --git a/tests/unit/db_scripts/test_db_commands.py b/tests/unit/db_scripts/test_db_commands.py new file mode 100644 index 000000000..9bb0e5ced --- /dev/null +++ b/tests/unit/db_scripts/test_db_commands.py @@ -0,0 +1,668 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `db_commands.py` module. +""" + +import logging +from argparse import Namespace +from unittest.mock import call + +from _pytest.capture import CaptureFixture +from pytest_mock import MockerFixture + +# Import the functions being tested +from merlin.db_scripts.db_commands import database_delete, database_get, database_info + + +class TestDatabaseInfo: + """Tests for the database_info function.""" + + def test_database_info(self, mocker: MockerFixture, capsys: CaptureFixture): + """ + Test that database_info prints the correct information. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + capsys: A built-in fixture from the pytest library to capture stdout and stderr. + """ + # Mock MerlinDatabase and its methods + mock_db = mocker.MagicMock() + mock_db.get_db_type.return_value = "SQLite" + mock_db.get_db_version.return_value = "1.0.0" + mock_db.get_connection_string.return_value = "sqlite:///merlin.db" + mock_db.get_all.side_effect = [ + ["study1", "study2"], # studies + ["run1", "run2", "run3"], # runs + ["worker1"], # logical workers + ["worker1", "worker2"], # physical workers + ] + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Call the function + database_info() + + # Capture the printed output + captured = capsys.readouterr() + + # Verify the output contains the expected information + assert "Merlin Database Information" in captured.out + assert "Database Type: SQLite" in captured.out + assert "Database Version: 1.0.0" in captured.out + assert "Connection String: sqlite:///merlin.db" in captured.out + assert "Studies:" in captured.out + assert "Total: 2" in captured.out + assert "Runs:" in captured.out + assert "Total: 3" in captured.out + assert "Logical Workers:" in captured.out + assert "Total: 1" in captured.out + assert "Physical Workers:" in captured.out + assert "Total: 2" in captured.out + + # Verify get_all was called with the correct entity types + mock_db.get_all.assert_has_calls([call("study"), call("run"), call("logical_worker"), call("physical_worker")]) + + +class TestDatabaseGet: + """Tests for the database_get function.""" + + def test_get_study(self, mocker: MockerFixture, capsys: CaptureFixture): + """ + Test getting studies by ID. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + capsys: A built-in fixture from the pytest library to capture stdout and stderr. + """ + # Mock MerlinDatabase and its get method + mock_db = mocker.MagicMock() + mock_study1 = mocker.MagicMock() + mock_study1.__str__.return_value = "Study 1" + mock_study2 = mocker.MagicMock() + mock_study2.__str__.return_value = "Study 2" + mock_db.get.side_effect = [mock_study1, mock_study2] + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args with study IDs + args = Namespace(get_type="study", study=["study1", "study2"]) + + # Call the function + database_get(args) + + # Capture the printed output + captured = capsys.readouterr() + + # Verify the output contains the expected information + assert "Study 1" in captured.out + assert "Study 2" in captured.out + + # Verify the get method was called with the correct arguments + mock_db.get.assert_has_calls([call("study", "study1"), call("study", "study2")]) + + def test_get_run(self, mocker: MockerFixture, capsys: CaptureFixture): + """ + Test getting runs by ID. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + capsys: A built-in fixture from the pytest library to capture stdout and stderr. + """ + # Mock MerlinDatabase and its get method + mock_db = mocker.MagicMock() + mock_run = mocker.MagicMock() + mock_run.__str__.return_value = "Run 1" + mock_db.get.return_value = mock_run + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args with run IDs + args = Namespace(get_type="run", run=["run1"]) + + # Call the function + database_get(args) + + # Capture the printed output + captured = capsys.readouterr() + + # Verify the output contains the expected information + assert "Run 1" in captured.out + + # Verify the get method was called with the correct arguments + mock_db.get.assert_called_once_with("run", "run1") + + def test_get_logical_worker(self, mocker: MockerFixture, capsys: CaptureFixture): + """ + Test getting logical workers by ID. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + capsys: A built-in fixture from the pytest library to capture stdout and stderr. + """ + # Mock MerlinDatabase and its get method + mock_db = mocker.MagicMock() + mock_worker = mocker.MagicMock() + mock_worker.__str__.return_value = "Logical Worker 1" + mock_db.get.return_value = mock_worker + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args with worker IDs + args = Namespace(get_type="logical-worker", worker=["worker1"]) + + # Call the function + database_get(args) + + # Capture the printed output + captured = capsys.readouterr() + + # Verify the output contains the expected information + assert "Logical Worker 1" in captured.out + + # Verify the get method was called with the correct arguments + mock_db.get.assert_called_once_with("logical_worker", "worker1") + + def test_get_physical_worker(self, mocker: MockerFixture, capsys: CaptureFixture): + """ + Test getting physical workers by ID. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + capsys: A built-in fixture from the pytest library to capture stdout and stderr. + """ + # Mock MerlinDatabase and its get method + mock_db = mocker.MagicMock() + mock_worker = mocker.MagicMock() + mock_worker.__str__.return_value = "Physical Worker 1" + mock_db.get.return_value = mock_worker + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args with worker IDs + args = Namespace(get_type="physical-worker", worker=["worker1"]) + + # Call the function + database_get(args) + + # Capture the printed output + captured = capsys.readouterr() + + # Verify the output contains the expected information + assert "Physical Worker 1" in captured.out + + # Verify the get method was called with the correct arguments + mock_db.get.assert_called_once_with("physical_worker", "worker1") + + def test_get_all_studies(self, mocker: MockerFixture, capsys: CaptureFixture): + """ + Test getting all studies. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + capsys: A built-in fixture from the pytest library to capture stdout and stderr. + """ + # Mock MerlinDatabase and its get_all method + mock_db = mocker.MagicMock() + mock_study1 = mocker.MagicMock() + mock_study1.__str__.return_value = "Study 1" + mock_study2 = mocker.MagicMock() + mock_study2.__str__.return_value = "Study 2" + mock_db.get_all.return_value = [mock_study1, mock_study2] + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args + args = Namespace(get_type="all-studies") + + # Call the function + database_get(args) + + # Capture the printed output + captured = capsys.readouterr() + + # Verify the output contains the expected information + assert "Study 1" in captured.out + assert "Study 2" in captured.out + + # Verify the get_all method was called with the correct entity type + mock_db.get_all.assert_called_once_with("study") + + def test_get_all_runs(self, mocker: MockerFixture, capsys: CaptureFixture): + """ + Test getting all runs. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + capsys: A built-in fixture from the pytest library to capture stdout and stderr. + """ + # Mock MerlinDatabase and its get_all method + mock_db = mocker.MagicMock() + mock_run1 = mocker.MagicMock() + mock_run1.__str__.return_value = "Run 1" + mock_run2 = mocker.MagicMock() + mock_run2.__str__.return_value = "Run 2" + mock_db.get_all.return_value = [mock_run1, mock_run2] + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args + args = Namespace(get_type="all-runs") + + # Call the function + database_get(args) + + # Capture the printed output + captured = capsys.readouterr() + + # Verify the output contains the expected information + assert "Run 1" in captured.out + assert "Run 2" in captured.out + + # Verify the get_all method was called with the correct entity type + mock_db.get_all.assert_called_once_with("run") + + def test_get_all_logical_workers(self, mocker: MockerFixture, capsys: CaptureFixture): + """ + Test getting all logical workers. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + capsys: A built-in fixture from the pytest library to capture stdout and stderr. + """ + # Mock MerlinDatabase and its get_all method + mock_db = mocker.MagicMock() + mock_worker = mocker.MagicMock() + mock_worker.__str__.return_value = "Logical Worker 1" + mock_db.get_all.return_value = [mock_worker] + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args + args = Namespace(get_type="all-logical-workers") + + # Call the function + database_get(args) + + # Capture the printed output + captured = capsys.readouterr() + + # Verify the output contains the expected information + assert "Logical Worker 1" in captured.out + + # Verify the get_all method was called with the correct entity type + mock_db.get_all.assert_called_once_with("logical_worker") + + def test_get_all_physical_workers(self, mocker: MockerFixture, capsys: CaptureFixture): + """ + Test getting all physical workers. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + capsys: A built-in fixture from the pytest library to capture stdout and stderr. + """ + # Mock MerlinDatabase and its get_all method + mock_db = mocker.MagicMock() + mock_worker = mocker.MagicMock() + mock_worker.__str__.return_value = "Physical Worker 1" + mock_db.get_all.return_value = [mock_worker] + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args + args = Namespace(get_type="all-physical-workers") + + # Call the function + database_get(args) + + # Capture the printed output + captured = capsys.readouterr() + + # Verify the output contains the expected information + assert "Physical Worker 1" in captured.out + + # Verify the get_all method was called with the correct entity type + mock_db.get_all.assert_called_once_with("physical_worker") + + def test_get_everything(self, mocker: MockerFixture, capsys: CaptureFixture): + """ + Test getting everything from the database. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + capsys: A built-in fixture from the pytest library to capture stdout and stderr. + """ + # Mock MerlinDatabase and its get_everything method + mock_db = mocker.MagicMock() + mock_entity = mocker.MagicMock() + mock_entity.__str__.return_value = "Database Entity" + mock_db.get_everything.return_value = [mock_entity] + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args + args = Namespace(get_type="everything") + + # Call the function + database_get(args) + + # Capture the printed output + captured = capsys.readouterr() + + # Verify the output contains the expected information + assert "Database Entity" in captured.out + + # Verify the get_everything method was called + mock_db.get_everything.assert_called_once() + + def test_get_empty_studies(self, mocker: MockerFixture, caplog: CaptureFixture): + """ + Test getting studies when none are found. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + caplog: A built-in fixture from the pytest library to capture logs. + """ + caplog.set_level(logging.INFO) + + # Mock MerlinDatabase and its get_all method + mock_db = mocker.MagicMock() + mock_db.get_all.return_value = [] + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args + args = Namespace(get_type="all-studies") + + # Call the function + database_get(args) + + # Verify LOG was called with the correct message + assert "No studies found in the database." in caplog.text + + def test_get_invalid_option(self, mocker: MockerFixture, caplog: CaptureFixture): + """ + Test providing an invalid get option. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + caplog: A built-in fixture from the pytest library to capture logs. + """ + # Mock MerlinDatabase + mock_db = mocker.MagicMock() + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args with invalid get_type + args = Namespace(get_type="invalid-option") + + # Call the function + database_get(args) + + # Verify LOG was called with the correct message + assert "No valid get option provided." in caplog.text + + +class TestDatabaseDelete: + """Tests for the database_delete function.""" + + def test_delete_study(self, mocker: MockerFixture): + """ + Test deleting studies by ID. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + """ + # Mock MerlinDatabase and its delete method + mock_db = mocker.MagicMock() + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args with study IDs + args = Namespace(delete_type="study", study=["study1", "study2"], keep_associated_runs=False) + + # Call the function + database_delete(args) + + # Verify the delete method was called with the correct arguments + mock_db.delete.assert_has_calls( + [call("study", "study1", remove_associated_runs=True), call("study", "study2", remove_associated_runs=True)] + ) + + def test_delete_study_keep_runs(self, mocker: MockerFixture): + """ + Test deleting studies by ID while keeping associated runs. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + """ + # Mock MerlinDatabase and its delete method + mock_db = mocker.MagicMock() + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args with study IDs and keep_associated_runs=True + args = Namespace(delete_type="study", study=["study1"], keep_associated_runs=True) + + # Call the function + database_delete(args) + + # Verify the delete method was called with the correct arguments + mock_db.delete.assert_called_once_with("study", "study1", remove_associated_runs=False) + + def test_delete_run(self, mocker: MockerFixture): + """ + Test deleting runs by ID. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + """ + # Mock MerlinDatabase and its delete method + mock_db = mocker.MagicMock() + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args with run IDs + args = Namespace(delete_type="run", run=["run1", "run2"]) + + # Call the function + database_delete(args) + + # Verify the delete method was called with the correct arguments + mock_db.delete.assert_has_calls([call("run", "run1"), call("run", "run2")]) + + def test_delete_logical_worker(self, mocker: MockerFixture): + """ + Test deleting logical workers by ID. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + """ + # Mock MerlinDatabase and its delete method + mock_db = mocker.MagicMock() + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args with worker IDs + args = Namespace(delete_type="logical-worker", worker=["worker1"]) + + # Call the function + database_delete(args) + + # Verify the delete method was called with the correct arguments + mock_db.delete.assert_called_once_with("logical_worker", "worker1") + + def test_delete_physical_worker(self, mocker: MockerFixture): + """ + Test deleting physical workers by ID. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + """ + # Mock MerlinDatabase and its delete method + mock_db = mocker.MagicMock() + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args with worker IDs + args = Namespace(delete_type="physical-worker", worker=["worker1"]) + + # Call the function + database_delete(args) + + # Verify the delete method was called with the correct arguments + mock_db.delete.assert_called_once_with("physical_worker", "worker1") + + def test_delete_all_studies(self, mocker: MockerFixture): + """ + Test deleting all studies. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + """ + # Mock MerlinDatabase and its delete_all method + mock_db = mocker.MagicMock() + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args + args = Namespace(delete_type="all-studies", keep_associated_runs=False) + + # Call the function + database_delete(args) + + # Verify the delete_all method was called with the correct arguments + mock_db.delete_all.assert_called_once_with("study", remove_associated_runs=True) + + def test_delete_all_runs(self, mocker: MockerFixture): + """ + Test deleting all runs. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + """ + # Mock MerlinDatabase and its delete_all method + mock_db = mocker.MagicMock() + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args + args = Namespace(delete_type="all-runs") + + # Call the function + database_delete(args) + + # Verify the delete_all method was called with the correct entity type + mock_db.delete_all.assert_called_once_with("run") + + def test_delete_all_logical_workers(self, mocker: MockerFixture): + """ + Test deleting all logical workers. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + """ + # Mock MerlinDatabase and its delete_all method + mock_db = mocker.MagicMock() + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args + args = Namespace(delete_type="all-logical-workers") + + # Call the function + database_delete(args) + + # Verify the delete_all method was called with the correct entity type + mock_db.delete_all.assert_called_once_with("logical_worker") + + def test_delete_all_physical_workers(self, mocker: MockerFixture): + """ + Test deleting all physical workers. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + """ + # Mock MerlinDatabase and its delete_all method + mock_db = mocker.MagicMock() + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args + args = Namespace(delete_type="all-physical-workers") + + # Call the function + database_delete(args) + + # Verify the delete_all method was called with the correct entity type + mock_db.delete_all.assert_called_once_with("physical_worker") + + def test_delete_everything(self, mocker: MockerFixture): + """ + Test deleting everything from the database. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + """ + # Mock MerlinDatabase and its delete_everything method + mock_db = mocker.MagicMock() + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args + args = Namespace(delete_type="everything", force=True) + + # Call the function + database_delete(args) + + # Verify the delete_everything method was called with the correct arguments + mock_db.delete_everything.assert_called_once_with(force=True) + + def test_delete_invalid_option(self, mocker: MockerFixture, caplog: CaptureFixture): + """ + Test providing an invalid delete option. + + Args: + mocker: A built-in fixture from the pytest-mock library to create a Mock object. + caplog: A built-in fixture from the pytest library to capture logs. + """ + # Mock MerlinDatabase + mock_db = mocker.MagicMock() + + # Patch the MerlinDatabase class + mocker.patch("merlin.db_scripts.db_commands.MerlinDatabase", return_value=mock_db) + + # Create args with invalid delete_type + args = Namespace(delete_type="invalid-option") + + # Call the function + database_delete(args) + + # Verify LOG.error was called with the correct message + assert "No valid delete option provided." in caplog.text diff --git a/tests/unit/db_scripts/test_merlin_db.py b/tests/unit/db_scripts/test_merlin_db.py new file mode 100644 index 000000000..c793db6f8 --- /dev/null +++ b/tests/unit/db_scripts/test_merlin_db.py @@ -0,0 +1,449 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `merlin_db.py` module. +""" + +import logging +from contextlib import ExitStack +from typing import Generator +from unittest.mock import MagicMock, patch + +import pytest +from _pytest.capture import CaptureFixture + +from merlin.backends.results_backend import ResultsBackend +from merlin.db_scripts.entity_managers.logical_worker_manager import LogicalWorkerManager +from merlin.db_scripts.entity_managers.physical_worker_manager import PhysicalWorkerManager +from merlin.db_scripts.entity_managers.run_manager import RunManager +from merlin.db_scripts.entity_managers.study_manager import StudyManager +from merlin.db_scripts.merlin_db import MerlinDatabase +from merlin.exceptions import EntityManagerNotSupportedError +from tests.fixture_types import FixtureDict + + +@pytest.fixture +def mock_backend() -> MagicMock: + """ + Create a mock backend instance. + + Returns: + A mocked `ResultsBackend` instance. + """ + mock = MagicMock(spec=ResultsBackend) + mock.get_name.return_value = "mock_redis" + mock.get_version.return_value = "1.0" + mock.get_connection_string.return_value = "redis://localhost:6379/0" + return mock + + +@pytest.fixture +def mock_entity_managers() -> FixtureDict[str, MagicMock]: + """ + Create mock entity managers. + + Returns: + A dictionary of mocked objects of `DatabaseManager` types. + """ + study_manager = MagicMock(spec=StudyManager) + run_manager = MagicMock(spec=RunManager) + logical_worker_manager = MagicMock(spec=LogicalWorkerManager) + physical_worker_manager = MagicMock(spec=PhysicalWorkerManager) + + managers = { + "study": study_manager, + "run": run_manager, + "logical_worker": logical_worker_manager, + "physical_worker": physical_worker_manager, + } + + return managers + + +@pytest.fixture +def mock_merlin_db( + mock_backend: MagicMock, mock_entity_managers: FixtureDict[str, MagicMock] +) -> Generator[MerlinDatabase, None, None]: + """ + Create a `MerlinDatabase` instance with mocked components. + + Args: + mock_backend: A mocked `ResultsBackend` instance. + mock_entity_managers: A dictionary of mocked objects of `DatabaseManager` types. + + Returns: + A `MerlinDatabase` instance with moocked attributes. + """ + # TODO when we drop support for python 3.8, replace the ExitStack call with the comment below + # with ( + # patch("merlin.db_scripts.merlin_db.backend_factory") as mock_factory, + # patch("merlin.db_scripts.merlin_db.StudyManager", return_value=mock_entity_managers["study"]), + # patch("merlin.db_scripts.merlin_db.RunManager", return_value=mock_entity_managers["run"]), + # patch("merlin.db_scripts.merlin_db.LogicalWorkerManager", return_value=mock_entity_managers["logical_worker"]), + # patch("merlin.db_scripts.merlin_db.PhysicalWorkerManager", return_value=mock_entity_managers["physical_worker"]), + # ): + with ExitStack() as stack: + mock_factory = stack.enter_context(patch("merlin.db_scripts.merlin_db.backend_factory")) + stack.enter_context(patch("merlin.db_scripts.merlin_db.StudyManager", return_value=mock_entity_managers["study"])) + stack.enter_context(patch("merlin.db_scripts.merlin_db.RunManager", return_value=mock_entity_managers["run"])) + stack.enter_context( + patch("merlin.db_scripts.merlin_db.LogicalWorkerManager", return_value=mock_entity_managers["logical_worker"]) + ) + stack.enter_context( + patch("merlin.db_scripts.merlin_db.PhysicalWorkerManager", return_value=mock_entity_managers["physical_worker"]) + ) + + mock_factory.get_backend.return_value = mock_backend + db = MerlinDatabase() + + # Replace backend and entity managers with mocks + db.backend = mock_backend + db._entity_managers = mock_entity_managers + + yield db + + +class TestMerlinDatabase: + """Test cases for `MerlinDatabase` class.""" + + def test_init(self, mock_backend: MagicMock): + """ + Test initialization of `MerlinDatabase`. + + Args: + mock_backend: A mocked `ResultsBackend` instance. + """ + # TODO when we drop support for python 3.8, replace the ExitStack call with the comment below + # with ( + # patch("merlin.db_scripts.merlin_db.backend_factory") as mock_factory, + # patch("merlin.db_scripts.merlin_db.StudyManager") as mock_study_manager, + # patch("merlin.db_scripts.merlin_db.RunManager") as mock_run_manager, + # patch("merlin.db_scripts.merlin_db.LogicalWorkerManager") as mock_logical_worker_manager, + # patch("merlin.db_scripts.merlin_db.PhysicalWorkerManager") as mock_physical_worker_manager, + # patch("merlin.config.configfile.CONFIG") as mock_config, + # ): + with ExitStack() as stack: + mock_factory = stack.enter_context(patch("merlin.db_scripts.merlin_db.backend_factory")) + mock_study_manager = stack.enter_context(patch("merlin.db_scripts.merlin_db.StudyManager")) + mock_run_manager = stack.enter_context(patch("merlin.db_scripts.merlin_db.RunManager")) + mock_logical_worker_manager = stack.enter_context(patch("merlin.db_scripts.merlin_db.LogicalWorkerManager")) + mock_physical_worker_manager = stack.enter_context(patch("merlin.db_scripts.merlin_db.PhysicalWorkerManager")) + mock_config = stack.enter_context(patch("merlin.config.configfile.CONFIG")) + + # Configure mocks + mock_factory.get_backend.return_value = mock_backend + mock_config.results_backend.name = "redis" + + # Create instances + db = MerlinDatabase() + + # Verify backend was created + mock_factory.get_backend.assert_called_once_with("redis") + + # Verify entity managers were created with the backend + mock_study_manager.assert_called_once_with(mock_backend) + mock_run_manager.assert_called_once_with(mock_backend) + mock_logical_worker_manager.assert_called_once_with(mock_backend) + mock_physical_worker_manager.assert_called_once_with(mock_backend) + + # Verify entity managers that have set_db_reference method were set up properly + for manager in [ + mock_study_manager.return_value, + mock_run_manager.return_value, + mock_logical_worker_manager.return_value, + mock_physical_worker_manager.return_value, + ]: + if hasattr(manager, "set_db_reference"): + manager.set_db_reference.assert_called_once_with(db) + + def test_properties(self, mock_merlin_db: MerlinDatabase, mock_entity_managers: FixtureDict[str, MagicMock]): + """ + Test property accessors for entity managers. + + Args: + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + mock_entity_managers: A dictionary of mocked objects of `DatabaseManager` types. + """ + assert mock_merlin_db.studies is mock_entity_managers["study"] + assert mock_merlin_db.runs is mock_entity_managers["run"] + assert mock_merlin_db.logical_workers is mock_entity_managers["logical_worker"] + assert mock_merlin_db.physical_workers is mock_entity_managers["physical_worker"] + + def test_get_db_type(self, mock_merlin_db: MerlinDatabase, mock_backend: MagicMock): + """ + Test `get_db_type` method. + + Args: + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + mock_backend: A mocked `ResultsBackend` instance. + """ + result = mock_merlin_db.get_db_type() + mock_backend.get_name.assert_called_once() + assert result == "mock_redis" + + def test_get_db_version(self, mock_merlin_db: MerlinDatabase, mock_backend: MagicMock): + """ + Test `get_db_version` method. + + Args: + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + mock_backend: A mocked `ResultsBackend` instance. + """ + result = mock_merlin_db.get_db_version() + mock_backend.get_version.assert_called_once() + assert result == "1.0" + + def test_get_connection_string(self, mock_merlin_db: MerlinDatabase, mock_backend: MagicMock): + """ + Test `get_connection_string` method. + + Args: + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + mock_backend: A mocked `ResultsBackend` instance. + """ + result = mock_merlin_db.get_connection_string() + mock_backend.get_connection_string.assert_called_once() + assert result == "redis://localhost:6379/0" + + def test_validate_entity_type_valid(self, mock_merlin_db: MerlinDatabase): + """ + Test `_validate_entity_type` with valid entity types. + + Args: + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + """ + for entity_type in ["study", "run", "logical_worker", "physical_worker"]: + # Should not raise an exception + mock_merlin_db._validate_entity_type(entity_type) + + def test_validate_entity_type_invalid(self, mock_merlin_db: MerlinDatabase): + """ + Test `_validate_entity_type` with invalid entity type. + + Args: + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + """ + with pytest.raises(EntityManagerNotSupportedError) as excinfo: + mock_merlin_db._validate_entity_type("invalid_type") + assert "Entity type not supported: invalid_type" in str(excinfo.value) + + def test_create(self, mock_merlin_db: MerlinDatabase, mock_entity_managers: FixtureDict[str, MagicMock]): + """ + Test `create` method. + + Args: + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + mock_entity_managers: A dictionary of mocked objects of `DatabaseManager` types. + """ + # Test with valid entity type + entity_type = "study" + mock_entity_managers[entity_type].create.return_value = "created_study" + + result = mock_merlin_db.create(entity_type, "arg1", key1="value1") + + mock_entity_managers[entity_type].create.assert_called_once_with("arg1", key1="value1") + assert result == "created_study" + + # Test with invalid entity type + with pytest.raises(EntityManagerNotSupportedError): + mock_merlin_db.create("invalid_type") + + def test_get(self, mock_merlin_db: MerlinDatabase, mock_entity_managers: FixtureDict[str, MagicMock]): + """ + Test `get` method. + + Args: + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + mock_entity_managers: A dictionary of mocked objects of `DatabaseManager` types. + """ + # Test with valid entity type + entity_type = "run" + mock_entity_managers[entity_type].get.return_value = "run_entity" + + result = mock_merlin_db.get(entity_type, "run_id", extra_param=True) + + mock_entity_managers[entity_type].get.assert_called_once_with("run_id", extra_param=True) + assert result == "run_entity" + + # Test with invalid entity type + with pytest.raises(EntityManagerNotSupportedError): + mock_merlin_db.get("invalid_type", "id") + + def test_get_all(self, mock_merlin_db: MerlinDatabase, mock_entity_managers: FixtureDict[str, MagicMock]): + """ + Test `get_all` method. + + Args: + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + mock_entity_managers: A dictionary of mocked objects of `DatabaseManager` types. + """ + # Test with valid entity type + entity_type = "logical_worker" + mock_entity_managers[entity_type].get_all.return_value = ["worker1", "worker2"] + + result = mock_merlin_db.get_all(entity_type) + + mock_entity_managers[entity_type].get_all.assert_called_once() + assert result == ["worker1", "worker2"] + + # Test with invalid entity type + with pytest.raises(EntityManagerNotSupportedError): + mock_merlin_db.get_all("invalid_type") + + def test_delete(self, mock_merlin_db: MerlinDatabase, mock_entity_managers: FixtureDict[str, MagicMock]): + """ + Test `delete` method. + + Args: + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + mock_entity_managers: A dictionary of mocked objects of `DatabaseManager` types. + """ + # Test with valid entity type + entity_type = "physical_worker" + + mock_merlin_db.delete(entity_type, "worker_id", force=True) + + mock_entity_managers[entity_type].delete.assert_called_once_with("worker_id", force=True) + + # Test with invalid entity type + with pytest.raises(EntityManagerNotSupportedError): + mock_merlin_db.delete("invalid_type", "id") + + def test_delete_all(self, mock_merlin_db: MerlinDatabase, mock_entity_managers: FixtureDict[str, MagicMock]): + """ + Test `delete_all` method. + + Args: + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + mock_entity_managers: A dictionary of mocked objects of `DatabaseManager` types. + """ + # Test with valid entity type + entity_type = "study" + + mock_merlin_db.delete_all(entity_type, force=True) + + mock_entity_managers[entity_type].delete_all.assert_called_once_with(force=True) + + # Test with invalid entity type + with pytest.raises(EntityManagerNotSupportedError): + mock_merlin_db.delete_all("invalid_type") + + def test_get_everything(self, mock_merlin_db: MerlinDatabase, mock_entity_managers: FixtureDict[str, MagicMock]): + """ + Test `get_everything` method. + + Args: + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + mock_entity_managers: A dictionary of mocked objects of `DatabaseManager` types. + """ + # Set up mock returns + mock_entity_managers["study"].get_all.return_value = ["study1", "study2"] + mock_entity_managers["run"].get_all.return_value = ["run1", "run2", "run3"] + mock_entity_managers["logical_worker"].get_all.return_value = ["logical_worker1"] + mock_entity_managers["physical_worker"].get_all.return_value = ["physical_worker1", "physical_worker2"] + + result = mock_merlin_db.get_everything() + + # Verify all get_all methods were called + for manager in mock_entity_managers.values(): + manager.get_all.assert_called_once() + + # Verify the result contains all entities + assert result == [ + "study1", + "study2", + "run1", + "run2", + "run3", + "logical_worker1", + "physical_worker1", + "physical_worker2", + ] + + def test_delete_everything_confirmed( + self, caplog: CaptureFixture, mock_merlin_db: MerlinDatabase, mock_backend: MagicMock + ): + """ + Test `delete_everything` method with confirmation. + + Args: + caplog: A built-in fixture from the pytest library to capture logs. + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + mock_backend: A mocked `ResultsBackend` instance. + """ + caplog.set_level(logging.INFO) + with patch("builtins.input", return_value="y"): + + mock_merlin_db.delete_everything() + + # Verify database flush was called + mock_backend.flush_database.assert_called_once() + assert "Flushing the database..." in caplog.text + assert "Database successfully flushed." in caplog.text + + def test_delete_everything_cancelled( + self, caplog: CaptureFixture, mock_merlin_db: MerlinDatabase, mock_backend: MagicMock + ): + """ + Test `delete_everything` method when cancelled. + + Args: + caplog: A built-in fixture from the pytest library to capture logs. + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + mock_backend: A mocked `ResultsBackend` instance. + """ + caplog.set_level(logging.INFO) + with patch("builtins.input", return_value="n"): + + mock_merlin_db.delete_everything() + + # Verify database flush was NOT called + mock_backend.flush_database.assert_not_called() + assert "Database flush cancelled." in caplog.text + + def test_delete_everything_invalid_input_then_confirmed( + self, + caplog: CaptureFixture, + mock_merlin_db: MerlinDatabase, + mock_backend: MagicMock, + ): + """ + Test `delete_everything` method with invalid input followed by confirmation. + + Args: + caplog: A built-in fixture from the pytest library to capture logs. + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + mock_backend: A mocked `ResultsBackend` instance. + """ + caplog.set_level(logging.INFO) + # First input is invalid, second is valid 'y' + with patch("builtins.input", side_effect=["invalid", "y"]): + + mock_merlin_db.delete_everything() + + # Verify database flush was called after valid input + mock_backend.flush_database.assert_called_once() + assert "Flushing the database..." in caplog.text + assert "Database successfully flushed." in caplog.text + + def test_delete_everything_force(self, caplog: CaptureFixture, mock_merlin_db: MerlinDatabase, mock_backend: MagicMock): + """ + Test `delete_everything` method with force=True. + + Args: + caplog: A built-in fixture from the pytest library to capture logs. + mock_merlin_db: A `MerlinDatabase` instance with moocked attributes. + mock_backend: A mocked `ResultsBackend` instance. + """ + caplog.set_level(logging.INFO) + + mock_merlin_db.delete_everything(force=True) + + # Verify database flush was called without prompting + mock_backend.flush_database.assert_called_once() + assert "Flushing the database..." in caplog.text + assert "Database successfully flushed." in caplog.text diff --git a/tests/unit/server/__init__.py b/tests/unit/server/__init__.py index e69de29bb..3232b50b9 100644 --- a/tests/unit/server/__init__.py +++ b/tests/unit/server/__init__.py @@ -0,0 +1,5 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## diff --git a/tests/unit/server/test_RedisConfig.py b/tests/unit/server/test_RedisConfig.py index 321d2f38a..10be437d0 100644 --- a/tests/unit/server/test_RedisConfig.py +++ b/tests/unit/server/test_RedisConfig.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the RedisConfig class of the `server_util.py` module. diff --git a/tests/unit/server/test_server_commands.py b/tests/unit/server/test_server_commands.py index ec52df2a0..d390183fd 100644 --- a/tests/unit/server/test_server_commands.py +++ b/tests/unit/server/test_server_commands.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the `server_commands.py` module. """ diff --git a/tests/unit/server/test_server_config.py b/tests/unit/server/test_server_config.py index 51b74a842..c2ea0b040 100644 --- a/tests/unit/server/test_server_config.py +++ b/tests/unit/server/test_server_config.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the `server_config.py` module. """ @@ -6,13 +12,13 @@ import logging import os import string +from importlib import resources from typing import Dict, Tuple, Union import pytest import yaml from merlin.server.server_config import ( - MERLIN_CONFIG_DIR, PASSWORD_LENGTH, ServerStatus, check_process_file_format, @@ -30,12 +36,6 @@ from merlin.server.server_util import CONTAINER_TYPES, MERLIN_SERVER_SUBDIR, ServerConfig -try: - from importlib import resources -except ImportError: - import importlib_resources as resources - - def test_generate_password_no_pass_command(): """ Test the `generate_password` function with no password command. @@ -201,7 +201,7 @@ def test_create_server_config_merlin_config_dir_nonexistent( server_testing_dir: str, ): """ - Tests the `create_server_config` function with MERLIN_CONFIG_DIR not existing. + Tests the `create_server_config` function with MERLIN_HOME not existing. This should log an error and return False. :param mocker: A built-in fixture from the pytest-mock library to create a Mock object @@ -209,7 +209,7 @@ def test_create_server_config_merlin_config_dir_nonexistent( :param server_testing_dir: The path to the the temp output directory for server tests """ nonexistent_dir = f"{server_testing_dir}/merlin_config_dir" - mocker.patch("merlin.server.server_config.MERLIN_CONFIG_DIR", nonexistent_dir) + mocker.patch("merlin.server.server_config.MERLIN_HOME", nonexistent_dir) assert not create_server_config() assert f"Unable to find main merlin configuration directory at {nonexistent_dir}" in caplog.text @@ -220,7 +220,7 @@ def test_create_server_config_server_subdir_nonexistent_oserror( server_testing_dir: str, ): """ - Tests the `create_server_config` function with MERLIN_CONFIG_DIR/MERLIN_SERVER_SUBDIR + Tests the `create_server_config` function with MERLIN_HOME/MERLIN_SERVER_SUBDIR not existing and an OSError being raised. This should log an error and return False. :param mocker: A built-in fixture from the pytest-mock library to create a Mock object @@ -228,9 +228,9 @@ def test_create_server_config_server_subdir_nonexistent_oserror( :param server_testing_dir: The path to the the temp output directory for server tests """ - # Mock MERLIN_CONFIG_DIR and MERLIN_SERVER_SUBDIR + # Mock MERLIN_HOME and MERLIN_SERVER_SUBDIR nonexistent_server_subdir = "test_create_server_config_server_subdir_nonexistent" - mocker.patch("merlin.server.server_config.MERLIN_CONFIG_DIR", server_testing_dir) + mocker.patch("merlin.server.server_config.MERLIN_HOME", server_testing_dir) mocker.patch("merlin.server.server_config.MERLIN_SERVER_SUBDIR", nonexistent_server_subdir) # Mock os.mkdir so it raises an OSError @@ -255,7 +255,7 @@ def test_create_server_config_no_server_config( """ # Mock the necessary variables/functions to get us to the pull_server_config call - mocker.patch("merlin.server.server_config.MERLIN_CONFIG_DIR", server_testing_dir) + mocker.patch("merlin.server.server_config.MERLIN_HOME", server_testing_dir) mocker.patch("merlin.server.server_config.copy_container_command_files", return_value=True) mock_open_func = mocker.mock_open(read_data="key: value") mocker.patch("builtins.open", mock_open_func) @@ -285,14 +285,14 @@ def test_create_server_config_no_server_dir( caplog.set_level(logging.INFO) # Mock the necessary variables/functions to get us to the get_config_dir call - mocker.patch("merlin.server.server_config.MERLIN_CONFIG_DIR", server_testing_dir) + mocker.patch("merlin.server.server_config.MERLIN_HOME", server_testing_dir) mocker.patch("merlin.server.server_config.copy_container_command_files", return_value=True) mock_open_func = mocker.mock_open(read_data="key: value") mocker.patch("builtins.open", mock_open_func) - mocker.patch("merlin.server.server_config.pull_server_config", return_value=ServerConfig(server_server_config)) + mocker.patch("merlin.server.server_config.load_server_config", return_value=ServerConfig(server_server_config)) # Mock the get_config_dir call to return a directory that doesn't exist yet - nonexistent_dir = f"{server_testing_dir}/merlin_server" + nonexistent_dir = os.path.join(server_testing_dir, "merlin_server") mocker.patch("merlin.server.server_util.ContainerConfig.get_config_dir", return_value=nonexistent_dir) assert create_server_config() @@ -394,7 +394,7 @@ def setup_pull_server_config_mock( :param server_server_config: A pytest fixture of test data to pass to the ServerConfig class """ mocker.patch("merlin.server.server_util.AppYaml.get_data", return_value=server_app_yaml_contents) - mocker.patch("merlin.server.server_config.MERLIN_CONFIG_DIR", server_testing_dir) + mocker.patch("merlin.server.server_config.MERLIN_HOME", server_testing_dir) mock_data = mocker.mock_open(read_data=str(server_server_config)) mocker.patch("builtins.open", mock_data) @@ -402,9 +402,9 @@ def setup_pull_server_config_mock( @pytest.mark.parametrize( "key_to_delete, expected_log_message", [ - ("container", 'Unable to find "container" object in {default_app_yaml}'), - ("container.format", 'Unable to find "format" in {default_app_yaml}'), - ("process", "Process config not found in {default_app_yaml}"), + ("container", 'Unable to find "container"'), + ("container.format", 'Unable to find "format"'), + ("process", 'Unable to find "process"'), ], ) def test_pull_server_config_missing_config_keys( @@ -438,8 +438,7 @@ def test_pull_server_config_missing_config_keys( setup_pull_server_config_mock(mocker, server_testing_dir, server_app_yaml_contents, server_server_config) assert pull_server_config() is None - default_app_yaml = os.path.join(MERLIN_CONFIG_DIR, "app.yaml") - assert expected_log_message.format(default_app_yaml=default_app_yaml) in caplog.text + assert expected_log_message in caplog.text @pytest.mark.parametrize("key_to_delete", ["command", "run_command", "stop_command", "pull_command"]) @@ -500,8 +499,7 @@ def test_pull_server_config_missing_process_needed_key( setup_pull_server_config_mock(mocker, server_testing_dir, server_app_yaml_contents, server_server_config) assert pull_server_config() is None - default_app_yaml = os.path.join(MERLIN_CONFIG_DIR, "app.yaml") - assert f'Process necessary "{key_to_delete}" command configuration not found in {default_app_yaml}' in caplog.text + assert f'Process necessary "{key_to_delete}" command configuration not found in' in caplog.text def test_pull_server_config_no_issues( diff --git a/tests/unit/server/test_server_util.py b/tests/unit/server/test_server_util.py index 909cb7cdf..e0a308f45 100644 --- a/tests/unit/server/test_server_util.py +++ b/tests/unit/server/test_server_util.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the `server_util.py` module. """ @@ -10,6 +16,7 @@ import pytest from merlin.server.server_util import ( + CONFIG_DIR, AppYaml, ContainerConfig, ContainerFormatConfig, @@ -134,7 +141,7 @@ def test_init_with_missing_data(self): assert config.image == ContainerConfig.IMAGE_NAME assert config.url == ContainerConfig.REDIS_URL assert config.config == ContainerConfig.CONFIG_FILE - assert config.config_dir == ContainerConfig.CONFIG_DIR + assert config.config_dir == CONFIG_DIR assert config.pfile == ContainerConfig.PROCESS_FILE assert config.pass_file == ContainerConfig.PASSWORD_FILE assert config.user_file == ContainerConfig.USERS_FILE diff --git a/tests/unit/spec/__init__.py b/tests/unit/spec/__init__.py index e69de29bb..3232b50b9 100644 --- a/tests/unit/spec/__init__.py +++ b/tests/unit/spec/__init__.py @@ -0,0 +1,5 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## diff --git a/tests/unit/spec/test_specification.py b/tests/unit/spec/test_specification.py index afc930a6e..fe3d33126 100644 --- a/tests/unit/spec/test_specification.py +++ b/tests/unit/spec/test_specification.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + import os import shutil import tempfile diff --git a/tests/unit/study/__init__.py b/tests/unit/study/__init__.py index 37cabcad1..3232b50b9 100644 --- a/tests/unit/study/__init__.py +++ b/tests/unit/study/__init__.py @@ -1,29 +1,5 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## diff --git a/tests/unit/study/status_test_files/combine_status_files.py b/tests/unit/study/status_test_files/combine_status_files.py index b52b35f1d..1f494bde5 100644 --- a/tests/unit/study/status_test_files/combine_status_files.py +++ b/tests/unit/study/status_test_files/combine_status_files.py @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Script to combine all status files from a study into one diff --git a/tests/unit/study/status_test_files/shared_tests.py b/tests/unit/study/status_test_files/shared_tests.py index 3e6b0fde8..2b1ced611 100644 --- a/tests/unit/study/status_test_files/shared_tests.py +++ b/tests/unit/study/status_test_files/shared_tests.py @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ This module contains all shared tests needed for testing both the Status object and the DetailedStatus object. diff --git a/tests/unit/study/status_test_files/status_test_variables.py b/tests/unit/study/status_test_files/status_test_variables.py index 6a7f7eb28..de7ec86fe 100644 --- a/tests/unit/study/status_test_files/status_test_variables.py +++ b/tests/unit/study/status_test_files/status_test_variables.py @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """This module holds variables that will be used to test against output from calls to status methods""" import os diff --git a/tests/unit/study/test_detailed_status.py b/tests/unit/study/test_detailed_status.py index e6eea4748..70220215a 100644 --- a/tests/unit/study/test_detailed_status.py +++ b/tests/unit/study/test_detailed_status.py @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the DetailedStatus class in the status.py module """ diff --git a/tests/unit/study/test_status.py b/tests/unit/study/test_status.py index 6786a1230..414f66fac 100644 --- a/tests/unit/study/test_status.py +++ b/tests/unit/study/test_status.py @@ -1,32 +1,9 @@ -############################################################################### -# Copyright (c) 2023, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory -# Written by the Merlin dev team, listed in the CONTRIBUTORS file. -# -# -# LLNL-CODE-797170 -# All rights reserved. -# This file is part of Merlin, Version: 1.12.2. -# -# For details, see https://github.com/LLNL/merlin. -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -############################################################################### +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the Status class in the status.py module """ diff --git a/tests/unit/study/test_study.py b/tests/unit/study/test_study.py index 7dbef0905..bf60895cd 100644 --- a/tests/unit/study/test_study.py +++ b/tests/unit/study/test_study.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the maestroadapter.py module. """ diff --git a/tests/unit/test_examples_generator.py b/tests/unit/test_examples_generator.py index 7d4d879fb..5002bcc20 100644 --- a/tests/unit/test_examples_generator.py +++ b/tests/unit/test_examples_generator.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the `merlin/examples/generator.py` module. """ diff --git a/tests/unit/utils/test_dict_deep_merge.py b/tests/unit/utils/test_dict_deep_merge.py index 133897f36..d1866deff 100644 --- a/tests/unit/utils/test_dict_deep_merge.py +++ b/tests/unit/utils/test_dict_deep_merge.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Tests for the `dict_deep_merge` function defined in the `utils.py` module. """ diff --git a/tests/unit/utils/test_get_package_version.py b/tests/unit/utils/test_get_package_version.py index fad4623cc..01b10d703 100644 --- a/tests/unit/utils/test_get_package_version.py +++ b/tests/unit/utils/test_get_package_version.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + import sys from unittest.mock import patch diff --git a/tests/unit/utils/test_time_formats.py b/tests/unit/utils/test_time_formats.py index 93fe819d9..a9861ef81 100644 --- a/tests/unit/utils/test_time_formats.py +++ b/tests/unit/utils/test_time_formats.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + import datetime from typing import List, Optional, Union diff --git a/tests/utils.py b/tests/utils.py index 0b408db54..a3bdf6eee 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -1,3 +1,9 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + """ Utility functions for our test suite. """ From b79c5677816fb26abd72f7812dc5e2a7b0e6246b Mon Sep 17 00:00:00 2001 From: Brian Gunnarson <49216024+bgunnar5@users.noreply.github.com> Date: Tue, 1 Jul 2025 07:49:43 -0700 Subject: [PATCH 2/2] Version/1.13.0b2 (#518) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * remove github text that was causing errors * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning Co-authored-by: Joe Koning * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning Co-authored-by: Joe Koning * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks * Fix filenames for OpenFoam tutorial (#475) * bugfix/deep-merge-existing-keys (#476) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * remove DeepMergeException and add conflict_handler to dict_deep_merge * add conflict handler to dict_deep_merge * fix broken tests for detailed-status * use caplog fixture rather than IO stream * add ability to define module-specific fixtures * add tests for read/write status files and conlict handling * add caplog explanation to docstrings * update CHANGELOG * run fix-style * add pytest-mock as dependency for test suite * clean up input check in dict_deep_merge * Improved Info (#477) * Add merlin version to banner * Add python package info to and clean up 'merlin info' * Add some unit tests * Force GitHub runner checkout to grab the whole history, fixing CHANGELOG test bug * Update CHANGELOG to show bugfix to CHANGELOG test * Target is in source's history (#478) * New github action test to make sure target has been merged into source * Fix link to merlin banner image (#479) * bugfix/status_nested_workspace (#480) * remove a merge conflict statement that was missed * have status ignore nested workspaces and modify merge rules * update CHANGELOG * fixed issue with escape sequences in ascii art * apply Luc's suggestion * add setuptools as a requirement since python 3.12 doesn't have it natively * modify unit tests for status to use pytest rather than unittest * update CHANGELOG * add fixtures for status testing and add nested workflow test * update CHANGELOG * bugfix/celery-chord-error (#481) * remove a merge conflict statement that was missed * add celery results backend patch to stop ChordErrors * add MERLIN_RAISE_ERROR return code * add tests to ensure chord error isn't raised * add RAISE_ERROR to docs * update CHANGELOG * fix lint issues * up the sleep time on the chord error test * add new steps to the chord err test spec * add tree statement to the new test for debugging * upping sleep time to see if that fixes github action for python 3.7 * change sleep time for new test based on python version * run fix style * remove specific sleep time for diff python versions * release/1.12.2b1 (#482) * remove a merge conflict statement that was missed * bump version to 1.12.2b1 * bugfix/flux-nodes (#484) * remove a merge conflict statement that was missed * fix flux node allocation issue * allow for vars to be used with nodes settings of workers/batch * add tests for var usage with nodes * update CHANGELOG * run fix-style * bugfix/flux-nodes-prior-versions (#487) * add a version check for flux when getting node count * update CHANGELOG * add major version check for flux * Change Task ID to directory path (#486) * Modifying task id to include directory * Adding Several New Unit Tests (#490) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * split up create_server_config and write tests for it * add tests for config_merlin_server function * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * update CHANGELOG * add Ryan and Joe's suggestions * update tests to use newly named functions * fix linter issue * release/1.12.2 (#491) * update version to 1.12.2 * fix style issue * Refactor/distributed-tests (#493) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * Include celerymanager and update celeryadapter to check the status of celery workers. * Fixed issue where the update status was outside of if statement for checking workers * Include worker status stop and add template for merlin restart * Added comment to the CeleryManager init * Increment db_num instead of being fixed * Added other subprocess parameters and created a linking system for redis to store env dict * Implemented stopping of celery workers and restarting workers properly * Update stopped to stalled for when the worker doesn't respond to restart * Working merlin manager run but start and stop not working properly * Made fix for subprocess to start new shell and fixed manager start and stop * Added comments and update changelog * Include style fixes * Fix style for black * Revert launch_job script that was edited when doing automated lint * Move importing of CONFIG to be within redis_connection due to error of config not being created yet * Added space to fix style * Revert launch_jobs.py: * Update import of all merlin.config to be in the function * suggested changes plus beginning work on monitor/manager collab * move managers to their own folder and fix ssl problems * final PR touch ups * Fix lint style changes * Fixed issue with context manager * Reset file that was incorrect changed * Check for ssl cert before applying to Redis connection * Comment out Active tests for celerymanager * split up create_server_config and write tests for it * add tests for config_merlin_server function * Fix lint issue with unused import after commenting out Active celery tests * Fixed style for import * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * Fixed kwargs being modified when making a copy for saving to redis worker args. * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * Added password check and omit if a password doesn't exist * update CHANGELOG * change testing log level to debug * add debug statement for redis_connection * change debug log to info so github ci will display it * attempt to fix password missing from Namespace error * run checks for all necessary configurations * convert stop-workers tests to pytest format * update github wf and comment out stop-workers tests in definitions.py * add missing key to GH wf file * fix invalid syntax in definitions.py * comment out stop_workers tests * playing with new caches for workflow CI * fix yaml syntax error * fix typo for getting runner os * fix test and add python version to CI cache * add in common-setup step again with caches this time * run fix-style * update CHANGELOG * fix remaining style issues * run without caches to compare execution time of test suite * allow redis config to not use ssl * remove stop-workers and query-workers tests from definitions.py * create helper_funcs file with common testing functions * move query-workers to pytest and add base class w/ stop-workers tests * update CHANGELOG * final changes for the stop-workers & query-workers tests * run fix-style * move stop and query workers tests to the same file * run fix-style * go back to original cache setup * try new cache for singularity install * fix syntax issue in github workflow * attempt to fix singularity cache * remove ls statement that breaks workflow * revert back to no common setup * remove unnecessary dependency * update github actions versions to use latest * update action versions that didn't save * run fix-style * move distributed test suite actions back to v2 * add 'merlin run' tests and port existing ones to pytest * update CHANGELOG * add aliased fixture types for typehinting * add tests for the purge command * update CHANGELOG * update run command tests to use conditions when appropriate * start work on adding workflow tests * create function and class scoped config fixtures * add Tuple fixture type * get e2e test of feature_demo workflow running * add check for proper variable substitution in e2e test * generalize functionality to run workflows * add create_testing_dir fixture * port chord error workflow to pytest * create dataclasses to house common fixtures and reduce fixture import requirements * fix lint issues * remove hard requirement of Annotated type for python 3.7 and 3.8 * remove distributed test CI and add unit test CI * fix typo in fixture_types and fix lint issues * run fix-style * add check for python2 before adding that condition check * convert local run test to use StepFinishedFilesCount condition * update CHANGELOG.md * fix problem created by merge conflict when mergin develop * remove manager functionality from this PR * update README for test suite * change SIGTERM to SIGKILL * update Makefile to include new changes to test suite --------- Co-authored-by: Ryan Lee Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> * Drop Python 3.7 and Add Python 3.12 & 3.13 (#495) * drop support for py 3.7, add support for py 3.12 and 3.13 * fix docs build issue * remove py 3.7 and add py 3.12/3.13 to integration tests * update Makefile to use target_version variable and update spellbook requirements in examples * Requirements fixes (#501) * add fix for broken deepdiff dependency in py38 and remove py37 specific requirements that were missed * remove try/except imports that are no longer necessary * update CHANGELOG * docs/api_docs (#496) * add/update docstrings for top-level files and enable API docs * remove Generator type hint * fix invalid escape sequences * add docstrings for the common/ directory * fix styling and add in cross-references to portions of Merlin codebase * give code blocks types for formatting * add api docs for half of the study directory * add API docs for study.py * ignore the data path for API docs * finish API docs for study directory * finish api docs for spec/ directory * update CHANGELOG * final cleanup of API docs for common, spec, and study folders * finish API docs for examples folder * began work on API docs for config folder * write API docs for server utils * finish api docs for the server directory * finish API docs for config directory * final cleanup * fix doc build issues * add section explaining API docs to the docs' README file * run fix-style * update readthedocs to build with latest python * fix too few arguments to generator issue * rename in MerlinStepRecord so that it's hidden again * add most of Charles' suggestions * remove unused openfilelist.py, opennpylib.py, and merlin_templates.py files, and add remainder of Charles suggestions * run fix-style * rename load_default_user_names to set_username_and_vhost * Update README.md to remove lgtm banner (#488) * Update README.md to remove lgtm banner * Update CHANGELOG.md * Delete .lgtm.yml * move changelog update to the unreleased section * remove lgtm.yml --------- Co-authored-by: Brian Gunnarson * Refactor/simplified-ci (#504) * split python and singularity setup into individual actions * add shell to new actions * try to fix shell issue * remove install-deps * reorder cache step and check pip version in the output * update CHANGELOG and make path to definitions.py relative * fix style issues * move cache check to setup-python action * fix an issue with docs build on python 3.9 (#503) * fix an issue with docs build on python 3.9 * adding ChatGPT suggestion to use get-pip.py script * add python version check to reinstall pip CI step * add reinstall pip step to all jobs that need it * save get-pip.py to tmp folder instead of directly to the repo * update CHANGELOG * fix .wci.yml links (#505) * Refactor/config (#498) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * Refactor/config-defaults (#497) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * remove debug step from github ci * add user permission check to local test suite * create a CI workflow for push and pull_request_target so secrets can be accessed * add link to article where I found this solution * add a merlin info test so we can see what server CI is connecting to * add unit tests for new config commands * add MerlinConfigManager class to replace config-related functions * add unit tests for MerlinConfigManager and update integration tests * comment out the pr-target workflow since it's not working * run fix-style * rewrite tests for find_config_file * remove uses of CONFIGFILE_DIR and chdir from configfile unit tests * run fix-style * re-order the find_config_file function to be local, config_path, merlin home * remove pr-target workflow, add missing docstrings, and run fix-style * add create and use commands, update MerlinConfigManager as necessary * update test suite for config command * update all documentation where 'merlin config' is mentioned * change merlin config to merlin config create * update CHANGELOG * run fix-style * update GitHub workflow to use 'merlin config create' * try to fix f-string issue * add Charles' suggestions * fix broken tests and security issue with new log statement * remove app.yaml file that was accidentally pushed * fix config file validation as Charles suggested * Refactor/server-config (#506) * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * change print to log statement * bugfix/local-config (#507) * add local mode config initialization * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * add unit tests for new functions in configfile.py * run fix-style and update CHANGELOG * change defaults to not use getpass * run fix-style * add changes that Charles suggested * feature/monitor-auto-restart (#494) * add classes for backends to help abstract them * establish templates and basic functionality for database interactions * flush out creation and retrieval of studies and runs * finish study/run storage, retrieval, and deletion to db functionality * add StudyNotFoundError and RunNotFoundError * add ability to dump run metadata to merlin_info directory * add new Celery signature to mark a run as complete * create merlin database commands * integrate database changes with monitor command * run fix-style and fix lint issues * update CHANGELOG and run fix-style with python 12 * last run of style fixes before push * run fix style with python 3.13 * create foundation for monitor auto-restart test * move get_backend_password import inside RedisBackend class * fix lint issue and comment out monitor test for now * fix redis connection so it works for ssl and non-ssl connections * add test for workflow auto-restart functionality * run linter * clean up print for DatabaseStudy and add -k option to delete all-studies * add docs for new monitor and database command * run fix-style * update built-in celery settings and retry exceptions to better handle timeouts * add new Monitor classes to handle the monitor command * update CHANGELOG and fix monitor test * fix lint issues * add worker entries and interactions with the database * add ABC class for database entities * add multi-run and multi-study queries, queries by study id, and queries by run workspace * add ability to delete multiple runs, studies, and workers. Also add ability to delete study by id and run by workspace * update docstrings for db and backend files * run fix-style * add repr method to HasRegex to make parsing GitHub CI easier * add ability to delete everything from db and retrieve connection string from MerlinDatabase * rename DatabaseRun and DatabaseStudy to more appropriate RunEntity and StudyEntity * split WorkerModel into LogicalWorkerModel and PhysicalWorkerModel * split redis backend class up into store classes for different entities. Removes db worker interaction temporarily * add logical worker functionality to database * add CLI functionality for physical workers and clean up MerlinDatabase * add physical worker db entry creation on celery worker startup * comment out code that's breaking worker launch * fix bug with get_all_runs method not existing anymore * run fix-style * fix bug with load/delete of physical workers and add functionality to set worker status to stopped * fix issue where ids were passed to wait_for_workers instead of names * move stop-worker celery functionality to a signal handler * add Logical Worker with ID 7620359f-6deb-0056-8956-39495dba8e59 ------------------------------------------------ Name: worker2 Runs: ['2e12e7cd-e7b9-468b-92c7-1ba1dffba852'] Queues: ['[merlin]_sim_queue', '[merlin]_seq_queue'] Physical Workers: ['dc863387-6b54-4a93-8a27-507bf0fd789e', '41a28c9e-f58d-49da-87bf-cea1569398b9'] Additional Data: {} Physical Worker with ID 41a28c9e-f58d-49da-87bf-cea1569398b9 ------------------------------------------------ Name: celery@worker2.%rzadams1011 Logical Worker ID: 7620359f-6deb-0056-8956-39495dba8e59 Launch Command: None Args: {} Process ID: None Status: WorkerStatus.RUNNING Last Heartbeat: 2025-04-14 12:40:05.786023 Last Spinup: 2025-04-14 12:40:05.786028 Host: rzadams1011 Restart Count: 0.0 Additional Data: {} Physical Worker with ID dc863387-6b54-4a93-8a27-507bf0fd789e ------------------------------------------------ Name: celery@worker2.%rzadams1010 Logical Worker ID: 7620359f-6deb-0056-8956-39495dba8e59 Launch Command: None Args: {} Process ID: None Status: WorkerStatus.RUNNING Last Heartbeat: 2025-04-14 12:40:05.027347 Last Spinup: 2025-04-14 12:40:05.027353 Host: rzadams1010 Restart Count: 0.0 Additional Data: {} Run with ID 2e12e7cd-e7b9-468b-92c7-1ba1dffba852 ------------------------------------------------ Workspace: /usr/WS1/gunny/scalability_testing/studies/long_running_wf_20250414-124613 Study ID: 64def1c2-8ee8-40a4-9f74-36d5b4872b9e Queues: ['[merlin]_seq_queue', '[merlin]_sim_queue'] Workers: ['7620359f-6deb-0056-8956-39495dba8e59'] Parent: None Child: None Run Complete: False Additional Data: {} Study with ID 64def1c2-8ee8-40a4-9f74-36d5b4872b9e ------------------------------------------------ Name: long_running_wf Runs: ['2e12e7cd-e7b9-468b-92c7-1ba1dffba852'] Additional Data: {} command * factor out common code from entity classes into mixin classes * run fix-style * remove process kill from celery shutdown * write tests for high-level modules of backends folder * add tests for the RedisBackend class * write unit tests for redis_logical_worker_store.py * add unit tests for RedisPhysicalWorkerStore * add tests for RedisRunStore * move db entities to their own folder * update version to 1.13.0a1 * add init.py file to backends/redis folder * change the monitor test so that it has more samples and purges tasks rather than stopping workers * run fix-style * fix error from merging develop * fix import issue with enums * condense store logic into base class and mixin class * add tests for new combined logic, remove tests for separated logic * run fix-style * fix imports and string representations of entities * update logical worker queue entry to be a set instead of list * make worker queues a set and fix str representations * update docs for database command * run fix style * change some info logs to debug * fix the majority of the docs warnings * fix docs warnings and run fix style * finish the database command docs * fix log messages and bug with deleting physical worker entry * add log files per Charles' suggestion * refactor db entities to share additional common logic * fix bug when deleting runs by workspace after workspace has been removed * add unit tests and some integration tests for data_models and db_commands * add unit tests for entity and mixin classes * refactor MerlinDatabase to use newly added entity managers * move mixins folder into entities folder * run fix-style * move mixin tests and remove todos from physical worker test file * add tests for the merlin_db module * add tests for entity managers * run fix-style * update CHANGELOG * fix unit tests for python 3.8 * fix typo * log warning if workspace doesn't exist rather than error * remove a function that was added by accident in merge from develop * fix import issue * reorganize redis_utils as they're no longer needed * move serialize and deserialize to common utils * add base store and move save, retrieve, delete methods up to ResultsBackend * initial work on adding sqlite support for local runs * finish implementing sqlite stores * update docstrings for the backend package * move existing tests of redis/results backend after refactor * add unit tests for sqlite modules and move around fixtures * add tests for StoreBase * fix unit tests for configfile * remove duplicate fixture * add some docs about sqlite and move database docs out of monitoring section * first attempt at fixing port issue * fix lint issues * second attempt at fixing the tests and style issues * implement Charles comments * update CHANGELOG * Update copyright (#510) * add COPYRIGHT file, update LICENSE, and change copyright header in all files * update Makefile commands related to copyright/copyright headers * Update CHANGELOG * add a check for the copyright header in 'make check-style' * update CHANGELOG * release/1.13.0b1 (#511) * Potential fix for code scanning alert no. 23: Workflow does not contain permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * codeql-fixes (#513) * fix logging security vulnerabilities * update CHANGELOG * feature/disable-auto-restart (#514) * add no-restart option * run fix-style * update docstrings * add documentation about the no-restart option * update CHANGELOG * add Luc's suggestions * refactor/main-module (#515) * move each command's entry point logic to their own files * add missing docstrings * add copyright to the top of all new files * fix lint issues * rename backend utils test file * begin work on adding unit tests to the new cli folder * add common create_parser fixture * add tests for example and info * fix docstrings and move conftest file up a directory * add unit tests for 3 more commands * add tests for queue-info, restart, and run-workers * add tests for the remaining commands * remove parser fixtures when only one test uses it * run fix-style * add the no_restart functionality that got wiped from the merge * update CHANGELOG * feature/monitor-tests (#516) * split while loop inside monitor into smaller pieces for testing * add tests for task_server_monitor and celery_monitor * fix typo in list comprehension that broke a test * add tests for the Monitor class and MonitorFactory * update CHANGELOG * run fix-style and add missing docstrings/copyright headers * rename test_monitor to test_monitor_entry_point * release/1.13.0b2 (#517) --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning Co-authored-by: Joe Koning Co-authored-by: Jane Herriman Co-authored-by: Luc Peterson Co-authored-by: Ryan Lee Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> Co-authored-by: Wout De Nolf Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> --- CHANGELOG.md | 8 + docs/user_guide/command_line.md | 7 + .../monitoring/monitor_for_allocation.md | 2 +- merlin/__init__.py | 2 +- merlin/cli/__init__.py | 25 + merlin/cli/argparse_main.py | 72 + merlin/cli/commands/__init__.py | 66 + merlin/cli/commands/command_entry_point.py | 36 + merlin/cli/commands/config.py | 209 ++ merlin/cli/commands/database.py | 322 ++++ merlin/cli/commands/example.py | 84 + merlin/cli/commands/info.py | 59 + merlin/cli/commands/monitor.py | 127 ++ merlin/cli/commands/purge.py | 104 + merlin/cli/commands/query_workers.py | 99 + merlin/cli/commands/queue_info.py | 151 ++ merlin/cli/commands/restart.py | 102 + merlin/cli/commands/run.py | 182 ++ merlin/cli/commands/run_workers.py | 137 ++ merlin/cli/commands/server.py | 224 +++ merlin/cli/commands/status.py | 260 +++ merlin/cli/commands/stop_workers.py | 99 + merlin/cli/utils.py | 110 ++ merlin/main.py | 1689 +---------------- merlin/monitor/monitor.py | 86 +- .../{test_utils.py => test_backend_utils.py} | 0 .../cli/commands/test_command_entry_point.py | 58 + tests/unit/cli/commands/test_config.py | 152 ++ tests/unit/cli/commands/test_database.py | 157 ++ tests/unit/cli/commands/test_example.py | 70 + tests/unit/cli/commands/test_info.py | 44 + .../cli/commands/test_monitor_entry_point.py | 102 + tests/unit/cli/commands/test_purge.py | 96 + tests/unit/cli/commands/test_query_workers.py | 120 ++ tests/unit/cli/commands/test_queue_info.py | 148 ++ tests/unit/cli/commands/test_restart.py | 127 ++ tests/unit/cli/commands/test_run.py | 159 ++ tests/unit/cli/commands/test_run_workers.py | 106 ++ tests/unit/cli/commands/test_server.py | 132 ++ tests/unit/cli/commands/test_status.py | 166 ++ tests/unit/cli/commands/test_stop_workers.py | 102 + tests/unit/cli/conftest.py | 56 + tests/unit/cli/test_argparse_main.py | 103 + tests/unit/cli/test_cli_utils.py | 92 + tests/unit/monitor/test_celery_monitor.py | 196 ++ tests/unit/monitor/test_monitor.py | 227 +++ tests/unit/monitor/test_monitor_factory.py | 63 + .../unit/monitor/test_task_server_monitor.py | 130 ++ 48 files changed, 5157 insertions(+), 1711 deletions(-) create mode 100644 merlin/cli/__init__.py create mode 100644 merlin/cli/argparse_main.py create mode 100644 merlin/cli/commands/__init__.py create mode 100644 merlin/cli/commands/command_entry_point.py create mode 100644 merlin/cli/commands/config.py create mode 100644 merlin/cli/commands/database.py create mode 100644 merlin/cli/commands/example.py create mode 100644 merlin/cli/commands/info.py create mode 100644 merlin/cli/commands/monitor.py create mode 100644 merlin/cli/commands/purge.py create mode 100644 merlin/cli/commands/query_workers.py create mode 100644 merlin/cli/commands/queue_info.py create mode 100644 merlin/cli/commands/restart.py create mode 100644 merlin/cli/commands/run.py create mode 100644 merlin/cli/commands/run_workers.py create mode 100644 merlin/cli/commands/server.py create mode 100644 merlin/cli/commands/status.py create mode 100644 merlin/cli/commands/stop_workers.py create mode 100644 merlin/cli/utils.py rename tests/unit/backends/{test_utils.py => test_backend_utils.py} (100%) create mode 100644 tests/unit/cli/commands/test_command_entry_point.py create mode 100644 tests/unit/cli/commands/test_config.py create mode 100644 tests/unit/cli/commands/test_database.py create mode 100644 tests/unit/cli/commands/test_example.py create mode 100644 tests/unit/cli/commands/test_info.py create mode 100644 tests/unit/cli/commands/test_monitor_entry_point.py create mode 100644 tests/unit/cli/commands/test_purge.py create mode 100644 tests/unit/cli/commands/test_query_workers.py create mode 100644 tests/unit/cli/commands/test_queue_info.py create mode 100644 tests/unit/cli/commands/test_restart.py create mode 100644 tests/unit/cli/commands/test_run.py create mode 100644 tests/unit/cli/commands/test_run_workers.py create mode 100644 tests/unit/cli/commands/test_server.py create mode 100644 tests/unit/cli/commands/test_status.py create mode 100644 tests/unit/cli/commands/test_stop_workers.py create mode 100644 tests/unit/cli/conftest.py create mode 100644 tests/unit/cli/test_argparse_main.py create mode 100644 tests/unit/cli/test_cli_utils.py create mode 100644 tests/unit/monitor/test_celery_monitor.py create mode 100644 tests/unit/monitor/test_monitor.py create mode 100644 tests/unit/monitor/test_monitor_factory.py create mode 100644 tests/unit/monitor/test_task_server_monitor.py diff --git a/CHANGELOG.md b/CHANGELOG.md index f2ee0eccb..bf6611cf0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,14 @@ All notable changes to Merlin will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [1.13.0b2] +### Added +- Ability to turn off the auto-restart functionality of the monitor with `--no-restart` +- Tests for the monitor files + +### Changed +- Refactored the `main.py` module so that it's broken into smaller, more-manageable pieces + ## [1.13.0b1] ### Added - API documentation for Merlin's core codebase diff --git a/docs/user_guide/command_line.md b/docs/user_guide/command_line.md index 8072372cf..6cb41e512 100644 --- a/docs/user_guide/command_line.md +++ b/docs/user_guide/command_line.md @@ -1251,8 +1251,14 @@ There are multiple options to modify the way task statuses are displayed. ### Monitor (`merlin monitor`) +!!! tip "Tip When Running Multiple Monitor Instances" + + When having multiple instances of `merlin monitor` running for the same study, we recommend turning off the restart functionality with `--no-restart` for all except one of them, so as to avoid potential race conditions and trying to restart the workflow multiple times at once. + Batch submission scripts may not keep the batch allocation alive if there is not a blocking process in the submission script. The `merlin monitor` command addresses this by providing a blocking process that checks for tasks in the queues every (sleep) seconds ("sleep" here can be defined with the `--sleep` option). When the queues are empty, the monitor will query Celery to see if any workers are still processing tasks from the queues. If no workers are processing any tasks from the queues and the queues are empty, the blocking process will exit and allow the allocation to end. +If for some reason your workflow enters a stalled state where the queues are empty, no workers are processing tasks, but your workflow has not yet finished, then the monitor will attempt to automatically restart the workflow for you. This functionality can be disabled with the `--no-restart` option. + The `monitor` functionality will check for Celery workers for up to 10*(sleep) seconds before monitoring begins. The loop happens when the queue(s) in the spec contain tasks, but no running workers are detected. This is to protect against a failed worker launch. For more information, see the [Monitoring Studies for Persistent Allocations documentation](./monitoring/monitor_for_allocation.md). @@ -1272,6 +1278,7 @@ merlin monitor [OPTIONS] SPECIFICATION | `--vars` | List[string] | A space-delimited list of variables to override in the spec file. This list should be given after the spec file is provided. Ex: `--vars SIMWORKER=new_sim_worker` | None | | `--sleep` | integer | The duration in seconds between checks for workers/tasks | 60 | | `--task_server` | string | Task server type for which to monitor the workers. Currently only "celery" is implemented. | "celery" | +| `--no-restart`, `-n` | boolean | Disable the automatic restart functionality for this monitor. | `False` | !!! example "Basic Monitor" diff --git a/docs/user_guide/monitoring/monitor_for_allocation.md b/docs/user_guide/monitoring/monitor_for_allocation.md index 0c2fcc435..984ee4750 100644 --- a/docs/user_guide/monitoring/monitor_for_allocation.md +++ b/docs/user_guide/monitoring/monitor_for_allocation.md @@ -18,7 +18,7 @@ For each run of a study, the monitor ensures completion by performing the follow 1. Verifying the presence of tasks in the designated queues. 2. Confirming the ongoing processing of tasks by the assigned workers when the queues are empty. -3. Restarting the study if there are no tasks in the queues and no workers processing tasks, but the workflow has not yet finished. +3. Restarting the study if there are no tasks in the queues and no workers processing tasks, but the workflow has not yet finished. *This can be disabled using the `--no-restart` option.* The monitor includes a [`--sleep` option](#sleep), which introduces a deliberate delay. Before starting, the monitor waits for the specified `--sleep` duration, giving users time to populate the task queues for their run using the [`merlin run`](../command_line.md#run-merlin-run) command. Additionally, the monitor pauses for the `--sleep` duration between each check of the run. Finally, it will wait up to 10 times the specified `--sleep` duration for workers to spin up for the run. diff --git a/merlin/__init__.py b/merlin/__init__.py index c4c11fe8b..fb5cf4720 100644 --- a/merlin/__init__.py +++ b/merlin/__init__.py @@ -14,7 +14,7 @@ import sys -__version__ = "1.13.0b1" +__version__ = "1.13.0b2" VERSION = __version__ PATH_TO_PROJ = os.path.join(os.path.dirname(__file__), "") diff --git a/merlin/cli/__init__.py b/merlin/cli/__init__.py new file mode 100644 index 000000000..31870c55b --- /dev/null +++ b/merlin/cli/__init__.py @@ -0,0 +1,25 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Merlin CLI Package. + +This package defines the core components and supporting utilities for the +Merlin command-line interface (CLI). It provides the entry point parser for +the `merlin` CLI tool, a suite of modular subcommands for workflow and +infrastructure management, and shared helper functions used across CLI +handlers. + +Subpackages: + commands: Contains all command implementations for the Merlin CLI, including + workflow execution, monitoring, database interaction, and more. + +Modules: + argparse_main: Sets up the top-level argument parser and integrates all + registered CLI subcommands into the `merlin` CLI interface. + utils: Provides shared utility functions for parsing arguments, loading + YAML specifications, and handling configuration logic across commands. +""" diff --git a/merlin/cli/argparse_main.py b/merlin/cli/argparse_main.py new file mode 100644 index 000000000..1a5377d77 --- /dev/null +++ b/merlin/cli/argparse_main.py @@ -0,0 +1,72 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Main CLI parser setup for the Merlin command-line interface. + +This module defines the primary argument parser for the `merlin` CLI tool, +including custom error handling and integration of all available subcommands. +""" + +import sys +from argparse import ArgumentParser, RawDescriptionHelpFormatter + +from merlin import VERSION +from merlin.ascii_art import banner_small +from merlin.cli.commands import ALL_COMMANDS + + +DEFAULT_LOG_LEVEL = "INFO" + + +class HelpParser(ArgumentParser): + """ + This class overrides the error message of the argument parser to + print the help message when an error happens. + + Methods: + error: Override the error message of the `ArgumentParser` class. + """ + + def error(self, message: str): + """ + Override the error message of the `ArgumentParser` class. + + Args: + message: The error message to log. + """ + sys.stderr.write(f"error: {message}\n") + self.print_help() + sys.exit(2) + + +def build_main_parser() -> ArgumentParser: + """ + Set up the command-line argument parser for the Merlin package. + + Returns: + An `ArgumentParser` object with every parser defined in Merlin's codebase. + """ + parser = HelpParser( + prog="merlin", + description=banner_small, + formatter_class=RawDescriptionHelpFormatter, + epilog="See merlin --help for more info", + ) + parser.add_argument("-v", "--version", action="version", version=VERSION) + parser.add_argument( + "-lvl", + "--level", + type=str, + default=DEFAULT_LOG_LEVEL, + help="Set log level: DEBUG, INFO, WARNING, ERROR [Default: %(default)s]", + ) + subparsers = parser.add_subparsers(dest="subparsers", required=True) + + for command in ALL_COMMANDS: + command.add_parser(subparsers) + + return parser diff --git a/merlin/cli/commands/__init__.py b/merlin/cli/commands/__init__.py new file mode 100644 index 000000000..65d21454d --- /dev/null +++ b/merlin/cli/commands/__init__.py @@ -0,0 +1,66 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Merlin CLI Commands Package. + +This package defines all top-level and subcommand implementations for the Merlin +command-line interface. Each module encapsulates the logic and argument parsing +for a distinct Merlin command, following a consistent structure built around the +`CommandEntryPoint` interface. + +Modules: + command_entry_point: Defines the abstract base class `CommandEntryPoint` for all CLI commands. + config: Implements the `config` command for managing Merlin configuration files. + database: Implements the `database` command for interacting with the underlying database (view, delete, inspect). + example: Implements the `example` command to download and set up example workflows. + info: Implements the `info` command for displaying configuration and environment diagnostics. + monitor: Implements the `monitor` command to keep workflow allocations alive. + purge: Implements the `purge` command for removing tasks from queues. + query_workers: Implements the `query-workers` command for inspecting active task server workers. + queue_info: Implements the `queue-info` command for querying task server queue statistics. + restart: Implements the `restart` command to resume a workflow from a previous state. + run_workers: Implements the `run-workers` command to launch task-executing workers. + run: Implements the `run` command to execute Merlin or Maestro workflows. + server: Implements the `server` command to manage containerized Redis server components. + status: Implements the `status` and `detailed-status` commands for workflow state inspection. + stop_workers: Implements the `stop-workers` command for terminating active workers. +""" + +from merlin.cli.commands.config import ConfigCommand +from merlin.cli.commands.database import DatabaseCommand +from merlin.cli.commands.example import ExampleCommand +from merlin.cli.commands.info import InfoCommand +from merlin.cli.commands.monitor import MonitorCommand +from merlin.cli.commands.purge import PurgeCommand +from merlin.cli.commands.query_workers import QueryWorkersCommand +from merlin.cli.commands.queue_info import QueueInfoCommand +from merlin.cli.commands.restart import RestartCommand +from merlin.cli.commands.run import RunCommand +from merlin.cli.commands.run_workers import RunWorkersCommand +from merlin.cli.commands.server import ServerCommand +from merlin.cli.commands.status import DetailedStatusCommand, StatusCommand +from merlin.cli.commands.stop_workers import StopWorkersCommand + + +# Keep these in alphabetical order +ALL_COMMANDS = [ + ConfigCommand(), + DatabaseCommand(), + DetailedStatusCommand(), + ExampleCommand(), + InfoCommand(), + MonitorCommand(), + PurgeCommand(), + QueryWorkersCommand(), + QueueInfoCommand(), + RestartCommand(), + RunCommand(), + RunWorkersCommand(), + ServerCommand(), + StatusCommand(), + StopWorkersCommand(), +] diff --git a/merlin/cli/commands/command_entry_point.py b/merlin/cli/commands/command_entry_point.py new file mode 100644 index 000000000..aaad695a3 --- /dev/null +++ b/merlin/cli/commands/command_entry_point.py @@ -0,0 +1,36 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Defines the abstract base class for Merlin CLI commands. + +This module provides the `CommandEntryPoint` abstract base class that all +Merlin command implementations must inherit from. It standardizes the interface +for adding command-specific argument parsers and processing CLI command logic. +""" + +from abc import ABC, abstractmethod +from argparse import ArgumentParser, Namespace + + +class CommandEntryPoint(ABC): + """ + Abstract base class for a Merlin CLI command entry point. + + Methods: + add_parser: Adds the parser for a specific command to the main `ArgumentParser`. + process_command: Executes the logic for this CLI command. + """ + + @abstractmethod + def add_parser(self, subparsers: ArgumentParser): + """Add the parser for this command to the main `ArgumentParser`.""" + raise NotImplementedError("Subclasses of `CommandEntryPoint` must implement an `add_parser` method.") + + @abstractmethod + def process_command(self, args: Namespace): + """Execute the logic for this CLI command.""" + raise NotImplementedError("Subclasses of `CommandEntryPoint` must implement an `process_command` method.") diff --git a/merlin/cli/commands/config.py b/merlin/cli/commands/config.py new file mode 100644 index 000000000..07015b5cd --- /dev/null +++ b/merlin/cli/commands/config.py @@ -0,0 +1,209 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +CLI command for managing Merlin configuration files. + +This module defines the `ConfigCommand` class, which provides the CLI interface +to create, update, and switch between different Merlin configuration files. +These configurations control task server, broker, and backend settings used +by the Merlin workflow system. +""" + +# pylint: disable=duplicate-code + +import logging +import os +from argparse import SUPPRESS, ArgumentDefaultsHelpFormatter, ArgumentParser, ArgumentTypeError, Namespace + +import yaml + +from merlin.cli.commands.command_entry_point import CommandEntryPoint +from merlin.config.merlin_config_manager import MerlinConfigManager + + +LOG = logging.getLogger("merlin") + + +class ConfigCommand(CommandEntryPoint): + """ + CLI command group for managing Merlin configuration files. + + This class adds the `config` command and its subcommands to the Merlin CLI, + and dispatches the CLI input to Merlin's codebase. + + Attributes: + default_config_file (str): The default path to the Merlin config file (`~/.merlin/app.yaml`). + + Methods: + add_parser: Adds the `config` command and its subcommands to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + default_config_file = os.path.join(os.path.expanduser("~"), ".merlin", "app.yaml") + + def _add_create_subcommand(self, mconfig_subparsers: ArgumentParser): + """ + Add the `create` subcommand to generate a new configuration file. + + Parameters: + mconfig_subparsers (ArgumentParser): The subparsers object to add the subcommand to. + """ + config_create_parser = mconfig_subparsers.add_parser("create", help="Create a new configuration file.") + config_create_parser.add_argument( + "--task-server", + type=str, + default="celery", + help="Task server type for which to create the config. Default: %(default)s", + ) + config_create_parser.add_argument( + "-o", + "--output-file", + dest="config_file", + type=str, + default=self.default_config_file, + help=f"Optional file name for your configuration. Default: {self.default_config_file}", + ) + config_create_parser.add_argument( + "--broker", + type=str, + default=None, + help="Optional broker type, backend will be redis. Default: rabbitmq", + ) + + def _add_update_broker_subcommand(self, mconfig_subparsers: ArgumentParser): + """ + Add the `update-broker` subcommand to modify broker-related settings. + + Parameters: + mconfig_subparsers (ArgumentParser): The subparsers object to add the subcommand to. + """ + config_broker_parser = mconfig_subparsers.add_parser("update-broker", help="Update broker settings in app.yaml") + config_broker_parser.add_argument( + "-t", + "--type", + required=True, + choices=["redis", "rabbitmq"], + help="Type of broker to configure (redis or rabbitmq).", + ) + config_broker_parser.add_argument( + "--cf", + "--config-file", + dest="config_file", + default=self.default_config_file, + help=f"The path to the config file that will be updated. Default: {self.default_config_file}", + ) + config_broker_parser.add_argument("-u", "--username", help="Broker username (only for rabbitmq)") + config_broker_parser.add_argument("--pf", "--password-file", dest="password_file", help="Path to password file") + config_broker_parser.add_argument("-s", "--server", help="The URL of the server") + config_broker_parser.add_argument("-p", "--port", type=int, help="Broker port") + config_broker_parser.add_argument("-v", "--vhost", help="Broker vhost (only for rabbitmq)") + config_broker_parser.add_argument("-c", "--cert-reqs", help="Broker cert requirements") + config_broker_parser.add_argument("-d", "--db-num", type=int, help="Redis database number (only for redis).") + + def _add_update_backend_subcommand(self, mconfig_subparsers: ArgumentParser): + """ + Add the `update-backend` subcommand to modify results backend settings. + + Parameters: + mconfig_subparsers (ArgumentParser): The subparsers object to add the subcommand to. + """ + config_backend_parser = mconfig_subparsers.add_parser( + "update-backend", help="Update results backend settings in app.yaml" + ) + config_backend_parser.add_argument( + "-t", + "--type", + required=True, + choices=["redis"], + help="Type of results backend to configure.", + ) + config_backend_parser.add_argument( + "--cf", + "--config-file", + dest="config_file", + default=self.default_config_file, + help=f"The path to the config file that will be updated. Default: {self.default_config_file}", + ) + config_backend_parser.add_argument("-u", "--username", help="Backend username") + config_backend_parser.add_argument("--pf", "--password-file", dest="password_file", help="Path to password file") + config_backend_parser.add_argument("-s", "--server", help="The URL of the server") + config_backend_parser.add_argument("-p", "--port", help="Backend port") + config_backend_parser.add_argument("-d", "--db-num", help="Backend database number") + config_backend_parser.add_argument("-c", "--cert-reqs", help="Backend cert requirements") + config_backend_parser.add_argument("-e", "--encryption-key", help="Path to encryption key file") + + def _add_use_subcommand(self, mconfig_subparsers: ArgumentParser): + """ + Add the `use` subcommand to switch to a different Merlin config file. + + Parameters: + mconfig_subparsers (ArgumentParser): The subparsers object to add the subcommand to. + """ + config_use_parser = mconfig_subparsers.add_parser("use", help="Use a different configuration file.") + config_use_parser.add_argument("config_file", type=str, help="The path to the new configuration file to use.") + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `config` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `config` command parser will be added. + """ + mconfig: ArgumentParser = subparsers.add_parser( + "config", + help="Create a default merlin server config file in ~/.merlin", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + mconfig.set_defaults(func=self.process_command) + # The below option makes it so the `config_path.txt` file is written to the test directory + mconfig.add_argument( + "-t", + "--test", + action="store_true", + help=SUPPRESS, # Hides from `--help` + ) + mconfig_subparsers = mconfig.add_subparsers(dest="commands", help="Subcommands for 'config'") + + self._add_create_subcommand(mconfig_subparsers) + self._add_update_broker_subcommand(mconfig_subparsers) + self._add_update_backend_subcommand(mconfig_subparsers) + self._add_use_subcommand(mconfig_subparsers) + + def process_command(self, args: Namespace): + """ + CLI command to manage Merlin configuration files. + + This function handles various configuration-related operations based on + the provided subcommand. It ensures that the specified configuration + file has a valid YAML extension (i.e., `.yaml` or `.yml`). + + If no output file is explicitly provided, a default path is used. + + Args: + args (Namespace): Parsed command-line arguments. + """ + if args.commands != "create": # Check that this is a valid yaml file + try: + with open(args.config_file, "r") as conf_file: + yaml.safe_load(conf_file) + except FileNotFoundError as fnf_exc: + raise ArgumentTypeError(f"The file '{args.config_file}' does not exist.") from fnf_exc + except yaml.YAMLError as yaml_exc: + raise ArgumentTypeError(f"The file '{args.config_file}' is not a valid YAML file.") from yaml_exc + + config_manager = MerlinConfigManager(args) + + if args.commands == "create": + config_manager.create_template_config() + config_manager.save_config_path() + elif args.commands == "update-broker": + config_manager.update_broker() + elif args.commands == "update-backend": + config_manager.update_backend() + elif args.commands == "use": # Config file path is updated in constructor of MerlinConfigManager + config_manager.config_file = args.config_file + config_manager.save_config_path() diff --git a/merlin/cli/commands/database.py b/merlin/cli/commands/database.py new file mode 100644 index 000000000..94db4d08c --- /dev/null +++ b/merlin/cli/commands/database.py @@ -0,0 +1,322 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +This module defines the `DatabaseCommand` class, which provides CLI subcommands +for interacting with the Merlin application's underlying database. It supports +commands for retrieving, deleting, and inspecting database contents, including +entities like studies, runs, and workers. + +The commands are registered under the `database` top-level command and integrated +into Merlin's argument parser system. +""" + +# pylint: disable=duplicate-code + +import logging +from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, Namespace + +from merlin.cli.commands.command_entry_point import CommandEntryPoint +from merlin.config.configfile import initialize_config +from merlin.db_scripts.db_commands import database_delete, database_get, database_info + + +LOG = logging.getLogger("merlin") + + +class DatabaseCommand(CommandEntryPoint): + """ + Handles `database` CLI commands for interacting with Merlin's database. + + Methods: + add_parser: Adds the `database` command and its subcommands to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + def _add_delete_subcommand(self, database_commands: ArgumentParser): + """ + Add the `delete` subcommand and its options to remove data from the database. + + Parameters: + database_commands (ArgumentParser): The parent parser for database subcommands. + """ + db_delete: ArgumentParser = database_commands.add_parser( + "delete", + help="Delete information stored in the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Add subcommands for delete + delete_subcommands = db_delete.add_subparsers(dest="delete_type", required=True) + + # TODO enable support for deletion of study by passing in spec file + # Subcommand: delete study + delete_study = delete_subcommands.add_parser( + "study", + help="Delete one or more studies by ID or name.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + delete_study.add_argument( + "study", + type=str, + nargs="+", + help="A space-delimited list of IDs or names of studies to delete.", + ) + delete_study.add_argument( + "-k", + "--keep-associated-runs", + action="store_true", + help="Keep runs associated with the studies.", + ) + + # Subcommand: delete run + delete_run = delete_subcommands.add_parser( + "run", + help="Delete one or more runs by ID or workspace.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + delete_run.add_argument( + "run", + type=str, + nargs="+", + help="A space-delimited list of IDs or workspaces of runs to delete.", + ) + # TODO implement the below option; this removes the output workspace from file system + # delete_run.add_argument( + # "--delete-workspace", + # action="store_true", + # help="Delete the output workspace for the run.", + # ) + + # Subcommand: delete logical-worker + delete_logical_worker = delete_subcommands.add_parser( + "logical-worker", + help="Delete one or more logical workers by ID.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + delete_logical_worker.add_argument( + "worker", + type=str, + nargs="+", + help="A space-delimited list of IDs of logical workers to delete.", + ) + + # Subcommand: delete physical-worker + delete_physical_worker = delete_subcommands.add_parser( + "physical-worker", + help="Delete one or more physical workers by ID or name.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + delete_physical_worker.add_argument( + "worker", + type=str, + nargs="+", + help="A space-delimited list of IDs of physical workers to delete.", + ) + + # Subcommand: delete all-studies + delete_all_studies = delete_subcommands.add_parser( + "all-studies", + help="Delete all studies from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + delete_all_studies.add_argument( + "-k", + "--keep-associated-runs", + action="store_true", + help="Keep runs associated with the studies.", + ) + + # Subcommand: delete all-runs + delete_subcommands.add_parser( + "all-runs", + help="Delete all runs from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: delete all-logical-workers + delete_subcommands.add_parser( + "all-logical-workers", + help="Delete all logical workers from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: delete all-physical-workers + delete_subcommands.add_parser( + "all-physical-workers", + help="Delete all physical workers from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: delete everything + delete_everything = delete_subcommands.add_parser( + "everything", + help="Delete everything from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + delete_everything.add_argument( + "-f", + "--force", + action="store_true", + help="Delete everything in the database without confirmation.", + ) + + def _add_get_subcommand(self, database_commands: ArgumentParser): + """ + Add the `get` subcommand and its options to retrieve data from the database. + + Parameters: + database_commands (ArgumentParser): The parent parser for database subcommands. + """ + db_get: ArgumentParser = database_commands.add_parser( + "get", + help="Get information stored in the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Add subcommands for get + get_subcommands = db_get.add_subparsers(dest="get_type", required=True) + + # TODO enable support for retrieval of study by passing in spec file + # Subcommand: get study + get_study = get_subcommands.add_parser( + "study", + help="Get one or more studies by ID or name.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + get_study.add_argument( + "study", + type=str, + nargs="+", + help="A space-delimited list of IDs or names of the studies to get.", + ) + + # Subcommand: get run + get_run = get_subcommands.add_parser( + "run", + help="Get one or more runs by ID or workspace.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + get_run.add_argument( + "run", + type=str, + nargs="+", + help="A space-delimited list of IDs or workspaces of the runs to get.", + ) + + # Subcommand get logical-worker + get_logical_worker = get_subcommands.add_parser( + "logical-worker", + help="Get one or more logical workers by ID.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + get_logical_worker.add_argument( + "worker", + type=str, + nargs="+", + help="A space-delimited list of IDs of the logical workers to get.", + ) + + # Subcommand get physical-worker + get_physical_worker = get_subcommands.add_parser( + "physical-worker", + help="Get one or more physical workers by ID or name.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + get_physical_worker.add_argument( + "worker", + type=str, + nargs="+", + help="A space-delimited list of IDs or names of the physical workers to get.", + ) + + # Subcommand: get all-studies + get_subcommands.add_parser( + "all-studies", + help="Get all studies from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: get all-runs + get_subcommands.add_parser( + "all-runs", + help="Get all runs from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: get all-logical-workers + get_subcommands.add_parser( + "all-logical-workers", + help="Get all logical workers from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: get all-physical-workers + get_subcommands.add_parser( + "all-physical-workers", + help="Get all physical workers from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: get everything + get_subcommands.add_parser( + "everything", + help="Get everything from the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `database` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `database` command parser will be added. + """ + database: ArgumentParser = subparsers.add_parser( + "database", + help="Interact with Merlin's database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + database.set_defaults(func=self.process_command) + + database.add_argument( + "-l", + "--local", + action="store_true", + help="Use the local SQLite database for this command.", + ) + + database_commands: ArgumentParser = database.add_subparsers(dest="commands") + + # Subcommand: database info + database_commands.add_parser( + "info", + help="Print information about the database.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # Subcommand: database delete + self._add_delete_subcommand(database_commands) + + # Subcommand: database get + self._add_get_subcommand(database_commands) + + def process_command(self, args: Namespace): + """ + Process database commands by routing to the correct function. + + Args: + args: An argparse Namespace containing user arguments. + """ + if args.local: + initialize_config(local_mode=True) + + if args.commands == "info": + database_info() + elif args.commands == "get": + database_get(args) + elif args.commands == "delete": + database_delete(args) diff --git a/merlin/cli/commands/example.py b/merlin/cli/commands/example.py new file mode 100644 index 000000000..4245433c9 --- /dev/null +++ b/merlin/cli/commands/example.py @@ -0,0 +1,84 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Merlin CLI `example` command module. + +This module defines the `ExampleCommand` class, which integrates into the Merlin +command-line interface to support downloading and setting up example workflows. +""" + +# pylint: disable=duplicate-code + +import logging +from argparse import ArgumentParser, Namespace, RawTextHelpFormatter + +from merlin.ascii_art import banner_small +from merlin.cli.commands.command_entry_point import CommandEntryPoint +from merlin.examples.generator import list_examples, setup_example + + +LOG = logging.getLogger("merlin") + + +class ExampleCommand(CommandEntryPoint): + """ + Handles `example` CLI command for downloading built-in Merlin examples. + + Methods: + add_parser: Adds the `example` command to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `example` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `example` command parser will be added. + """ + example: ArgumentParser = subparsers.add_parser( + "example", + help="Generate an example merlin workflow.", + formatter_class=RawTextHelpFormatter, + ) + example.set_defaults(func=self.process_command) + example.add_argument( + "workflow", + action="store", + type=str, + help="The name of the example workflow to setup. Use 'merlin example list' to see available options.", + ) + example.add_argument( + "-p", + "--path", + action="store", + type=str, + default=None, + help="Specify a path to write the workflow to. Defaults to current working directory", + ) + + def process_command(self, args: Namespace): + """ + CLI command to set up or list Merlin example workflows. + + This function either lists all available example workflows or sets + up a specified example workflow to be run in the root directory. The + behavior is determined by the `workflow` argument. + + Args: + args: Parsed command-line arguments, which may include:\n + - `workflow`: The action to perform; should be "list" + to display all examples or the name of a specific example + workflow to set up. + - `path`: The directory where the example workflow + should be set up. Only applicable when `workflow` is not "list". + """ + if args.workflow == "list": + print(list_examples()) + else: + print(banner_small) + setup_example(args.workflow, args.path) diff --git a/merlin/cli/commands/info.py b/merlin/cli/commands/info.py new file mode 100644 index 000000000..79aaf8cb6 --- /dev/null +++ b/merlin/cli/commands/info.py @@ -0,0 +1,59 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +CLI module for displaying configuration and environment information. + +This module defines the `InfoCommand` class, which handles the `info` subcommand +of the Merlin CLI. The `info` command is intended to display detailed information +about the current Merlin configuration, Python environment, and other diagnostic +data useful for debugging or verifying setup. +""" + +# pylint: disable=duplicate-code + +import logging +from argparse import ArgumentParser, Namespace + +from merlin.cli.commands.command_entry_point import CommandEntryPoint + + +LOG = logging.getLogger("merlin") + + +class InfoCommand(CommandEntryPoint): + """ + Handles `info` CLI command for viewing information about server connections. + + Methods: + add_parser: Adds the `info` command to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `info` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `info` command parser will be added. + """ + info: ArgumentParser = subparsers.add_parser( + "info", + help="display info about the merlin configuration and the python configuration. Useful for debugging.", + ) + info.set_defaults(func=self.process_command) + + def process_command(self, args: Namespace): + """ + CLI command to print merlin configuration info. + + Args: + args: Parsed CLI arguments. + """ + # if this is moved to the toplevel per standard style, merlin is unable to generate the (needed) default config file + from merlin import display # pylint: disable=import-outside-toplevel + + display.print_info(args) diff --git a/merlin/cli/commands/monitor.py b/merlin/cli/commands/monitor.py new file mode 100644 index 000000000..9909d5366 --- /dev/null +++ b/merlin/cli/commands/monitor.py @@ -0,0 +1,127 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +CLI module for monitoring Merlin workflows and maintaining active allocations. + +This module provides the `MonitorCommand` class, which implements the `monitor` +subcommand in the Merlin CLI. The purpose of the `monitor` command is to +periodically check the status of workflow tasks and worker activity to ensure +the allocation (e.g., on a computing cluster) remains alive while jobs are +in progress. +""" + +# pylint: disable=duplicate-code + +import logging +import time +from argparse import ArgumentParser, Namespace, RawTextHelpFormatter + +from merlin.cli.commands.command_entry_point import CommandEntryPoint +from merlin.cli.utils import get_merlin_spec_with_override +from merlin.monitor.monitor import Monitor +from merlin.router import check_merlin_status + + +LOG = logging.getLogger("merlin") + + +class MonitorCommand(CommandEntryPoint): + """ + Handles `monitor` CLI command for monitoring workflows to ensure the allocation remains alive. + + Methods: + add_parser: Adds the `monitor` command to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `monitor` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `monitor` command parser will be added. + """ + monitor: ArgumentParser = subparsers.add_parser( + "monitor", + help="Check for active workers on an allocation.", + formatter_class=RawTextHelpFormatter, + ) + monitor.set_defaults(func=self.process_command) + monitor.add_argument("specification", type=str, help="Path to a Merlin YAML spec file") + monitor.add_argument( + "--steps", + nargs="+", + type=str, + dest="steps", + default=["all"], + help="The specific steps (tasks on the server) in the YAML file defining the queues you want to monitor", + ) + monitor.add_argument( + "--vars", + action="store", + dest="variables", + type=str, + nargs="+", + default=None, + help="Specify desired Merlin variable values to override those found in the specification. Space-delimited. " + "Example: '--vars LEARN=path/to/new_learn.py EPOCHS=3'", + ) + monitor.add_argument( + "--task_server", + type=str, + default="celery", + help="Task server type for which to monitor the workers.\ + Default: %(default)s", + ) + monitor.add_argument( + "--sleep", + type=int, + default=60, + help="Sleep duration between checking for workers.\ + Default: %(default)s", + ) + monitor.add_argument( + "--no-restart", + "-n", + action="store_true", + help="Disable the automatic restart functionality for this monitor.", + ) + + def process_command(self, args: Namespace): + """ + CLI command to monitor Merlin workers and queues to maintain + allocation status. + + This function periodically checks the status of Merlin workers and + the associated queues to ensure that the allocation remains active. + It includes a sleep interval to wait before each check, including + the initial one. + + Args: + args: Parsed command-line arguments, which may include:\n + - `sleep`: The duration (in seconds) to wait before + checking the queue status again. + """ + spec, _ = get_merlin_spec_with_override(args) + + # Give the user time to queue up jobs in case they haven't already + time.sleep(args.sleep) + + if args.steps != ["all"]: + LOG.warning( + "The `--steps` argument of the `merlin monitor` command is set to be deprecated in Merlin v1.14 " + "For now, using this argument will tell merlin to use the version of the monitor command from Merlin v1.12." + ) + # Check if we still need our allocation + while check_merlin_status(args, spec): + LOG.info("Monitor: found tasks in queues and/or tasks being processed") + time.sleep(args.sleep) + else: + monitor = Monitor(spec, args.sleep, args.task_server, args.no_restart) + monitor.monitor_all_runs() + + LOG.info("Monitor: ... stop condition met") diff --git a/merlin/cli/commands/purge.py b/merlin/cli/commands/purge.py new file mode 100644 index 000000000..b090ea5a1 --- /dev/null +++ b/merlin/cli/commands/purge.py @@ -0,0 +1,104 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +CLI module for purging tasks from Merlin queues on the task server. + +This module defines the `PurgeCommand` class, which implements the `purge` +subcommand in the Merlin CLI. The command is used to remove tasks from +queues either entirely or selectively, based on the specified steps in +a Merlin YAML workflow specification. +""" + +# pylint: disable=duplicate-code + +import logging +from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, Namespace + +from merlin.ascii_art import banner_small +from merlin.cli.commands.command_entry_point import CommandEntryPoint +from merlin.cli.utils import get_merlin_spec_with_override +from merlin.router import purge_tasks + + +LOG = logging.getLogger("merlin") + + +class PurgeCommand(CommandEntryPoint): + """ + Handles `purge` CLI command for removing tasks from queues on the server. + + Methods: + add_parser: Adds the `purge` command to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `purge` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `purge` command parser will be added. + """ + purge: ArgumentParser = subparsers.add_parser( + "purge", + help="Remove all tasks from all merlin queues (default). " + "If a user would like to purge only selected queues use: " + "--steps to give a steplist, the queues will be defined from the step list", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + purge.set_defaults(func=self.process_command) + purge.add_argument("specification", type=str, help="Path to a Merlin YAML spec file") + purge.add_argument( + "-f", + "--force", + action="store_true", + dest="purge_force", + default=False, + help="Purge the tasks without confirmation", + ) + purge.add_argument( + "--steps", + nargs="+", + type=str, + dest="purge_steps", + default=["all"], + help="The specific steps in the YAML file from which you want to purge the queues. \ + The input is a space separated list.", + ) + purge.add_argument( # pylint: disable=duplicate-code + "--vars", + action="store", + dest="variables", + type=str, + nargs="+", + default=None, + help="Specify desired Merlin variable values to override those found in the specification. Space-delimited. " + "Example: '--vars MY_QUEUE=hello'", + ) + + def process_command(self, args: Namespace): + """ + CLI command for purging tasks from the task server. + + This function removes specified tasks from the task server based on the provided + Merlin specification. It allows for targeted purging or forced removal of tasks. + + Args: + args: Parsed CLI arguments containing:\n + - `purge_force`: If True, forces the purge operation without confirmation. + - `purge_steps`: Steps or criteria based on which tasks will be purged. + """ + print(banner_small) + spec, _ = get_merlin_spec_with_override(args) + ret = purge_tasks( + spec.merlin["resources"]["task_server"], + spec, + args.purge_force, + args.purge_steps, + ) + + LOG.info(f"Purge return = {ret} .") diff --git a/merlin/cli/commands/query_workers.py b/merlin/cli/commands/query_workers.py new file mode 100644 index 000000000..f09225eeb --- /dev/null +++ b/merlin/cli/commands/query_workers.py @@ -0,0 +1,99 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +CLI module for querying active Merlin task server workers. + +This module defines the `QueryWorkersCommand` class, which implements the +`query-workers` subcommand for the Merlin CLI. The command allows users to +inspect the state of connected workers on a task server (e.g., Celery), +optionally filtering by queues or worker names. +""" + +# pylint: disable=duplicate-code + +import logging +from argparse import ArgumentParser, Namespace + +from merlin.ascii_art import banner_small +from merlin.cli.commands.command_entry_point import CommandEntryPoint +from merlin.router import query_workers +from merlin.spec.specification import MerlinSpec +from merlin.utils import verify_filepath + + +LOG = logging.getLogger("merlin") + + +class QueryWorkersCommand(CommandEntryPoint): + """ + Handles `query-workers` CLI command for querying information about Merlin workers. + + Methods: + add_parser: Adds the `query-workers` command to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `query-workers` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `query-workers` command parser will be added. + """ + query: ArgumentParser = subparsers.add_parser("query-workers", help="List connected task server workers.") + query.set_defaults(func=self.process_command) + query.add_argument( + "--task_server", + type=str, + default="celery", + help="Task server type from which to query workers.\ + Default: %(default)s", + ) + query.add_argument( + "--spec", + type=str, + default=None, + help="Path to a Merlin YAML spec file from which to read worker names to query.", + ) + query.add_argument("--queues", type=str, default=None, nargs="+", help="Specific queues to query workers from.") + query.add_argument( + "--workers", + type=str, + action="store", + nargs="+", + default=None, + help="Regex match for specific workers to query.", + ) + + def process_command(self, args: Namespace): + """ + CLI command for finding all workers. + + This function retrieves and queries the names of any active workers. + If the `--spec` argument is included, only query the workers defined in the spec file. + + Args: + args: Parsed command-line arguments, which may include:\n + - `spec`: Path to the specification file. + - `task_server`: Address of the task server to query. + - `queues`: List of queue names to filter workers. + - `workers`: List of specific worker names to query. + """ + print(banner_small) + + # Get the workers from the spec file if --spec provided + worker_names = [] + if args.spec: + spec_path = verify_filepath(args.spec) + spec = MerlinSpec.load_specification(spec_path) + worker_names = spec.get_worker_names() + for worker_name in worker_names: + if "$" in worker_name: + LOG.warning(f"Worker '{worker_name}' is unexpanded. Target provenance spec instead?") + LOG.debug(f"Searching for the following workers to stop based on the spec {args.spec}: {worker_names}") + + query_workers(args.task_server, worker_names, args.queues, args.workers) diff --git a/merlin/cli/commands/queue_info.py b/merlin/cli/commands/queue_info.py new file mode 100644 index 000000000..c9972a89b --- /dev/null +++ b/merlin/cli/commands/queue_info.py @@ -0,0 +1,151 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +CLI module for inspecting task server queue statistics in Merlin workflows. + +This module defines the `QueueInfoCommand` class, which implements the +`queue-info` subcommand for the Merlin CLI. The command enables users to query +detailed information about queues used in Merlin workflows, including the number +of tasks in each queue and the number of connected workers. +""" + +# pylint: disable=duplicate-code + +import logging +from argparse import ArgumentParser, Namespace + +from tabulate import tabulate + +from merlin.ascii_art import banner_small +from merlin.cli.commands.command_entry_point import CommandEntryPoint +from merlin.cli.utils import get_merlin_spec_with_override +from merlin.router import dump_queue_info, query_queues + + +LOG = logging.getLogger("merlin") + + +class QueueInfoCommand(CommandEntryPoint): + """ + Handles `queue-info` CLI command for querying info about the queues on the servers. + + Methods: + add_parser: Adds the `queue-info` command to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `queue-info` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `queue-info` command parser will be added. + """ + queue_info: ArgumentParser = subparsers.add_parser( + "queue-info", + help="List queue statistics (queue name, number of tasks in the queue, number of connected workers).", + ) + queue_info.set_defaults(func=self.process_command) + queue_info.add_argument( + "--dump", + type=str, + help="Dump the queue information to a file. Provide the filename (must be .csv or .json)", + default=None, + ) + queue_info.add_argument( + "--specific-queues", nargs="+", type=str, help="Display queue stats for specific queues you list here" + ) + queue_info.add_argument( + "--task_server", + type=str, + default="celery", + help="Task server type. Default: %(default)s", + ) + spec_group = queue_info.add_argument_group("specification options") + spec_group.add_argument( + "--spec", + dest="specification", + type=str, + help="Path to a Merlin YAML spec file. \ + This will only display information for queues defined in this spec file. \ + This is the same behavior as the status command prior to Merlin version 1.11.0.", + ) + spec_group.add_argument( + "--steps", + nargs="+", + type=str, + dest="steps", + default=["all"], + help="The specific steps in the YAML file you want to query the queues of. " + "This option MUST be used with the --spec option", + ) + spec_group.add_argument( + "--vars", + action="store", + dest="variables", + type=str, + nargs="+", + default=None, + help="Specify desired Merlin variable values to override those found in the specification. Space-delimited. " + "This option MUST be used with the --spec option. Example: '--vars LEARN=path/to/new_learn.py EPOCHS=3'", + ) + + def process_command(self, args: Namespace): + """ + CLI command for finding all workers and their associated queues. + + This function processes the command-line arguments to retrieve and display + information about the available workers and their queues within the task server. + It validates the necessary parameters, handles potential file dumping, and + formats the output for easy readability. + + Args: + args: Parsed CLI arguments containing user inputs related to the query. + + Raises: + ValueError: + - If a specification is not provided when steps are specified and the + steps do not include "all". + - If variables are included without a corresponding specification. + - If the specified dump filename does not end with '.json' or '.csv'. + """ + print(banner_small) + + # Ensure a spec is provided if steps are provided + if not args.specification: + if "all" not in args.steps: + raise ValueError("The --steps argument MUST be used with the --specification argument.") + if args.variables: + raise ValueError("The --vars argument MUST be used with the --specification argument.") + + # Ensure a supported file type is provided with the dump option + if args.dump is not None: + if not args.dump.endswith(".json") and not args.dump.endswith(".csv"): + raise ValueError("Unsupported file type. Dump files must be either '.json' or '.csv'.") + + spec = None + # Load the spec if necessary + if args.specification: + spec, _ = get_merlin_spec_with_override(args) + + # Obtain the queue information + queue_information = query_queues(args.task_server, spec, args.steps, args.specific_queues) + + if queue_information: + # Format the queue information so we can pass it to the tabulate library + formatted_queue_info = [("Queue Name", "Task Count", "Worker Count")] + for queue_name, queue_stats in queue_information.items(): + formatted_queue_info.append((queue_name, queue_stats["jobs"], queue_stats["consumers"])) + + # Print the queue information + print() + print(tabulate(formatted_queue_info, headers="firstrow")) + print() + + # Dump queue information to an output file if necessary + if args.dump: + dump_queue_info(args.task_server, queue_information, args.dump) diff --git a/merlin/cli/commands/restart.py b/merlin/cli/commands/restart.py new file mode 100644 index 000000000..1d8e0dddb --- /dev/null +++ b/merlin/cli/commands/restart.py @@ -0,0 +1,102 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +CLI module for restarting existing Merlin workflows. + +This module defines the `RestartCommand` class, which provides functionality +to restart an existing Merlin workflow from a previously saved workspace. + +The command verifies the provided workspace directory, locates the appropriate +provenance specification file (an expanded YAML spec), and re-initializes the +workflow study from that point. It supports running the restarted workflow +either locally or in a distributed mode. +""" + +# pylint: disable=duplicate-code + +import glob +import logging +import os +from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, Namespace +from typing import List, Optional + +from merlin.ascii_art import banner_small +from merlin.cli.commands.command_entry_point import CommandEntryPoint +from merlin.config.configfile import initialize_config +from merlin.router import run_task_server +from merlin.study.study import MerlinStudy +from merlin.utils import verify_dirpath, verify_filepath + + +LOG = logging.getLogger("merlin") + + +class RestartCommand(CommandEntryPoint): + """ + Handles `restart` CLI command for restarting existing workflows. + + Methods: + add_parser: Adds the `restart` command to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `restart` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `restart` command parser will be added. + """ + restart: ArgumentParser = subparsers.add_parser( + "restart", + help="Restart a workflow using an existing Merlin workspace.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + restart.set_defaults(func=self.process_command) + restart.add_argument("restart_dir", type=str, help="Path to an existing Merlin workspace directory") + restart.add_argument( # TODO should this just be boolean instead of store_const? + "--local", + action="store_const", + dest="run_mode", + const="local", + default="distributed", + help="Run locally instead of distributed", + ) + + def process_command(self, args: Namespace): + """ + CLI command for restarting a study. + + This function handles the restart process by verifying the specified restart + directory, locating a valid provenance specification file, and initiating + the study from that point. + + Args: + args: Parsed CLI arguments containing:\n + - `restart_dir`: Path to the directory where the restart specifications are located. + - `run_mode`: The mode for running the study (e.g., normal, dry-run). + + Raises: + ValueError: If the `restart_dir` does not contain a valid provenance spec file or + if multiple files match the specified pattern. + """ + print(banner_small) + restart_dir: str = verify_dirpath(args.restart_dir) + filepath: str = os.path.join(args.restart_dir, "merlin_info", "*.expanded.yaml") + possible_specs: Optional[List[str]] = glob.glob(filepath) + if not possible_specs: # len == 0 + raise ValueError(f"'{filepath}' does not match any provenance spec file to restart from.") + if len(possible_specs) > 1: + raise ValueError(f"'{filepath}' matches more than one provenance spec file to restart from.") + filepath: str = verify_filepath(possible_specs[0]) + LOG.info(f"Restarting workflow at '{restart_dir}'") + study: MerlinStudy = MerlinStudy(filepath, restart_dir=restart_dir) + + if args.run_mode == "local": + initialize_config(local_mode=True) + + run_task_server(study, args.run_mode) diff --git a/merlin/cli/commands/run.py b/merlin/cli/commands/run.py new file mode 100644 index 000000000..c7b9e9275 --- /dev/null +++ b/merlin/cli/commands/run.py @@ -0,0 +1,182 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +CLI module for executing Merlin or Maestro workflows. + +This module defines the `RunCommand` class, which handles the `run` subcommand in the +Merlin CLI. The command initializes and runs a workflow based on a specified YAML +study specification file. It supports overriding variables, supplying samples files, +dry-run execution, and parallel parameter generation options. +""" + +# pylint: disable=duplicate-code + +from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, Namespace +from typing import Optional + +from merlin.ascii_art import banner_small +from merlin.cli.commands.command_entry_point import CommandEntryPoint +from merlin.cli.utils import parse_override_vars +from merlin.config.configfile import initialize_config +from merlin.db_scripts.merlin_db import MerlinDatabase +from merlin.router import run_task_server +from merlin.study.study import MerlinStudy +from merlin.utils import ARRAY_FILE_FORMATS, verify_filepath + + +class RunCommand(CommandEntryPoint): + """ + Handles `run` CLI command for running a workflow (sending tasks to the queues on the broker). + + Methods: + add_parser: Adds the `run` command to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `run` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `run` command parser will be added. + """ + run: ArgumentParser = subparsers.add_parser( + "run", + help="Run a workflow using a Merlin or Maestro YAML study " "specification.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + run.set_defaults(func=self.process_command) + run.add_argument("specification", type=str, help="Path to a Merlin or Maestro YAML file") + run.add_argument( + "--local", + action="store_const", + dest="run_mode", + const="local", + default="distributed", + help="Run locally instead of distributed", + ) + run.add_argument( + "--vars", + action="store", + dest="variables", + type=str, + nargs="+", + default=None, + help="Specify desired Merlin variable values to override those found in the specification. Space-delimited. " + "Example: '--vars LEARN=path/to/new_learn.py EPOCHS=3'", + ) + # TODO add all supported formats to doc string # pylint: disable=fixme + run.add_argument( + "--samplesfile", + action="store", + dest="samples_file", + type=str, + default=None, + help=f"Specify file containing samples. Valid choices: {ARRAY_FILE_FORMATS}", + ) + run.add_argument( + "--dry", + action="store_true", + dest="dry", + default=False, + help="Flag to dry-run a workflow, which sets up the workspace but does not launch tasks.", + ) + run.add_argument( + "--no-errors", + action="store_true", + dest="no_errors", + default=False, + help="Flag to ignore some flux errors for testing (often used with --dry --local).", + ) + run.add_argument( + "--pgen", + action="store", + dest="pgen_file", + type=str, + default=None, + help="Provide a pgen file to override global.parameters.", + ) + run.add_argument( + "--pargs", + type=str, + action="append", + default=[], + help="A string that represents a single argument to pass " + "a custom parameter generation function. Reuse '--parg' " + "to pass multiple arguments. [Use with '--pgen']", + ) + + def process_command(self, args: Namespace): + """ + CLI command for running a study. + + This function initializes and runs a study using the specified parameters. + It handles file verification, variable parsing, and checks for required + arguments related to the study configuration and execution. + + Args: + args: Parsed CLI arguments containing:\n + - `specification`: Path to the specification file for the study. + - `variables`: Optional variable overrides for the study. + - `samples_file`: Optional path to a samples file. + - `dry`: If True, runs the study in dry-run mode (without actual execution). + - `no_errors`: If True, suppresses error reporting. + - `pgen_file`: Optional path to the pgen file, required if `pargs` is specified. + - `pargs`: Additional arguments for parallel processing. + + Raises: + ValueError: + If the `pargs` parameter is used without specifying a `pgen_file`. + """ + print(banner_small) + filepath: str = verify_filepath(args.specification) + variables_dict: str = parse_override_vars(args.variables) + samples_file: Optional[str] = None + if args.samples_file: + samples_file = verify_filepath(args.samples_file) + + # pgen checks + if args.pargs and not args.pgen_file: + raise ValueError("Cannot use the 'pargs' parameter without specifying a 'pgen'!") + if args.pgen_file: + verify_filepath(args.pgen_file) + + study: MerlinStudy = MerlinStudy( + filepath, + override_vars=variables_dict, + samples_file=samples_file, + dry_run=args.dry, + no_errors=args.no_errors, + pgen_file=args.pgen_file, + pargs=args.pargs, + ) + + if args.run_mode == "local": + initialize_config(local_mode=True) + + # Initialize the database + merlin_db = MerlinDatabase() + + # Create a run entry + run_entity = merlin_db.create( + "run", + study_name=study.expanded_spec.name, + workspace=study.workspace, + queues=study.expanded_spec.get_queue_list(["all"]), + ) + + # Create logical worker entries + step_queue_map = study.expanded_spec.get_task_queues() + for worker, steps in study.expanded_spec.get_worker_step_map().items(): + worker_queues = {step_queue_map[step] for step in steps} + logical_worker_entity = merlin_db.create("logical_worker", worker, worker_queues) + + # Add the run id to the worker entry and the worker id to the run entry + logical_worker_entity.add_run(run_entity.get_id()) + run_entity.add_worker(logical_worker_entity.get_id()) + + run_task_server(study, args.run_mode) diff --git a/merlin/cli/commands/run_workers.py b/merlin/cli/commands/run_workers.py new file mode 100644 index 000000000..ce08a7f4c --- /dev/null +++ b/merlin/cli/commands/run_workers.py @@ -0,0 +1,137 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +CLI module for launching Merlin worker processes. + +This module defines the `RunWorkersCommand` class, which implements the `run-workers` +subcommand in the Merlin CLI. The command starts worker processes that execute tasks +defined in a Merlin YAML workflow specification, associating workers with the +correct task queues without queuing tasks themselves. +""" + +# pylint: disable=duplicate-code + +import logging +from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, Namespace + +from merlin.ascii_art import banner_small +from merlin.cli.commands.command_entry_point import CommandEntryPoint +from merlin.cli.utils import get_merlin_spec_with_override +from merlin.config.configfile import initialize_config +from merlin.db_scripts.merlin_db import MerlinDatabase +from merlin.router import launch_workers + + +LOG = logging.getLogger("merlin") + + +class RunWorkersCommand(CommandEntryPoint): + """ + Handles `run-workers` CLI command for launching Merlin workers. + + Methods: + add_parser: Adds the `run-workers` command to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `run-workers` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `run-workers` command parser will be added. + """ + run_workers: ArgumentParser = subparsers.add_parser( + "run-workers", + help="Run the workers associated with the Merlin YAML study " + "specification. Does -not- queue tasks, just workers tied " + "to the correct queues.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + run_workers.set_defaults(func=self.process_command) + run_workers.add_argument("specification", type=str, help="Path to a Merlin YAML spec file") + run_workers.add_argument( + "--worker-args", + type=str, + dest="worker_args", + default="", + help="celery worker arguments in quotes.", + ) + run_workers.add_argument( + "--steps", + nargs="+", + type=str, + dest="worker_steps", + default=["all"], + help="The specific steps in the YAML file you want workers for", + ) + run_workers.add_argument( + "--echo", + action="store_true", + default=False, + dest="worker_echo_only", + help="Just echo the command; do not actually run it", + ) + run_workers.add_argument( + "--vars", + action="store", + dest="variables", + type=str, + nargs="+", + default=None, + help="Specify desired Merlin variable values to override those found in the specification. Space-delimited. " + "Example: '--vars LEARN=path/to/new_learn.py EPOCHS=3'", + ) + run_workers.add_argument( + "--disable-logs", + action="store_true", + help="Turn off the logs for the celery workers. Note: having the -l flag " + "in your workers' args section will overwrite this flag for that worker.", + ) + + def process_command(self, args: Namespace): + """ + CLI command for launching workers. + + This function initializes worker processes for executing tasks as defined + in the Merlin specification. + + Args: + args: Parsed CLI arguments containing:\n + - `worker_echo_only`: If True, don't start the workers and just echo the launch command + - Additional worker-related parameters such as: + - `worker_steps`: Only start workers for these steps. + - `worker_args`: Arguments to pass to the worker processes. + - `disable_logs`: If True, disables logging for the worker processes. + """ + if not args.worker_echo_only: + print(banner_small) + else: + initialize_config(local_mode=True) + + spec, filepath = get_merlin_spec_with_override(args) + if not args.worker_echo_only: + LOG.info(f"Launching workers from '{filepath}'") + + # Initialize the database + merlin_db = MerlinDatabase() + + # Create logical worker entries + step_queue_map = spec.get_task_queues() + for worker, steps in spec.get_worker_step_map().items(): + worker_queues = {step_queue_map[step] for step in steps} + merlin_db.create("logical_worker", worker, worker_queues) + + # Launch the workers + launch_worker_status = launch_workers( + spec, args.worker_steps, args.worker_args, args.disable_logs, args.worker_echo_only + ) + + if args.worker_echo_only: + print(launch_worker_status) + else: + LOG.debug(f"celery command: {launch_worker_status}") diff --git a/merlin/cli/commands/server.py b/merlin/cli/commands/server.py new file mode 100644 index 000000000..f17605cfe --- /dev/null +++ b/merlin/cli/commands/server.py @@ -0,0 +1,224 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Merlin CLI server command module. + +This module defines the `ServerCommand` class, which provides subcommands +for managing the Merlin server components such as initialization, starting, +stopping, restarting, and configuring the server. These subcommands are integrated +into the Merlin CLI via `argparse`. +""" + +# pylint: disable=duplicate-code + +import logging +import os +from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, Namespace + +from merlin.cli.commands.command_entry_point import CommandEntryPoint +from merlin.server.server_commands import config_server, init_server, restart_server, start_server, status_server, stop_server + + +LOG = logging.getLogger("merlin") + + +class ServerCommand(CommandEntryPoint): + """ + Handles `server` CLI commands for interacting with Merlin's containerized server. + + Methods: + add_parser: Adds the `server` command and its subcommands to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + def _add_config_subcommand(self, server_commands: ArgumentParser): + """ + Add the `config` subcommand to the server command parser. + + Parameters: + server_commands (ArgumentParser): The server subparser to which the config command will be added. + """ + server_config: ArgumentParser = server_commands.add_parser( + "config", + help="Making configurations for to the merlin server instance.", + description="Config server.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + server_config.add_argument( + "-ip", + "--ipaddress", + action="store", + type=str, + help="Set the binded IP address for the merlin server container.", + ) + server_config.add_argument( + "-p", + "--port", + action="store", + type=int, + help="Set the binded port for the merlin server container.", + ) + server_config.add_argument( + "-pwd", + "--password", + action="store", + type=str, + help="Set the password file to be used for merlin server container.", + ) + server_config.add_argument( + "--add-user", + action="store", + nargs=2, + type=str, + help="Create a new user for merlin server instance. (Provide both username and password)", + ) + server_config.add_argument("--remove-user", action="store", type=str, help="Remove an exisiting user.") + server_config.add_argument( + "-d", + "--directory", + action="store", + type=str, + help="Set the working directory of the merlin server container.", + ) + server_config.add_argument( + "-ss", + "--snapshot-seconds", + action="store", + type=int, + help="Set the number of seconds merlin server waits before checking if a snapshot is needed.", + ) + server_config.add_argument( + "-sc", + "--snapshot-changes", + action="store", + type=int, + help="Set the number of changes that are required to be made to the merlin server before a snapshot is made.", + ) + server_config.add_argument( + "-sf", + "--snapshot-file", + action="store", + type=str, + help="Set the snapshot filename for database dumps.", + ) + server_config.add_argument( + "-am", + "--append-mode", + action="store", + type=str, + help="The appendonly mode to be set. The avaiable options are always, everysec, no.", + ) + server_config.add_argument( + "-af", + "--append-file", + action="store", + type=str, + help="Set append only filename for merlin server container.", + ) + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `server` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `server` command parser will be added. + """ + server: ArgumentParser = subparsers.add_parser( + "server", + help="Manage broker and results server for merlin workflow.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + server.set_defaults(func=self.process_command) + + server_commands: ArgumentParser = server.add_subparsers(dest="commands") + + # `merlin server init` subcommand + server_commands.add_parser( + "init", + help="Initialize merlin server resources.", + description="Initialize merlin server", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # `merlin server status` subcommand + server_commands.add_parser( + "status", + help="View status of the current server containers.", + description="View status", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # `merlin server start` subcommand + server_commands.add_parser( + "start", + help="Start a containerized server to be used as an broker and results server.", + description="Start server", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # `merlin server stop` subcommand + server_commands.add_parser( + "stop", + help="Stop an instance of redis containers currently running.", + description="Stop server.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # `merlin server restart` subcommand + server_commands.add_parser( + "restart", + help="Restart merlin server instance", + description="Restart server.", + formatter_class=ArgumentDefaultsHelpFormatter, + ) + + # `merlin server config` subcommand + self._add_config_subcommand(server_commands) + + def process_command(self, args: Namespace): + """ + Route to the appropriate server function based on the command + specified via the CLI. + + This function processes commands related to server management, + directing the flow to the corresponding function for actions such + as initializing, starting, stopping, checking status, restarting, + or configuring the server. + + Args: + args: Parsed command-line arguments, which includes:\n + - `commands`: The server management command to execute. + Possible values are: + - `init`: Initialize the server. + - `start`: Start the server. + - `stop`: Stop the server. + - `status`: Check the server status. + - `restart`: Restart the server. + - `config`: Configure the server. + """ + try: + lc_all_val = os.environ["LC_ALL"] + if lc_all_val != "C": + raise ValueError( + f"The 'LC_ALL' environment variable is currently set to {lc_all_val} but it must be set to 'C'." + ) + except KeyError: + LOG.debug("The 'LC_ALL' environment variable was not set. Setting this to 'C'.") + os.environ["LC_ALL"] = "C" # Necessary for Redis to configure LOCALE + + if args.commands == "init": + init_server() + elif args.commands == "start": + start_server() + elif args.commands == "stop": + stop_server() + elif args.commands == "status": + status_server() + elif args.commands == "restart": + restart_server() + elif args.commands == "config": + config_server(args) diff --git a/merlin/cli/commands/status.py b/merlin/cli/commands/status.py new file mode 100644 index 000000000..4958a5400 --- /dev/null +++ b/merlin/cli/commands/status.py @@ -0,0 +1,260 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +CLI module for querying and displaying the status of Merlin workflows. + +This module defines two primary CLI command handlers: + +- `StatusCommand`: Handles the `status` command, which shows a high-level summary + of the current state of a Merlin study or workflow. It supports querying status + based on a Merlin YAML specification file or an existing study workspace, with + options for output formatting and filtering. + +- `DetailedStatusCommand`: Extends `StatusCommand` to provide a more granular, + task-by-task view of the workflow status via the `detailed-status` command. + This includes extensive filtering and display customization options. +""" + +# pylint: disable=duplicate-code + +import logging +from argparse import ArgumentParser, Namespace + +from merlin.ascii_art import banner_small +from merlin.cli.commands.command_entry_point import CommandEntryPoint +from merlin.spec.expansion import get_spec_with_expansion +from merlin.study.status import DetailedStatus, Status +from merlin.study.status_constants import VALID_RETURN_CODES, VALID_STATUS_FILTERS +from merlin.study.status_renderers import status_renderer_factory +from merlin.utils import verify_dirpath, verify_filepath + + +LOG = logging.getLogger("merlin") + + +class StatusCommand(CommandEntryPoint): + """ + Handles `status` CLI command for checking the high-level status of a workflow. + + Methods: + add_parser: Adds the `status` command to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `status` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `status` command parser will be added. + """ + status_cmd: ArgumentParser = subparsers.add_parser( + "status", + help="Display a summary of the status of a study.", + ) + status_cmd.set_defaults(func=self.process_command, detailed=False) + status_cmd.add_argument( + "spec_or_workspace", type=str, help="Path to a Merlin YAML spec file or a launched Merlin study" + ) + status_cmd.add_argument( + "--cb-help", action="store_true", help="Colorblind help; uses different symbols to represent different statuses" + ) + status_cmd.add_argument( + "--dump", type=str, help="Dump the status to a file. Provide the filename (must be .csv or .json).", default=None + ) + status_cmd.add_argument( + "--no-prompts", + action="store_true", + help="Ignore any prompts provided. This will default to the latest study \ + if you provide a spec file rather than a study workspace.", + ) + status_cmd.add_argument( + "--task_server", + type=str, + default="celery", + help="Task server type.\ + Default: %(default)s", + ) + status_cmd.add_argument( + "-o", + "--output-path", + action="store", + type=str, + default=None, + help="Specify a location to look for output workspaces. Only used when a spec file is passed as the argument " + "to 'status'; this will NOT be used if an output workspace is passed as the argument.", + ) + + def process_command(self, args: Namespace): + """ + CLI command for querying the status of studies. + + This function processes the given command-line arguments to determine the + status of a study. It constructs either a [`Status`][study.status.Status] object + or a [`DetailedStatus`][study.status.DetailedStatus] object based on the specified + command and the arguments provided. The function handles validations for the task + server input and the output format specified for status dumping. + + Object mapping: + - `merlin status` -> [`Status`][study.status.Status] object + - `merlin detailed-status` -> [`DetailedStatus`][study.status.DetailedStatus] + object + + Args: + args: Parsed CLI arguments containing user inputs for the status query. + + Raises: + ValueError: + - If the task server specified is not supported (only "celery" is valid). + - If the --dump filename provided does not end with ".csv" or ".json". + """ + print(banner_small) + + # Ensure task server is valid + if args.task_server != "celery": + raise ValueError("Currently the only supported task server is celery.") + + # Make sure dump is valid if provided + if args.dump and (not args.dump.endswith(".csv") and not args.dump.endswith(".json")): + raise ValueError("The --dump option takes a filename that must end with .csv or .json") + + # Establish whether the argument provided by the user was a spec file or a study directory + spec_display = False + try: + file_or_ws = verify_filepath(args.spec_or_workspace) + spec_display = True + except ValueError: + try: + file_or_ws = verify_dirpath(args.spec_or_workspace) + except ValueError: + LOG.error(f"The file or directory path {args.spec_or_workspace} does not exist.") + return + + # If we're loading status based on a spec, load in the spec provided + if spec_display: + args.specification = file_or_ws + args.spec_provided = get_spec_with_expansion(args.specification) + + # Get either a Status object or DetailedStatus object + if args.detailed: + status_obj = DetailedStatus(args, spec_display, file_or_ws) + else: + status_obj = Status(args, spec_display, file_or_ws) + + # Handle output appropriately + if args.dump: + status_obj.dump() + else: + status_obj.display() + + +class DetailedStatusCommand(StatusCommand): + """ + Handles `detailed-status` CLI command for checking the in-depth status of a workflow. + + Methods: + add_parser: Adds the `detailed-status` command to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `detailed-status` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `detailed-status` command parser will be added. + """ + detailed_status: ArgumentParser = subparsers.add_parser( + "detailed-status", + help="Display a task-by-task status of a study.", + ) + detailed_status.set_defaults(func=self.process_command, detailed=True) + detailed_status.add_argument( + "spec_or_workspace", type=str, help="Path to a Merlin YAML spec file or a launched Merlin study" + ) + detailed_status.add_argument( + "--dump", type=str, help="Dump the status to a file. Provide the filename (must be .csv or .json).", default=None + ) + detailed_status.add_argument( + "--task_server", + type=str, + default="celery", + help="Task server type.\ + Default: %(default)s", + ) + detailed_status.add_argument( + "-o", + "--output-path", + action="store", + type=str, + default=None, + help="Specify a location to look for output workspaces. Only used when a spec file is passed as the argument " + "to 'status'; this will NOT be used if an output workspace is passed as the argument.", + ) + status_filter_group = detailed_status.add_argument_group("filter options") + status_filter_group.add_argument( + "--max-tasks", action="store", type=int, help="Sets a limit on how many tasks can be displayed" + ) + status_filter_group.add_argument( + "--return-code", + action="store", + nargs="+", + type=str, + choices=VALID_RETURN_CODES, + help="Filter which tasks to display based on their return code", + ) + status_filter_group.add_argument( + "--steps", + nargs="+", + type=str, + dest="steps", + default=["all"], + help="Filter which tasks to display based on the steps they're associated with", + ) + status_filter_group.add_argument( + "--task-queues", + nargs="+", + type=str, + help="Filter which tasks to display based on the task queue they're in", + ) + status_filter_group.add_argument( + "--task-status", + action="store", + nargs="+", + type=str, + choices=VALID_STATUS_FILTERS, + help="Filter which tasks to display based on their status", + ) + status_filter_group.add_argument( + "--workers", + nargs="+", + type=str, + help="Filter which tasks to display based on which workers are processing them", + ) + status_display_group = detailed_status.add_argument_group("display options") + status_display_group.add_argument( + "--disable-pager", action="store_true", help="Turn off the pager functionality when viewing the status" + ) + status_display_group.add_argument( + "--disable-theme", + action="store_true", + help="Turn off styling for the status layout (If you want styling but it's not working, try modifying " + "the MANPAGER or PAGER environment variables to be 'less -r'; i.e. export MANPAGER='less -r')", + ) + status_display_group.add_argument( + "--layout", + type=str, + choices=status_renderer_factory.get_layouts(), + default="default", + help="Alternate status layouts [Default: %(default)s]", + ) + status_display_group.add_argument( + "--no-prompts", + action="store_true", + help="Ignore any prompts provided. This will default to the latest study \ + if you provide a spec file rather than a study workspace.", + ) diff --git a/merlin/cli/commands/stop_workers.py b/merlin/cli/commands/stop_workers.py new file mode 100644 index 000000000..3d5ef6d82 --- /dev/null +++ b/merlin/cli/commands/stop_workers.py @@ -0,0 +1,99 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +CLI module for shutting down Merlin workers. + +This module defines the `StopWorkersCommand` class, which handles the `stop-workers` +subcommand in the Merlin CLI. It provides functionality to stop running workers that +are connected to a task server such as Celery. +""" + +# pylint: disable=duplicate-code + +import logging +from argparse import ArgumentParser, Namespace + +from merlin.ascii_art import banner_small +from merlin.cli.commands.command_entry_point import CommandEntryPoint +from merlin.router import stop_workers +from merlin.spec.specification import MerlinSpec +from merlin.utils import verify_filepath + + +LOG = logging.getLogger("merlin") + + +class StopWorkersCommand(CommandEntryPoint): + """ + Handles `stop-workers` CLI command for shutting down Merlin workers. + + Methods: + add_parser: Adds the `stop-workers` command to the CLI parser. + process_command: Processes the CLI input and dispatches the appropriate action. + """ + + def add_parser(self, subparsers: ArgumentParser): + """ + Add the `stop-workers` command parser to the CLI argument parser. + + Parameters: + subparsers (ArgumentParser): The subparsers object to which the `stop-workers` command parser will be added. + """ + stop: ArgumentParser = subparsers.add_parser("stop-workers", help="Attempt to stop all task server workers.") + stop.set_defaults(func=self.process_command) + stop.add_argument( + "--spec", + type=str, + default=None, + help="Path to a Merlin YAML spec file from which to read worker names to stop.", + ) + stop.add_argument( + "--task_server", + type=str, + default="celery", + help="Task server type from which to stop workers.\ + Default: %(default)s", + ) + stop.add_argument("--queues", type=str, default=None, nargs="+", help="specific queues to stop") + stop.add_argument( + "--workers", + type=str, + action="store", + nargs="+", + default=None, + help="regex match for specific workers to stop", + ) + + def process_command(self, args: Namespace): + """ + CLI command for stopping all workers. + + This function stops any active workers connected to a user's task server. + If the `--spec` argument is provided, this function retrieves the names of + workers from a the spec file and then issues a command to stop them. + + Args: + args: Parsed command-line arguments, which may include:\n + - `spec`: Path to the specification file to load worker names. + - `task_server`: Address of the task server to send the stop command to. + - `queues`: List of queue names to filter the workers. + - `workers`: List of specific worker names to stop. + """ + print(banner_small) + worker_names = [] + + # Load in the spec if one was provided via the CLI + if args.spec: + spec_path = verify_filepath(args.spec) + spec = MerlinSpec.load_specification(spec_path) + worker_names = spec.get_worker_names() + for worker_name in worker_names: + if "$" in worker_name: + LOG.warning(f"Worker '{worker_name}' is unexpanded. Target provenance spec instead?") + + # Send stop command to router + stop_workers(args.task_server, worker_names, args.queues, args.workers) diff --git a/merlin/cli/utils.py b/merlin/cli/utils.py new file mode 100644 index 000000000..4f461d348 --- /dev/null +++ b/merlin/cli/utils.py @@ -0,0 +1,110 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Utility functions to support Merlin CLI command handlers. + +This module provides common helper functions used by various CLI commands +in the Merlin application. These utilities focus on parsing and validating +command-line arguments related to specification files and variable overrides, +as well as loading and expanding Merlin YAML specifications. +""" + +import logging +from argparse import Namespace +from contextlib import suppress +from typing import Dict, List, Optional, Tuple, Union + +from merlin.spec.expansion import RESERVED, get_spec_with_expansion +from merlin.spec.specification import MerlinSpec +from merlin.utils import verify_filepath + + +LOG = logging.getLogger("merlin") + + +def parse_override_vars( + variables_list: Optional[List[str]], +) -> Optional[Dict[str, Union[str, int]]]: + """ + Parse a list of command-line variables into a dictionary of key-value pairs. + + This function takes an optional list of strings following the syntax + "KEY=val" and converts them into a dictionary. It validates the format + of the variables and ensures that keys are valid according to specified rules. + + Args: + variables_list: An optional list of strings, where each string should be in the + format "KEY=val", e.g., ["KEY1=value1", "KEY2=42"]. + + Returns: + A dictionary where the keys are variable names (str) and the + values are either strings or integers. If `variables_list` is + None or empty, returns None. + + Raises: + ValueError: If the input format is incorrect, including:\n + - Missing '=' operator. + - Excess '=' operators in a variable assignment. + - Invalid variable names (must be alphanumeric and underscores). + - Attempting to override reserved variable names. + """ + if variables_list is None: + return None + LOG.debug(f"Command line override variables = {variables_list}") + result: Dict[str, Union[str, int]] = {} + arg: str + for arg in variables_list: + try: + if "=" not in arg: + raise ValueError("--vars requires '=' operator. See 'merlin run --help' for an example.") + entry: str = arg.split("=") + if len(entry) != 2: + raise ValueError("--vars requires ONE '=' operator (without spaces) per variable assignment.") + key: str = entry[0] + if key is None or key == "" or "$" in key: + raise ValueError("--vars requires valid variable names comprised of alphanumeric characters and underscores.") + if key in RESERVED: + raise ValueError(f"Cannot override reserved word '{key}'! Reserved words are: {RESERVED}.") + + val: Union[str, int] = entry[1] + with suppress(ValueError): + int(val) + val = int(val) + result[key] = val + + except Exception as excpt: + raise ValueError( + f"{excpt} Bad '--vars' formatting on command line. See 'merlin run --help' for an example." + ) from excpt + return result + + +def get_merlin_spec_with_override(args: Namespace) -> Tuple[MerlinSpec, str]: + """ + Shared command to retrieve a [`MerlinSpec`][spec.specification.MerlinSpec] object + and an expanded filepath. + + This function processes parsed command-line interface (CLI) arguments to validate + and expand the specified filepath and any associated variables. It then constructs + and returns a [`MerlinSpec`][spec.specification.MerlinSpec] object based on the + provided specification. + + Args: + args: Parsed CLI arguments containing:\n + - `specification`: the path to the specification file + - `variables`: optional variable overrides to customize the spec. + + Returns: + spec (spec.specification.MerlinSpec): An instance of the + [`MerlinSpec`][spec.specification.MerlinSpec] class with the expanded + configuration based on the provided filepath and variables. + filepath: The expanded filepath derived from the specification. + """ + filepath = verify_filepath(args.specification) + variables_dict = parse_override_vars(args.variables) + spec = get_spec_with_expansion(filepath, override_vars=variables_dict) + return spec, filepath diff --git a/merlin/main.py b/merlin/main.py index 6cde71339..ca4287d51 100644 --- a/merlin/main.py +++ b/merlin/main.py @@ -4,1694 +4,19 @@ # other details. No copyright assignment is required to contribute to Merlin. ############################################################################## -"""The top level main function for invoking Merlin.""" +""" +Main entry point into Merlin's codebase. +""" -from __future__ import print_function - -import glob import logging -import os import sys -import time import traceback -from argparse import ( - SUPPRESS, - ArgumentDefaultsHelpFormatter, - ArgumentParser, - ArgumentTypeError, - Namespace, - RawDescriptionHelpFormatter, - RawTextHelpFormatter, -) -from contextlib import suppress -from typing import Dict, List, Optional, Tuple, Union - -import yaml -from tabulate import tabulate -from merlin import VERSION, router -from merlin.ascii_art import banner_small -from merlin.config.configfile import initialize_config -from merlin.config.merlin_config_manager import MerlinConfigManager -from merlin.db_scripts.db_commands import database_delete, database_get, database_info -from merlin.db_scripts.merlin_db import MerlinDatabase -from merlin.examples.generator import list_examples, setup_example +from merlin.cli.argparse_main import build_main_parser from merlin.log_formatter import setup_logging -from merlin.monitor.monitor import Monitor -from merlin.server.server_commands import config_server, init_server, restart_server, start_server, status_server, stop_server -from merlin.spec.expansion import RESERVED, get_spec_with_expansion -from merlin.spec.specification import MerlinSpec -from merlin.study.status import DetailedStatus, Status -from merlin.study.status_constants import VALID_RETURN_CODES, VALID_STATUS_FILTERS -from merlin.study.status_renderers import status_renderer_factory -from merlin.study.study import MerlinStudy -from merlin.utils import ARRAY_FILE_FORMATS, verify_dirpath, verify_filepath LOG = logging.getLogger("merlin") -DEFAULT_LOG_LEVEL = "INFO" - - -class HelpParser(ArgumentParser): - """ - This class overrides the error message of the argument parser to - print the help message when an error happens. - - Methods: - error: Override the error message of the `ArgumentParser` class. - """ - - def error(self, message: str): - """ - Override the error message of the `ArgumentParser` class. - - Args: - message: The error message to log. - """ - sys.stderr.write(f"error: {message}\n") - self.print_help() - sys.exit(2) - - -def parse_override_vars( - variables_list: Optional[List[str]], -) -> Optional[Dict[str, Union[str, int]]]: - """ - Parse a list of command-line variables into a dictionary of key-value pairs. - - This function takes an optional list of strings following the syntax - "KEY=val" and converts them into a dictionary. It validates the format - of the variables and ensures that keys are valid according to specified rules. - - Args: - variables_list: An optional list of strings, where each string should be in the - format "KEY=val", e.g., ["KEY1=value1", "KEY2=42"]. - - Returns: - A dictionary where the keys are variable names (str) and the - values are either strings or integers. If `variables_list` is - None or empty, returns None. - - Raises: - ValueError: If the input format is incorrect, including:\n - - Missing '=' operator. - - Excess '=' operators in a variable assignment. - - Invalid variable names (must be alphanumeric and underscores). - - Attempting to override reserved variable names. - """ - if variables_list is None: - return None - LOG.debug(f"Command line override variables = {variables_list}") - result: Dict[str, Union[str, int]] = {} - arg: str - for arg in variables_list: - try: - if "=" not in arg: - raise ValueError("--vars requires '=' operator. See 'merlin run --help' for an example.") - entry: str = arg.split("=") - if len(entry) != 2: - raise ValueError("--vars requires ONE '=' operator (without spaces) per variable assignment.") - key: str = entry[0] - if key is None or key == "" or "$" in key: - raise ValueError("--vars requires valid variable names comprised of alphanumeric characters and underscores.") - if key in RESERVED: - raise ValueError(f"Cannot override reserved word '{key}'! Reserved words are: {RESERVED}.") - - val: Union[str, int] = entry[1] - with suppress(ValueError): - int(val) - val = int(val) - result[key] = val - - except Exception as excpt: - raise ValueError( - f"{excpt} Bad '--vars' formatting on command line. See 'merlin run --help' for an example." - ) from excpt - return result - - -def get_merlin_spec_with_override(args: Namespace) -> Tuple[MerlinSpec, str]: - """ - Shared command to retrieve a [`MerlinSpec`][spec.specification.MerlinSpec] object - and an expanded filepath. - - This function processes parsed command-line interface (CLI) arguments to validate - and expand the specified filepath and any associated variables. It then constructs - and returns a [`MerlinSpec`][spec.specification.MerlinSpec] object based on the - provided specification. - - Args: - args: Parsed CLI arguments containing:\n - - `specification`: the path to the specification file - - `variables`: optional variable overrides to customize the spec. - - Returns: - spec (spec.specification.MerlinSpec): An instance of the - [`MerlinSpec`][spec.specification.MerlinSpec] class with the expanded - configuration based on the provided filepath and variables. - filepath: The expanded filepath derived from the specification. - """ - filepath = verify_filepath(args.specification) - variables_dict = parse_override_vars(args.variables) - spec = get_spec_with_expansion(filepath, override_vars=variables_dict) - return spec, filepath - - -def process_run(args: Namespace): - """ - CLI command for running a study. - - This function initializes and runs a study using the specified parameters. - It handles file verification, variable parsing, and checks for required - arguments related to the study configuration and execution. - - Args: - args: Parsed CLI arguments containing:\n - - `specification`: Path to the specification file for the study. - - `variables`: Optional variable overrides for the study. - - `samples_file`: Optional path to a samples file. - - `dry`: If True, runs the study in dry-run mode (without actual execution). - - `no_errors`: If True, suppresses error reporting. - - `pgen_file`: Optional path to the pgen file, required if `pargs` is specified. - - `pargs`: Additional arguments for parallel processing. - - Raises: - ValueError: - If the `pargs` parameter is used without specifying a `pgen_file`. - """ - print(banner_small) - filepath: str = verify_filepath(args.specification) - variables_dict: str = parse_override_vars(args.variables) - samples_file: Optional[str] = None - if args.samples_file: - samples_file = verify_filepath(args.samples_file) - - # pgen checks - if args.pargs and not args.pgen_file: - raise ValueError("Cannot use the 'pargs' parameter without specifying a 'pgen'!") - if args.pgen_file: - verify_filepath(args.pgen_file) - - study: MerlinStudy = MerlinStudy( - filepath, - override_vars=variables_dict, - samples_file=samples_file, - dry_run=args.dry, - no_errors=args.no_errors, - pgen_file=args.pgen_file, - pargs=args.pargs, - ) - - if args.run_mode == "local": - initialize_config(local_mode=True) - - # Initialize the database - merlin_db = MerlinDatabase() - - # Create a run entry - run_entity = merlin_db.create( - "run", - study_name=study.expanded_spec.name, - workspace=study.workspace, - queues=study.expanded_spec.get_queue_list(["all"]), - ) - - # Create logical worker entries - step_queue_map = study.expanded_spec.get_task_queues() - for worker, steps in study.expanded_spec.get_worker_step_map().items(): - worker_queues = set([step_queue_map[step] for step in steps]) - logical_worker_entity = merlin_db.create("logical_worker", worker, worker_queues) - - # Add the run id to the worker entry and the worker id to the run entry - logical_worker_entity.add_run(run_entity.get_id()) - run_entity.add_worker(logical_worker_entity.get_id()) - - router.run_task_server(study, args.run_mode) - - -def process_restart(args: Namespace): - """ - CLI command for restarting a study. - - This function handles the restart process by verifying the specified restart - directory, locating a valid provenance specification file, and initiating - the study from that point. - - Args: - args: Parsed CLI arguments containing:\n - - `restart_dir`: Path to the directory where the restart specifications are located. - - `run_mode`: The mode for running the study (e.g., normal, dry-run). - - Raises: - ValueError: If the `restart_dir` does not contain a valid provenance spec file or - if multiple files match the specified pattern. - """ - print(banner_small) - restart_dir: str = verify_dirpath(args.restart_dir) - filepath: str = os.path.join(args.restart_dir, "merlin_info", "*.expanded.yaml") - possible_specs: Optional[List[str]] = glob.glob(filepath) - if not possible_specs: # len == 0 - raise ValueError(f"'{filepath}' does not match any provenance spec file to restart from.") - if len(possible_specs) > 1: - raise ValueError(f"'{filepath}' matches more than one provenance spec file to restart from.") - filepath: str = verify_filepath(possible_specs[0]) - LOG.info(f"Restarting workflow at '{restart_dir}'") - study: MerlinStudy = MerlinStudy(filepath, restart_dir=restart_dir) - - if args.run_mode == "local": - initialize_config(local_mode=True) - - router.run_task_server(study, args.run_mode) - - -def launch_workers(args: Namespace): - """ - CLI command for launching workers. - - This function initializes worker processes for executing tasks as defined - in the Merlin specification. - - Args: - args: Parsed CLI arguments containing:\n - - `worker_echo_only`: If True, don't start the workers and just echo the launch command - - Additional worker-related parameters such as: - - `worker_steps`: Only start workers for these steps. - - `worker_args`: Arguments to pass to the worker processes. - - `disable_logs`: If True, disables logging for the worker processes. - """ - if not args.worker_echo_only: - print(banner_small) - else: - initialize_config(local_mode=True) - - spec, filepath = get_merlin_spec_with_override(args) - if not args.worker_echo_only: - LOG.info(f"Launching workers from '{filepath}'") - - # Initialize the database - merlin_db = MerlinDatabase() - - # Create logical worker entries - step_queue_map = spec.get_task_queues() - for worker, steps in spec.get_worker_step_map().items(): - worker_queues = set([step_queue_map[step] for step in steps]) - merlin_db.create("logical_worker", worker, worker_queues) - - # Launch the workers - launch_worker_status = router.launch_workers( - spec, args.worker_steps, args.worker_args, args.disable_logs, args.worker_echo_only - ) - - if args.worker_echo_only: - print(launch_worker_status) - else: - LOG.debug(f"celery command: {launch_worker_status}") - - -def purge_tasks(args: Namespace): - """ - CLI command for purging tasks from the task server. - - This function removes specified tasks from the task server based on the provided - Merlin specification. It allows for targeted purging or forced removal of tasks. - - Args: - args: Parsed CLI arguments containing:\n - - `purge_force`: If True, forces the purge operation without confirmation. - - `purge_steps`: Steps or criteria based on which tasks will be purged. - """ - print(banner_small) - spec, _ = get_merlin_spec_with_override(args) - ret = router.purge_tasks( - spec.merlin["resources"]["task_server"], - spec, - args.purge_force, - args.purge_steps, - ) - - LOG.info(f"Purge return = {ret} .") - - -def query_status(args: Namespace): - """ - CLI command for querying the status of studies. - - This function processes the given command-line arguments to determine the - status of a study. It constructs either a [`Status`][study.status.Status] object - or a [`DetailedStatus`][study.status.DetailedStatus] object based on the specified - command and the arguments provided. The function handles validations for the task - server input and the output format specified for status dumping. - - Object mapping: - - `merlin status` -> [`Status`][study.status.Status] object - - `merlin detailed-status` -> [`DetailedStatus`][study.status.DetailedStatus] - object - - Args: - args: Parsed CLI arguments containing user inputs for the status query. - - Raises: - ValueError: - - If the task server specified is not supported (only "celery" is valid). - - If the --dump filename provided does not end with ".csv" or ".json". - """ - print(banner_small) - - # Ensure task server is valid - if args.task_server != "celery": - raise ValueError("Currently the only supported task server is celery.") - - # Make sure dump is valid if provided - if args.dump and (not args.dump.endswith(".csv") and not args.dump.endswith(".json")): - raise ValueError("The --dump option takes a filename that must end with .csv or .json") - - # Establish whether the argument provided by the user was a spec file or a study directory - spec_display = False - try: - file_or_ws = verify_filepath(args.spec_or_workspace) - spec_display = True - except ValueError: - try: - file_or_ws = verify_dirpath(args.spec_or_workspace) - except ValueError: - LOG.error(f"The file or directory path {args.spec_or_workspace} does not exist.") - return None - - # If we're loading status based on a spec, load in the spec provided - if spec_display: - args.specification = file_or_ws - args.spec_provided = get_spec_with_expansion(args.specification) - - # Get either a Status object or DetailedStatus object - if args.detailed: - status_obj = DetailedStatus(args, spec_display, file_or_ws) - else: - status_obj = Status(args, spec_display, file_or_ws) - - # Handle output appropriately - if args.dump: - status_obj.dump() - else: - status_obj.display() - - return None - - -def query_queues(args: Namespace): - """ - CLI command for finding all workers and their associated queues. - - This function processes the command-line arguments to retrieve and display - information about the available workers and their queues within the task server. - It validates the necessary parameters, handles potential file dumping, and - formats the output for easy readability. - - Args: - args: Parsed CLI arguments containing user inputs related to the query. - - Raises: - ValueError: - - If a specification is not provided when steps are specified and the - steps do not include "all". - - If variables are included without a corresponding specification. - - If the specified dump filename does not end with '.json' or '.csv'. - """ - print(banner_small) - - # Ensure a spec is provided if steps are provided - if not args.specification: - if "all" not in args.steps: - raise ValueError("The --steps argument MUST be used with the --specification argument.") - if args.variables: - raise ValueError("The --vars argument MUST be used with the --specification argument.") - - # Ensure a supported file type is provided with the dump option - if args.dump is not None: - if not args.dump.endswith(".json") and not args.dump.endswith(".csv"): - raise ValueError("Unsupported file type. Dump files must be either '.json' or '.csv'.") - - spec = None - # Load the spec if necessary - if args.specification: - spec, _ = get_merlin_spec_with_override(args) - - # Obtain the queue information - queue_information = router.query_queues(args.task_server, spec, args.steps, args.specific_queues) - - if queue_information: - # Format the queue information so we can pass it to the tabulate library - formatted_queue_info = [("Queue Name", "Task Count", "Worker Count")] - for queue_name, queue_stats in queue_information.items(): - formatted_queue_info.append((queue_name, queue_stats["jobs"], queue_stats["consumers"])) - - # Print the queue information - print() - print(tabulate(formatted_queue_info, headers="firstrow")) - print() - - # Dump queue information to an output file if necessary - if args.dump: - router.dump_queue_info(args.task_server, queue_information, args.dump) - - -def query_workers(args: Namespace): - """ - CLI command for finding all workers. - - This function retrieves and queries the names of any active workers. - If the `--spec` argument is included, only query the workers defined in the spec file. - - Args: - args: Parsed command-line arguments, which may include:\n - - `spec`: Path to the specification file. - - `task_server`: Address of the task server to query. - - `queues`: List of queue names to filter workers. - - `workers`: List of specific worker names to query. - """ - print(banner_small) - - # Get the workers from the spec file if --spec provided - worker_names = [] - if args.spec: - spec_path = verify_filepath(args.spec) - spec = MerlinSpec.load_specification(spec_path) - worker_names = spec.get_worker_names() - for worker_name in worker_names: - if "$" in worker_name: - LOG.warning(f"Worker '{worker_name}' is unexpanded. Target provenance spec instead?") - LOG.debug(f"Searching for the following workers to stop based on the spec {args.spec}: {worker_names}") - - router.query_workers(args.task_server, worker_names, args.queues, args.workers) - - -def stop_workers(args: Namespace): - """ - CLI command for stopping all workers. - - This function stops any active workers connected to a user's task server. - If the `--spec` argument is provided, this function retrieves the names of - workers from a the spec file and then issues a command to stop them. - - Args: - args: Parsed command-line arguments, which may include:\n - - `spec`: Path to the specification file to load worker names. - - `task_server`: Address of the task server to send the stop command to. - - `queues`: List of queue names to filter the workers. - - `workers`: List of specific worker names to stop. - """ - print(banner_small) - worker_names = [] - - # Load in the spec if one was provided via the CLI - if args.spec: - spec_path = verify_filepath(args.spec) - spec = MerlinSpec.load_specification(spec_path) - worker_names = spec.get_worker_names() - for worker_name in worker_names: - if "$" in worker_name: - LOG.warning(f"Worker '{worker_name}' is unexpanded. Target provenance spec instead?") - - # Send stop command to router - router.stop_workers(args.task_server, worker_names, args.queues, args.workers) - - -def print_info(args: Namespace): - """ - CLI command to print merlin configuration info. - - Args: - args: Parsed CLI arguments. - """ - # if this is moved to the toplevel per standard style, merlin is unable to generate the (needed) default config file - from merlin import display # pylint: disable=import-outside-toplevel - - display.print_info(args) - - -def config_merlin(args: Namespace): - """ - CLI command to manage Merlin configuration files. - - This function handles various configuration-related operations based on - the provided subcommand. It ensures that the specified configuration - file has a valid YAML extension (i.e., `.yaml` or `.yml`). - - If no output file is explicitly provided, a default path is used. - - Args: - args: Parsed command-line arguments. - """ - if args.commands != "create": # Check that this is a valid yaml file - try: - with open(args.config_file, "r") as conf_file: - yaml.safe_load(conf_file) - except FileNotFoundError: - raise ArgumentTypeError(f"The file '{args.config_file}' does not exist.") - except yaml.YAMLError as e: - raise ArgumentTypeError(f"The file '{args.config_file}' is not a valid YAML file: {e}") - - config_manager = MerlinConfigManager(args) - - if args.commands == "create": - config_manager.create_template_config() - config_manager.save_config_path() - elif args.commands == "update-broker": - config_manager.update_broker() - elif args.commands == "update-backend": - config_manager.update_backend() - elif args.commands == "use": # Config file path is updated in constructor of MerlinConfigManager - config_manager.config_file = args.config_file - config_manager.save_config_path() - - -def process_example(args: Namespace) -> None: - """ - CLI command to set up or list Merlin example workflows. - - This function either lists all available example workflows or sets - up a specified example workflow to be run in the root directory. The - behavior is determined by the `workflow` argument. - - Args: - args: Parsed command-line arguments, which may include:\n - - `workflow`: The action to perform; should be "list" - to display all examples or the name of a specific example - workflow to set up. - - `path`: The directory where the example workflow - should be set up. Only applicable when `workflow` is not "list". - """ - if args.workflow == "list": - print(list_examples()) - else: - print(banner_small) - setup_example(args.workflow, args.path) - - -def process_monitor(args: Namespace): - """ - CLI command to monitor Merlin workers and queues to maintain - allocation status. - - This function periodically checks the status of Merlin workers and - the associated queues to ensure that the allocation remains active. - It includes a sleep interval to wait before each check, including - the initial one. - - Args: - args: Parsed command-line arguments, which may include:\n - - `sleep`: The duration (in seconds) to wait before - checking the queue status again. - """ - spec, _ = get_merlin_spec_with_override(args) - - # Give the user time to queue up jobs in case they haven't already - time.sleep(args.sleep) - - if args.steps != ["all"]: - LOG.warning( - "The `--steps` argument of the `merlin monitor` command is set to be deprecated in Merlin v1.14 " - "For now, using this argument will tell merlin to use the version of the monitor command from Merlin v1.12." - ) - # Check if we still need our allocation - while router.check_merlin_status(args, spec): - LOG.info("Monitor: found tasks in queues and/or tasks being processed") - time.sleep(args.sleep) - else: - monitor = Monitor(spec, args.sleep, args.task_server) - monitor.monitor_all_runs() - - LOG.info("Monitor: ... stop condition met") - - -def process_server(args: Namespace): - """ - Route to the appropriate server function based on the command - specified via the CLI. - - This function processes commands related to server management, - directing the flow to the corresponding function for actions such - as initializing, starting, stopping, checking status, restarting, - or configuring the server. - - Args: - args: Parsed command-line arguments, which includes:\n - - `commands`: The server management command to execute. - Possible values are: - - `init`: Initialize the server. - - `start`: Start the server. - - `stop`: Stop the server. - - `status`: Check the server status. - - `restart`: Restart the server. - - `config`: Configure the server. - """ - try: - lc_all_val = os.environ["LC_ALL"] - if lc_all_val != "C": - raise ValueError(f"The 'LC_ALL' environment variable is currently set to {lc_all_val} but it must be set to 'C'.") - except KeyError: - LOG.debug("The 'LC_ALL' environment variable was not set. Setting this to 'C'.") - os.environ["LC_ALL"] = "C" # Necessary for Redis to configure LOCALE - - if args.commands == "init": - init_server() - elif args.commands == "start": - start_server() - elif args.commands == "stop": - stop_server() - elif args.commands == "status": - status_server() - elif args.commands == "restart": - restart_server() - elif args.commands == "config": - config_server(args) - - -def process_database(args: Namespace): - """ - Process database commands by routing to the correct function. - - Args: - args: An argparse Namespace containing user arguments. - """ - if args.local: - initialize_config(local_mode=True) - - if args.commands == "info": - database_info() - elif args.commands == "get": - database_get(args) - elif args.commands == "delete": - database_delete(args) - - -# Pylint complains that there's too many statements here and wants us -# to split the function up but that wouldn't make much sense so we ignore it -def setup_argparse() -> None: # pylint: disable=R0915 - """ - Set up the command-line argument parser for the Merlin package. - - This function configures the ArgumentParser for the Merlin CLI, allowing users - to interact with various commands related to workflow management and task handling. - It includes options for running a workflow, restarting tasks, purging task queues, - generating configuration files, and managing/configuring the server. - """ - parser: HelpParser = HelpParser( - prog="merlin", - description=banner_small, - formatter_class=RawDescriptionHelpFormatter, - epilog="See merlin --help for more info", - ) - parser.add_argument("-v", "--version", action="version", version=VERSION) - subparsers: ArgumentParser = parser.add_subparsers(dest="subparsers") - subparsers.required = True - - # merlin --level - parser.add_argument( - "-lvl", - "--level", - action="store", - dest="level", - type=str, - default=DEFAULT_LOG_LEVEL, - help="Set the log level. Options: DEBUG, INFO, WARNING, ERROR. [Default: %(default)s]", - ) - - # merlin run - run: ArgumentParser = subparsers.add_parser( - "run", - help="Run a workflow using a Merlin or Maestro YAML study " "specification.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - run.set_defaults(func=process_run) - run.add_argument("specification", type=str, help="Path to a Merlin or Maestro YAML file") - run.add_argument( - "--local", - action="store_const", - dest="run_mode", - const="local", - default="distributed", - help="Run locally instead of distributed", - ) - run.add_argument( - "--vars", - action="store", - dest="variables", - type=str, - nargs="+", - default=None, - help="Specify desired Merlin variable values to override those found in the specification. Space-delimited. " - "Example: '--vars LEARN=path/to/new_learn.py EPOCHS=3'", - ) - # TODO add all supported formats to doc string # pylint: disable=fixme - run.add_argument( - "--samplesfile", - action="store", - dest="samples_file", - type=str, - default=None, - help=f"Specify file containing samples. Valid choices: {ARRAY_FILE_FORMATS}", - ) - run.add_argument( - "--dry", - action="store_true", - dest="dry", - default=False, - help="Flag to dry-run a workflow, which sets up the workspace but does not launch tasks.", - ) - run.add_argument( - "--no-errors", - action="store_true", - dest="no_errors", - default=False, - help="Flag to ignore some flux errors for testing (often used with --dry --local).", - ) - run.add_argument( - "--pgen", - action="store", - dest="pgen_file", - type=str, - default=None, - help="Provide a pgen file to override global.parameters.", - ) - run.add_argument( - "--pargs", - type=str, - action="append", - default=[], - help="A string that represents a single argument to pass " - "a custom parameter generation function. Reuse '--parg' " - "to pass multiple arguments. [Use with '--pgen']", - ) - - # merlin restart - restart: ArgumentParser = subparsers.add_parser( - "restart", - help="Restart a workflow using an existing Merlin workspace.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - restart.set_defaults(func=process_restart) - restart.add_argument("restart_dir", type=str, help="Path to an existing Merlin workspace directory") - restart.add_argument( # TODO should this just be boolean instead of store_const? - "--local", - action="store_const", - dest="run_mode", - const="local", - default="distributed", - help="Run locally instead of distributed", - ) - - # merlin purge - purge: ArgumentParser = subparsers.add_parser( - "purge", - help="Remove all tasks from all merlin queues (default). " - "If a user would like to purge only selected queues use: " - "--steps to give a steplist, the queues will be defined from the step list", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - purge.set_defaults(func=purge_tasks) - purge.add_argument("specification", type=str, help="Path to a Merlin YAML spec file") - purge.add_argument( - "-f", - "--force", - action="store_true", - dest="purge_force", - default=False, - help="Purge the tasks without confirmation", - ) - purge.add_argument( - "--steps", - nargs="+", - type=str, - dest="purge_steps", - default=["all"], - help="The specific steps in the YAML file from which you want to purge the queues. \ - The input is a space separated list.", - ) - purge.add_argument( - "--vars", - action="store", - dest="variables", - type=str, - nargs="+", - default=None, - help="Specify desired Merlin variable values to override those found in the specification. Space-delimited. " - "Example: '--vars MY_QUEUE=hello'", - ) - - mconfig: ArgumentParser = subparsers.add_parser( - "config", - help="Create a default merlin server config file in ~/.merlin", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - mconfig.set_defaults(func=config_merlin) - # The below option makes it so the `config_path.txt` file is written to the test directory - mconfig.add_argument( - "-t", - "--test", - action="store_true", - help=SUPPRESS, # Hides from `--help` - ) - mconfig_subparsers = mconfig.add_subparsers(dest="commands", help="Subcommands for 'config'") - default_config_file = os.path.join(os.path.expanduser("~"), ".merlin", "app.yaml") - - # Subcommand: melrin config create - config_create_parser = mconfig_subparsers.add_parser("create", help="Create a new configuration file.") - config_create_parser.add_argument( - "--task-server", - type=str, - default="celery", - help="Task server type for which to create the config. Default: %(default)s", - ) - config_create_parser.add_argument( - "-o", - "--output-file", - dest="config_file", - type=str, - default=default_config_file, - help=f"Optional file name for your configuration. Default: {default_config_file}", - ) - config_create_parser.add_argument( - "--broker", - type=str, - default=None, - help="Optional broker type, backend will be redis. Default: rabbitmq", - ) - - # Subcommand: merlin config update-broker - config_broker_parser = mconfig_subparsers.add_parser("update-broker", help="Update broker settings in app.yaml") - config_broker_parser.add_argument( - "-t", - "--type", - required=True, - choices=["redis", "rabbitmq"], - help="Type of broker to configure (redis or rabbitmq).", - ) - config_broker_parser.add_argument( - "--cf", - "--config-file", - dest="config_file", - default=default_config_file, - help=f"The path to the config file that will be updated. Default: {default_config_file}", - ) - config_broker_parser.add_argument("-u", "--username", help="Broker username (only for rabbitmq)") - config_broker_parser.add_argument("--pf", "--password-file", dest="password_file", help="Path to password file") - config_broker_parser.add_argument("-s", "--server", help="The URL of the server") - config_broker_parser.add_argument("-p", "--port", type=int, help="Broker port") - config_broker_parser.add_argument("-v", "--vhost", help="Broker vhost (only for rabbitmq)") - config_broker_parser.add_argument("-c", "--cert-reqs", help="Broker cert requirements") - config_broker_parser.add_argument("-d", "--db-num", type=int, help="Redis database number (only for redis).") - - # Subcommand: merlin config update-backend - config_backend_parser = mconfig_subparsers.add_parser("update-backend", help="Update results backend settings in app.yaml") - config_backend_parser.add_argument( - "-t", - "--type", - required=True, - choices=["redis"], - help="Type of results backend to configure.", - ) - config_backend_parser.add_argument( - "--cf", - "--config-file", - dest="config_file", - default=default_config_file, - help=f"The path to the config file that will be updated. Default: {default_config_file}", - ) - config_backend_parser.add_argument("-u", "--username", help="Backend username") - config_backend_parser.add_argument("--pf", "--password-file", dest="password_file", help="Path to password file") - config_backend_parser.add_argument("-s", "--server", help="The URL of the server") - config_backend_parser.add_argument("-p", "--port", help="Backend port") - config_backend_parser.add_argument("-d", "--db-num", help="Backend database number") - config_backend_parser.add_argument("-c", "--cert-reqs", help="Backend cert requirements") - config_backend_parser.add_argument("-e", "--encryption-key", help="Path to encryption key file") - - # Subcommand: merlin config use - config_use_parser = mconfig_subparsers.add_parser("use", help="Use a different configuration file.") - config_use_parser.add_argument("config_file", type=str, help="The path to the new configuration file to use.") - - # merlin example - example: ArgumentParser = subparsers.add_parser( - "example", - help="Generate an example merlin workflow.", - formatter_class=RawTextHelpFormatter, - ) - example.add_argument( - "workflow", - action="store", - type=str, - help="The name of the example workflow to setup. Use 'merlin example list' to see available options.", - ) - example.add_argument( - "-p", - "--path", - action="store", - type=str, - default=None, - help="Specify a path to write the workflow to. Defaults to current working directory", - ) - example.set_defaults(func=process_example) - - generate_worker_touching_parsers(subparsers) - - generate_diagnostic_parsers(subparsers) - - # merlin server - server: ArgumentParser = subparsers.add_parser( - "server", - help="Manage broker and results server for merlin workflow.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - server.set_defaults(func=process_server) - - server_commands: ArgumentParser = server.add_subparsers(dest="commands") - - server_init: ArgumentParser = server_commands.add_parser( - "init", - help="Initialize merlin server resources.", - description="Initialize merlin server", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - server_init.set_defaults(func=process_server) - - server_status: ArgumentParser = server_commands.add_parser( - "status", - help="View status of the current server containers.", - description="View status", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - server_status.set_defaults(func=process_server) - - server_start: ArgumentParser = server_commands.add_parser( - "start", - help="Start a containerized server to be used as an broker and results server.", - description="Start server", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - server_start.set_defaults(func=process_server) - - server_stop: ArgumentParser = server_commands.add_parser( - "stop", - help="Stop an instance of redis containers currently running.", - description="Stop server.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - server_stop.set_defaults(func=process_server) - - server_stop: ArgumentParser = server_commands.add_parser( - "restart", - help="Restart merlin server instance", - description="Restart server.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - server_stop.set_defaults(func=process_server) - - server_config: ArgumentParser = server_commands.add_parser( - "config", - help="Making configurations for to the merlin server instance.", - description="Config server.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - server_config.add_argument( - "-ip", - "--ipaddress", - action="store", - type=str, - # default="127.0.0.1", - help="Set the binded IP address for the merlin server container.", - ) - server_config.add_argument( - "-p", - "--port", - action="store", - type=int, - # default=6379, - help="Set the binded port for the merlin server container.", - ) - server_config.add_argument( - "-pwd", - "--password", - action="store", - type=str, - # default="~/.merlin/redis.pass", - help="Set the password file to be used for merlin server container.", - ) - server_config.add_argument( - "--add-user", - action="store", - nargs=2, - type=str, - help="Create a new user for merlin server instance. (Provide both username and password)", - ) - server_config.add_argument("--remove-user", action="store", type=str, help="Remove an exisiting user.") - server_config.add_argument( - "-d", - "--directory", - action="store", - type=str, - # default="./", - help="Set the working directory of the merlin server container.", - ) - server_config.add_argument( - "-ss", - "--snapshot-seconds", - action="store", - type=int, - # default=300, - help="Set the number of seconds merlin server waits before checking if a snapshot is needed.", - ) - server_config.add_argument( - "-sc", - "--snapshot-changes", - action="store", - type=int, - # default=100, - help="Set the number of changes that are required to be made to the merlin server before a snapshot is made.", - ) - server_config.add_argument( - "-sf", - "--snapshot-file", - action="store", - type=str, - # default="dump.db", - help="Set the snapshot filename for database dumps.", - ) - server_config.add_argument( - "-am", - "--append-mode", - action="store", - type=str, - # default="everysec", - help="The appendonly mode to be set. The avaiable options are always, everysec, no.", - ) - server_config.add_argument( - "-af", - "--append-file", - action="store", - type=str, - # default="appendonly.aof", - help="Set append only filename for merlin server container.", - ) - - # merlin database - database: ArgumentParser = subparsers.add_parser( - "database", - help="Interact with Merlin's database.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - database.set_defaults(func=process_database) - - database.add_argument( - "-l", - "--local", - action="store_true", - help="Use the local SQLite database for this command.", - ) - - database_commands: ArgumentParser = database.add_subparsers(dest="commands") - - # Subcommand: database info - database_commands.add_parser( - "info", - help="Print information about the database.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - - # Subcommand: database delete - db_delete: ArgumentParser = database_commands.add_parser( - "delete", - help="Delete information stored in the database.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - - # Add subcommands for delete - delete_subcommands = db_delete.add_subparsers(dest="delete_type", required=True) - - # TODO enable support for deletion of study by passing in spec file - # Subcommand: delete study - delete_study = delete_subcommands.add_parser( - "study", - help="Delete one or more studies by ID or name.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - delete_study.add_argument( - "study", - type=str, - nargs="+", - help="A space-delimited list of IDs or names of studies to delete.", - ) - delete_study.add_argument( - "-k", - "--keep-associated-runs", - action="store_true", - help="Keep runs associated with the studies.", - ) - - # Subcommand: delete run - delete_run = delete_subcommands.add_parser( - "run", - help="Delete one or more runs by ID or workspace.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - delete_run.add_argument( - "run", - type=str, - nargs="+", - help="A space-delimited list of IDs or workspaces of runs to delete.", - ) - # TODO implement the below option; this removes the output workspace from file system - # delete_run.add_argument( - # "--delete-workspace", - # action="store_true", - # help="Delete the output workspace for the run.", - # ) - - # Subcommand: delete logical-worker - delete_logical_worker = delete_subcommands.add_parser( - "logical-worker", - help="Delete one or more logical workers by ID.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - delete_logical_worker.add_argument( - "worker", - type=str, - nargs="+", - help="A space-delimited list of IDs of logical workers to delete.", - ) - - # Subcommand: delete physical-worker - delete_physical_worker = delete_subcommands.add_parser( - "physical-worker", - help="Delete one or more physical workers by ID or name.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - delete_physical_worker.add_argument( - "worker", - type=str, - nargs="+", - help="A space-delimited list of IDs of physical workers to delete.", - ) - - # Subcommand: delete all-studies - delete_all_studies = delete_subcommands.add_parser( - "all-studies", - help="Delete all studies from the database.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - delete_all_studies.add_argument( - "-k", - "--keep-associated-runs", - action="store_true", - help="Keep runs associated with the studies.", - ) - - # Subcommand: delete all-runs - delete_subcommands.add_parser( - "all-runs", - help="Delete all runs from the database.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - - # Subcommand: delete all-logical-workers - delete_subcommands.add_parser( - "all-logical-workers", - help="Delete all logical workers from the database.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - - # Subcommand: delete all-physical-workers - delete_subcommands.add_parser( - "all-physical-workers", - help="Delete all physical workers from the database.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - - # Subcommand: delete everything - delete_everything = delete_subcommands.add_parser( - "everything", - help="Delete everything from the database.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - delete_everything.add_argument( - "-f", - "--force", - action="store_true", - help="Delete everything in the database without confirmation.", - ) - - # Subcommand: database get - db_get: ArgumentParser = database_commands.add_parser( - "get", - help="Get information stored in the database.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - - # Add subcommands for get - get_subcommands = db_get.add_subparsers(dest="get_type", required=True) - - # TODO enable support for retrieval of study by passing in spec file - # Subcommand: get study - get_study = get_subcommands.add_parser( - "study", - help="Get one or more studies by ID or name.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - get_study.add_argument( - "study", - type=str, - nargs="+", - help="A space-delimited list of IDs or names of the studies to get.", - ) - - # Subcommand: get run - get_run = get_subcommands.add_parser( - "run", - help="Get one or more runs by ID or workspace.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - get_run.add_argument( - "run", - type=str, - nargs="+", - help="A space-delimited list of IDs or workspaces of the runs to get.", - ) - - # Subcommand get logical-worker - get_logical_worker = get_subcommands.add_parser( - "logical-worker", - help="Get one or more logical workers by ID.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - get_logical_worker.add_argument( - "worker", - type=str, - nargs="+", - help="A space-delimited list of IDs of the logical workers to get.", - ) - - # Subcommand get physical-worker - get_physical_worker = get_subcommands.add_parser( - "physical-worker", - help="Get one or more physical workers by ID or name.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - get_physical_worker.add_argument( - "worker", - type=str, - nargs="+", - help="A space-delimited list of IDs or names of the physical workers to get.", - ) - - # Subcommand: get all-studies - get_subcommands.add_parser( - "all-studies", - help="Get all studies from the database.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - - # Subcommand: get all-runs - get_subcommands.add_parser( - "all-runs", - help="Get all runs from the database.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - - # Subcommand: get all-logical-workers - get_subcommands.add_parser( - "all-logical-workers", - help="Get all logical workers from the database.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - - # Subcommand: get all-physical-workers - get_subcommands.add_parser( - "all-physical-workers", - help="Get all physical workers from the database.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - - # Subcommand: get everything - get_subcommands.add_parser( - "everything", - help="Get everything from the database.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - - return parser - - -def generate_worker_touching_parsers(subparsers: ArgumentParser) -> None: - """ - Generate command-line argument parsers for managing worker operations. - - This function sets up subparsers for CLI commands that directly control or invoke - workers in the context of the Merlin framework. It provides options for running, - querying, stopping, and monitoring workers associated with a Merlin YAML study - specification. - - Args: - subparsers: An instance of ArgumentParser for adding command-line subcommands - related to worker management. - """ - # merlin run-workers - run_workers: ArgumentParser = subparsers.add_parser( - "run-workers", - help="Run the workers associated with the Merlin YAML study " - "specification. Does -not- queue tasks, just workers tied " - "to the correct queues.", - formatter_class=ArgumentDefaultsHelpFormatter, - ) - run_workers.set_defaults(func=launch_workers) - run_workers.add_argument("specification", type=str, help="Path to a Merlin YAML spec file") - run_workers.add_argument( - "--worker-args", - type=str, - dest="worker_args", - default="", - help="celery worker arguments in quotes.", - ) - run_workers.add_argument( - "--steps", - nargs="+", - type=str, - dest="worker_steps", - default=["all"], - help="The specific steps in the YAML file you want workers for", - ) - run_workers.add_argument( - "--echo", - action="store_true", - default=False, - dest="worker_echo_only", - help="Just echo the command; do not actually run it", - ) - run_workers.add_argument( - "--vars", - action="store", - dest="variables", - type=str, - nargs="+", - default=None, - help="Specify desired Merlin variable values to override those found in the specification. Space-delimited. " - "Example: '--vars LEARN=path/to/new_learn.py EPOCHS=3'", - ) - run_workers.add_argument( - "--disable-logs", - action="store_true", - help="Turn off the logs for the celery workers. Note: having the -l flag " - "in your workers' args section will overwrite this flag for that worker.", - ) - - # merlin query-workers - query: ArgumentParser = subparsers.add_parser("query-workers", help="List connected task server workers.") - query.set_defaults(func=query_workers) - query.add_argument( - "--task_server", - type=str, - default="celery", - help="Task server type from which to query workers.\ - Default: %(default)s", - ) - query.add_argument( - "--spec", - type=str, - default=None, - help="Path to a Merlin YAML spec file from which to read worker names to query.", - ) - query.add_argument("--queues", type=str, default=None, nargs="+", help="Specific queues to query workers from.") - query.add_argument( - "--workers", - type=str, - action="store", - nargs="+", - default=None, - help="Regex match for specific workers to query.", - ) - - # merlin stop-workers - stop: ArgumentParser = subparsers.add_parser("stop-workers", help="Attempt to stop all task server workers.") - stop.set_defaults(func=stop_workers) - stop.add_argument( - "--spec", - type=str, - default=None, - help="Path to a Merlin YAML spec file from which to read worker names to stop.", - ) - stop.add_argument( - "--task_server", - type=str, - default="celery", - help="Task server type from which to stop workers.\ - Default: %(default)s", - ) - stop.add_argument("--queues", type=str, default=None, nargs="+", help="specific queues to stop") - stop.add_argument( - "--workers", - type=str, - action="store", - nargs="+", - default=None, - help="regex match for specific workers to stop", - ) - - # merlin monitor - monitor: ArgumentParser = subparsers.add_parser( - "monitor", - help="Check for active workers on an allocation.", - formatter_class=RawTextHelpFormatter, - ) - monitor.add_argument("specification", type=str, help="Path to a Merlin YAML spec file") - monitor.add_argument( - "--steps", - nargs="+", - type=str, - dest="steps", - default=["all"], - help="The specific steps (tasks on the server) in the YAML file defining the queues you want to monitor", - ) - monitor.add_argument( - "--vars", - action="store", - dest="variables", - type=str, - nargs="+", - default=None, - help="Specify desired Merlin variable values to override those found in the specification. Space-delimited. " - "Example: '--vars LEARN=path/to/new_learn.py EPOCHS=3'", - ) - monitor.add_argument( - "--task_server", - type=str, - default="celery", - help="Task server type for which to monitor the workers.\ - Default: %(default)s", - ) - monitor.add_argument( - "--sleep", - type=int, - default=60, - help="Sleep duration between checking for workers.\ - Default: %(default)s", - ) - monitor.set_defaults(func=process_monitor) - - -def generate_diagnostic_parsers(subparsers: ArgumentParser): - """ - Generate command-line argument parsers for diagnostic operations in the Merlin framework. - - This function sets up subparsers for CLI commands that handle diagnostics related - to Merlin jobs. It provides options to check the status of studies, gather queue - statistics, and retrieve configuration information, making it easier for users to - diagnose issues with their workflows. - - Args: - subparsers: An instance of ArgumentParser that will be used to add command-line - subcommands for various diagnostic activities. - """ - # merlin status - status_cmd: ArgumentParser = subparsers.add_parser( - "status", - help="Display a summary of the status of a study.", - ) - status_cmd.set_defaults(func=query_status, detailed=False) - status_cmd.add_argument("spec_or_workspace", type=str, help="Path to a Merlin YAML spec file or a launched Merlin study") - status_cmd.add_argument( - "--cb-help", action="store_true", help="Colorblind help; uses different symbols to represent different statuses" - ) - status_cmd.add_argument( - "--dump", type=str, help="Dump the status to a file. Provide the filename (must be .csv or .json).", default=None - ) - status_cmd.add_argument( - "--no-prompts", - action="store_true", - help="Ignore any prompts provided. This will default to the latest study \ - if you provide a spec file rather than a study workspace.", - ) - status_cmd.add_argument( - "--task_server", - type=str, - default="celery", - help="Task server type.\ - Default: %(default)s", - ) - status_cmd.add_argument( - "-o", - "--output-path", - action="store", - type=str, - default=None, - help="Specify a location to look for output workspaces. Only used when a spec file is passed as the argument " - "to 'status'; this will NOT be used if an output workspace is passed as the argument.", - ) - - # merlin detailed-status - detailed_status: ArgumentParser = subparsers.add_parser( - "detailed-status", - help="Display a task-by-task status of a study.", - ) - detailed_status.set_defaults(func=query_status, detailed=True) - detailed_status.add_argument( - "spec_or_workspace", type=str, help="Path to a Merlin YAML spec file or a launched Merlin study" - ) - detailed_status.add_argument( - "--dump", type=str, help="Dump the status to a file. Provide the filename (must be .csv or .json).", default=None - ) - detailed_status.add_argument( - "--task_server", - type=str, - default="celery", - help="Task server type.\ - Default: %(default)s", - ) - detailed_status.add_argument( - "-o", - "--output-path", - action="store", - type=str, - default=None, - help="Specify a location to look for output workspaces. Only used when a spec file is passed as the argument " - "to 'status'; this will NOT be used if an output workspace is passed as the argument.", - ) - status_filter_group = detailed_status.add_argument_group("filter options") - status_filter_group.add_argument( - "--max-tasks", action="store", type=int, help="Sets a limit on how many tasks can be displayed" - ) - status_filter_group.add_argument( - "--return-code", - action="store", - nargs="+", - type=str, - choices=VALID_RETURN_CODES, - help="Filter which tasks to display based on their return code", - ) - status_filter_group.add_argument( - "--steps", - nargs="+", - type=str, - dest="steps", - default=["all"], - help="Filter which tasks to display based on the steps they're associated with", - ) - status_filter_group.add_argument( - "--task-queues", - nargs="+", - type=str, - help="Filter which tasks to display based on the task queue they're in", - ) - status_filter_group.add_argument( - "--task-status", - action="store", - nargs="+", - type=str, - choices=VALID_STATUS_FILTERS, - help="Filter which tasks to display based on their status", - ) - status_filter_group.add_argument( - "--workers", - nargs="+", - type=str, - help="Filter which tasks to display based on which workers are processing them", - ) - status_display_group = detailed_status.add_argument_group("display options") - status_display_group.add_argument( - "--disable-pager", action="store_true", help="Turn off the pager functionality when viewing the status" - ) - status_display_group.add_argument( - "--disable-theme", - action="store_true", - help="Turn off styling for the status layout (If you want styling but it's not working, try modifying " - "the MANPAGER or PAGER environment variables to be 'less -r'; i.e. export MANPAGER='less -r')", - ) - status_display_group.add_argument( - "--layout", - type=str, - choices=status_renderer_factory.get_layouts(), - default="default", - help="Alternate status layouts [Default: %(default)s]", - ) - status_display_group.add_argument( - "--no-prompts", - action="store_true", - help="Ignore any prompts provided. This will default to the latest study \ - if you provide a spec file rather than a study workspace.", - ) - - # merlin queue-info - queue_info: ArgumentParser = subparsers.add_parser( - "queue-info", - help="List queue statistics (queue name, number of tasks in the queue, number of connected workers).", - ) - queue_info.set_defaults(func=query_queues) - queue_info.add_argument( - "--dump", - type=str, - help="Dump the queue information to a file. Provide the filename (must be .csv or .json)", - default=None, - ) - queue_info.add_argument( - "--specific-queues", nargs="+", type=str, help="Display queue stats for specific queues you list here" - ) - queue_info.add_argument( - "--task_server", - type=str, - default="celery", - help="Task server type. Default: %(default)s", - ) - spec_group = queue_info.add_argument_group("specification options") - spec_group.add_argument( - "--spec", - dest="specification", - type=str, - help="Path to a Merlin YAML spec file. \ - This will only display information for queues defined in this spec file. \ - This is the same behavior as the status command prior to Merlin version 1.11.0.", - ) - spec_group.add_argument( - "--steps", - nargs="+", - type=str, - dest="steps", - default=["all"], - help="The specific steps in the YAML file you want to query the queues of. " - "This option MUST be used with the --spec option", - ) - spec_group.add_argument( - "--vars", - action="store", - dest="variables", - type=str, - nargs="+", - default=None, - help="Specify desired Merlin variable values to override those found in the specification. Space-delimited. " - "This option MUST be used with the --spec option. Example: '--vars LEARN=path/to/new_learn.py EPOCHS=3'", - ) - - # merlin info - info: ArgumentParser = subparsers.add_parser( - "info", - help="display info about the merlin configuration and the python configuration. Useful for debugging.", - ) - info.set_defaults(func=print_info) def main(): @@ -1704,7 +29,7 @@ def main(): no arguments are provided and performs error handling for any exceptions that may occur during command execution. """ - parser = setup_argparse() + parser = build_main_parser() if len(sys.argv) == 1: parser.print_help(sys.stdout) return 1 @@ -1714,12 +39,12 @@ def main(): try: args.func(args) - # pylint complains that this exception is too broad - being at the literal top of the program stack, - # it's ok. + # pylint complains that this exception is too broad - being at the literal top of the program stack, it's ok. except Exception as excpt: # pylint: disable=broad-except LOG.debug(traceback.format_exc()) LOG.error(str(excpt)) sys.exit(1) + # All paths in a function ought to return an exit code, or none of them should. Given the # distributed nature of Merlin, maybe it doesn't make sense for it to exit 0 until the work is completed, but # if the work is dispatched with no errors, that is a 'successful' Merlin run - any other failures are runtime. diff --git a/merlin/monitor/monitor.py b/merlin/monitor/monitor.py index d35095e2f..f041e95d0 100644 --- a/merlin/monitor/monitor.py +++ b/merlin/monitor/monitor.py @@ -48,8 +48,10 @@ class Monitor: Attributes: spec (MerlinSpec): The Merlin specification that defines the workflow. sleep (int): The interval (in seconds) between monitoring checks. + no_restart (bool): If True, the monitor will not try to restart the workflow. task_server_monitor (TaskServerMonitor): A monitor for interacting with whichever task server that the user is utilizing. + merlin_db (MerlinDatabase): Interface for accessing and querying the Merlin database. Methods: monitor_all_runs: Monitors all runs of the current study until they are complete. @@ -57,7 +59,7 @@ class Monitor: restart_workflow: Restart a run of a workflow. """ - def __init__(self, spec: MerlinSpec, sleep: int, task_server: str): + def __init__(self, spec: MerlinSpec, sleep: int, task_server: str, no_restart: bool): """ Initializes the `Monitor` instance with the given Merlin specification, sleep interval, and task server type. The task server monitor is created using the @@ -67,9 +69,11 @@ def __init__(self, spec: MerlinSpec, sleep: int, task_server: str): spec (MerlinSpec): The Merlin specification that defines the workflow. sleep (int): The interval (in seconds) between monitoring checks. task_server (str): The type of task server being used (e.g., "celery"). + no_restart (bool): If True, the monitor will not try to restart the workflow. """ self.spec: MerlinSpec = spec self.sleep: int = sleep + self.no_restart: bool = no_restart self.task_server_monitor: TaskServerMonitor = monitor_factory.get_monitor(task_server) self.merlin_db = MerlinDatabase() @@ -112,6 +116,47 @@ def monitor_all_runs(self): index += 1 + def _check_task_activity(self, run: RunEntity) -> bool: + """ + Checks whether there is active task activity for the given run. + + This method first checks if there are any tasks in the task server's queues. If not, + it then checks whether any workers are currently processing tasks. If either of these + conditions is true, the method considers the workflow to be active and returns True. + + Args: + run (RunEntity): The run entity representing the workflow run to check for activity. + + Returns: + True if tasks are in the queues or being processed by workers, False otherwise. + """ + # Check if any tasks are currently in the queues + if self.task_server_monitor.check_tasks(run): + LOG.info("Monitor: Found tasks in queues, keeping allocation alive.") + return True + + # If no tasks are in the queues, check if workers are processing tasks + if self.task_server_monitor.check_workers_processing(run.get_queues()): + LOG.info("Monitor: Found workers processing tasks, keeping allocation alive.") + return True + + return False + + def _handle_transient_exception(self, exc: Exception): + """ + Handles transient exceptions that may occur during monitoring. + + This method logs the exception type, message, and full traceback, then + sleeps for the configured interval before retrying. It is designed to + gracefully handle recoverable errors such as Redis timeouts or broker issues. + + Args: + exc (Exception): The exception instance that was caught during execution. + """ + LOG.warning(f"{exc.__class__.__name__} occurred:\n{exc}") + LOG.warning(f"Full traceback:\n{traceback.format_exc()}") + time.sleep(self.sleep) + def monitor_single_run(self, run: RunEntity): """ Monitors a single run of a study until it completes to ensure that the allocation stays alive @@ -127,9 +172,7 @@ def monitor_single_run(self, run: RunEntity): LOG.info(f"Monitor: Monitoring run with workspace '{run_workspace}'...") # Wait for workers to spin up before checking on tasks - worker_names = [ - self.merlin_db.get("logical_worker", worker_id=worker_id).get_name() for worker_id in run.get_workers() - ] + worker_names = [self.merlin_db.get("logical_worker", worker_id=wid).get_name() for wid in run.get_workers()] LOG.info(f"Monitor: Waiting for the following workers to start: {worker_names}...") self.task_server_monitor.wait_for_workers(worker_names, self.sleep) LOG.info("Monitor: Workers have started.") @@ -139,37 +182,26 @@ def monitor_single_run(self, run: RunEntity): # Run worker health check (checks for dead workers and restarts them if necessary) self.task_server_monitor.run_worker_health_check(run.get_workers()) - # Check if any tasks are currently in the queues - active_tasks = self.task_server_monitor.check_tasks(run) - if active_tasks: - LOG.info("Monitor: Found tasks in queues, keeping allocation alive.") - else: - # If no tasks are in the queues, check if workers are processing tasks - active_tasks = self.task_server_monitor.check_workers_processing(run.get_queues()) - if active_tasks: - LOG.info("Monitor: Found workers processing tasks, keeping allocation alive.") + # Check if any tasks are currently in the queues or if workers are processing tasks + active_tasks = self._check_task_activity(run) + + run_complete = run.run_complete # Re-query db for this value # If no tasks are in the queues or being processed by workers and the run is not complete, we have a hanging # workflow so restart it - run_complete = run.run_complete # Re-query db for this value if not active_tasks and not run_complete: - self.restart_workflow(run) + if self.no_restart: + LOG.warning( + f"Monitor: Determined restart was required for '{run_workspace}' but auto-restart is disabled." + ) + else: + self.restart_workflow(run) if not run_complete: time.sleep(self.sleep) # The below exceptions do not modify the `run_complete` value so the loop should retry - except RedisTimeoutError as exc: - LOG.warning(f"Redis timed out:\n{exc}") - LOG.warning(f"Full traceback:\n{traceback.format_exc()}") - time.sleep(self.sleep) - except OperationalError as exc: - LOG.warning(f"Kombu raised an error:\n{exc}") - LOG.warning(f"Full traceback:\n{traceback.format_exc()}") - time.sleep(self.sleep) - except TimeoutError as exc: - LOG.warning(f"A standard TimeoutError has occurred:\n{exc}") - LOG.warning(f"Full traceback:\n{traceback.format_exc()}") - time.sleep(self.sleep) + except (RedisTimeoutError, OperationalError, TimeoutError) as exc: + self._handle_transient_exception(exc) LOG.info(f"Monitor: Run with workspace '{run_workspace}' has completed.") diff --git a/tests/unit/backends/test_utils.py b/tests/unit/backends/test_backend_utils.py similarity index 100% rename from tests/unit/backends/test_utils.py rename to tests/unit/backends/test_backend_utils.py diff --git a/tests/unit/cli/commands/test_command_entry_point.py b/tests/unit/cli/commands/test_command_entry_point.py new file mode 100644 index 000000000..08215cee5 --- /dev/null +++ b/tests/unit/cli/commands/test_command_entry_point.py @@ -0,0 +1,58 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `command_entry_point.py` file. +""" + +from argparse import ArgumentParser, Namespace + +import pytest + +from merlin.cli.commands.command_entry_point import CommandEntryPoint + + +def test_cannot_instantiate_abstract_class(): + """Ensure instantiating CommandEntryPoint directly raises TypeError.""" + with pytest.raises(TypeError): + CommandEntryPoint() + + +def test_concrete_subclass_must_implement_add_parser_and_process_command(): + """Ensure subclass missing methods raises TypeError.""" + + # Only implements add_parser + class IncompleteCommand(CommandEntryPoint): + def add_parser(self, subparsers: ArgumentParser): + pass + + with pytest.raises(TypeError): + IncompleteCommand() + + +def test_concrete_subclass_runs_successfully(): + """Test that a fully implemented subclass works as expected.""" + + class DummyCommand(CommandEntryPoint): + def __init__(self): + self.called_add = False + self.called_process = False + + def add_parser(self, subparsers: ArgumentParser): + self.called_add = True + + def process_command(self, args: Namespace): + self.called_process = True + + dummy = DummyCommand() + + # Add parser should run + dummy.add_parser(ArgumentParser()) + assert dummy.called_add is True + + # Process command should run + dummy.process_command(Namespace()) + assert dummy.called_process is True diff --git a/tests/unit/cli/commands/test_config.py b/tests/unit/cli/commands/test_config.py new file mode 100644 index 000000000..c59c259f5 --- /dev/null +++ b/tests/unit/cli/commands/test_config.py @@ -0,0 +1,152 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `config.py` file of the `cli/` folder. +""" + +import os +from argparse import ArgumentTypeError, Namespace, _SubParsersAction +from unittest.mock import MagicMock, patch + +import pytest +from pytest_mock import MockerFixture + +from merlin.cli.commands.config import ConfigCommand +from tests.fixture_types import FixtureCallable, FixtureStr + + +def test_add_parser_includes_all_subcommands(create_parser: FixtureCallable): + """ + Verify that the `config` command parser includes all expected subcommands: + `create`, `update-broker`, `update-backend`, and `use`. + + Args: + create_parser: A fixture to help create a parser. + """ + config_subparser = None + parser = create_parser(ConfigCommand()) + for action in parser._subparsers._actions: + if isinstance(action, _SubParsersAction): + config_parser = action.choices.get("config") + if config_parser: + config_subparser = config_parser + break + + assert config_subparser is not None, "Config subparser not found" + + help_text = config_subparser.format_help() + + assert "create" in help_text + assert "update-broker" in help_text + assert "update-backend" in help_text + assert "use" in help_text + + +def test_process_command_create_invokes_methods(mocker: MockerFixture): + """ + Ensure that running `config create` invokes the appropriate methods on MerlinConfigManager. + + Args: + mocker: PyTest mocker fixture. + """ + mock_config_manager_class = mocker.patch("merlin.cli.commands.config.MerlinConfigManager") + args = Namespace(commands="create", task_server="celery", config_file="dummy.yaml", test=False) + mock_config_manager = MagicMock() + mock_config_manager_class.return_value = mock_config_manager + + cmd = ConfigCommand() + cmd.process_command(args) + + mock_config_manager.create_template_config.assert_called_once() + mock_config_manager.save_config_path.assert_called_once() + + +def test_process_command_update_broker(mocker: MockerFixture): + """ + Ensure that running `config update-broker` invokes the `update_broker` method on `MerlinConfigManager`. + + Args: + mocker: PyTest mocker fixture. + """ + mock_config_manager_class = mocker.patch("merlin.cli.commands.config.MerlinConfigManager") + args = Namespace(commands="update-broker", config_file="dummy.yaml", type="redis") + mock_config_manager = MagicMock() + mock_config_manager_class.return_value = mock_config_manager + + with patch("builtins.open", create=True), patch("yaml.safe_load"): + cmd = ConfigCommand() + cmd.process_command(args) + + mock_config_manager.update_broker.assert_called_once() + + +def test_process_command_update_backend(mocker: MockerFixture): + """ + Ensure that running `config update-backend` invokes the `update_backend` method on `MerlinConfigManager`. + + Args: + mocker: PyTest mocker fixture. + """ + mock_config_manager_class = mocker.patch("merlin.cli.commands.config.MerlinConfigManager") + args = Namespace(commands="update-backend", config_file="dummy.yaml", type="redis") + mock_config_manager = MagicMock() + mock_config_manager_class.return_value = mock_config_manager + + with patch("builtins.open", create=True), patch("yaml.safe_load"): + cmd = ConfigCommand() + cmd.process_command(args) + + mock_config_manager.update_backend.assert_called_once() + + +def test_process_command_use(mocker: MockerFixture): + """ + Ensure that running `config use` sets the config file and calls `save_config_path` on `MerlinConfigManager`. + + Args: + mocker: PyTest mocker fixture. + """ + mock_config_manager_class = mocker.patch("merlin.cli.commands.config.MerlinConfigManager") + args = Namespace(commands="use", config_file="dummy.yaml") + mock_config_manager = MagicMock() + mock_config_manager_class.return_value = mock_config_manager + + with patch("builtins.open", create=True), patch("yaml.safe_load"): + cmd = ConfigCommand() + cmd.process_command(args) + + assert mock_config_manager.config_file == "dummy.yaml" + mock_config_manager.save_config_path.assert_called_once() + + +def test_process_command_raises_on_missing_file(): + """ + Verify that an `ArgumentTypeError` is raised if the specified config file does not exist. + """ + args = Namespace(commands="update-broker", config_file="nonexistent.yaml") + cmd = ConfigCommand() + + with pytest.raises(ArgumentTypeError, match="does not exist"): + cmd.process_command(args) + + +def test_process_command_raises_on_invalid_yaml(cli_testing_dir: FixtureStr): + """ + Verify that an `ArgumentTypeError` is raised if the config file contains invalid YAML. + + Args: + cli_testing_dir: The path to the temporary ouptut directory for cli tests. + """ + invalid_yaml = os.path.join(cli_testing_dir, "invalid.yaml") + with open(invalid_yaml, "w") as invalid_yaml_file: + invalid_yaml_file.write("foo: [bar") + + args = Namespace(commands="update-broker", config_file=str(invalid_yaml), type="redis") + + cmd = ConfigCommand() + with pytest.raises(ArgumentTypeError, match="is not a valid YAML file"): + cmd.process_command(args) diff --git a/tests/unit/cli/commands/test_database.py b/tests/unit/cli/commands/test_database.py new file mode 100644 index 000000000..b8fcdd798 --- /dev/null +++ b/tests/unit/cli/commands/test_database.py @@ -0,0 +1,157 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `database.py` file of the `cli/` folder. +""" + +from argparse import ArgumentParser, Namespace +from typing import List + +import pytest +from pytest_mock import MockerFixture + +from merlin.cli.commands.database import DatabaseCommand +from tests.fixture_types import FixtureCallable + + +@pytest.fixture +def parser(create_parser: FixtureCallable) -> ArgumentParser: + """ + Returns an `ArgumentParser` configured with the `database` command and its subcommands. + + Args: + create_parser: A fixture to help create a parser. + + Returns: + Parser with the `database` command and its subcommands registered. + """ + return create_parser(DatabaseCommand()) + + +def test_process_command_info_calls_info(mocker: MockerFixture): + """ + Ensure that when `commands` is `info`, database_info() is invoked. + + Args: + mocker: PyTest mocker fixture. + """ + mock_info = mocker.patch("merlin.cli.commands.database.database_info") + cmd = DatabaseCommand() + args = Namespace(commands="info", local=False) + cmd.process_command(args) + mock_info.assert_called_once() + + +def test_process_command_get_calls_get(mocker: MockerFixture): + """ + Ensure that when `commands` is `get`, database_get(args) is invoked. + + Args: + mocker: PyTest mocker fixture. + """ + mock_get = mocker.patch("merlin.cli.commands.database.database_get") + cmd = DatabaseCommand() + args = Namespace(commands="get", local=False) + cmd.process_command(args) + mock_get.assert_called_once_with(args) + + +def test_process_command_delete_calls_delete(mocker: MockerFixture): + """ + Ensure that when `commands` is `delete`, database_delete(args) is invoked. + + Args: + mocker: PyTest mocker fixture. + """ + mock_delete = mocker.patch("merlin.cli.commands.database.database_delete") + cmd = DatabaseCommand() + args = Namespace(commands="delete", local=False) + cmd.process_command(args) + mock_delete.assert_called_once_with(args) + + +def test_process_command_local_initializes_config(mocker: MockerFixture): + """ + Verify that the local flag triggers initialize_config(local_mode=True) before calling the info command. + + Args: + mocker: PyTest mocker fixture. + """ + mock_init_config = mocker.patch("merlin.cli.commands.database.initialize_config") + mock_info = mocker.patch("merlin.cli.commands.database.database_info") + + cmd = DatabaseCommand() + args = Namespace(commands="info", local=True) + cmd.process_command(args) + + mock_init_config.assert_called_once_with(local_mode=True) + mock_info.assert_called_once() + + +@pytest.mark.parametrize( + "command, args", + [ + ("info", ["database", "info"]), + ("get", ["database", "get", "study", "dummy_id"]), + ("delete", ["database", "delete", "study", "dummy_id"]), + ], +) +def test_add_parser_creates_expected_commands(parser: ArgumentParser, command: str, args: List[str]): + """ + Validate that the parser correctly sets `commands` for top-level database subcommands. + + Args: + parser: Parser with the `database` command and its subcommands registered. + command: The command to test against. + args: The arguments to give to the `database` parser. + """ + parsed = parser.parse_args(args) + assert parsed.commands == command + + +@pytest.mark.parametrize("command", ["get", "delete"]) +@pytest.mark.parametrize("subcmd", ["study", "run", "logical-worker", "physical-worker"]) +def test_subcommands_with_id(parser: ArgumentParser, command: str, subcmd: str): + """ + Test that subcommands requiring an ID are parsed correctly. + + Args: + parser: Parser with the `database` command and its subcommands registered. + command: The command to test against. + subcmd: The subcommand to test against. + """ + args = ["database", command, subcmd, "dummy-id"] + parsed = parser.parse_args(args) + + assert parsed.commands == command + + if command == "get": + assert parsed.get_type == subcmd + elif command == "delete": + assert parsed.delete_type == subcmd + + +@pytest.mark.parametrize("command", ["get", "delete"]) +@pytest.mark.parametrize("subcmd", ["all-studies", "all-runs", "all-logical-workers", "all-physical-workers", "everything"]) +def test_subcommands_without_id(parser: ArgumentParser, command: str, subcmd: str): + """ + Test that subcommands not requiring an ID are parsed correctly. + + Args: + parser: Parser with the `database` command and its subcommands registered. + command: The command to test against. + subcmd: The subcommand to test against. + """ + args = ["database", command, subcmd] + parsed = parser.parse_args(args) + + assert parsed.commands == command + + if command == "get": + assert parsed.get_type == subcmd + elif command == "delete": + assert parsed.delete_type == subcmd diff --git a/tests/unit/cli/commands/test_example.py b/tests/unit/cli/commands/test_example.py new file mode 100644 index 000000000..2c594c10a --- /dev/null +++ b/tests/unit/cli/commands/test_example.py @@ -0,0 +1,70 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `example.py` file of the `cli/` folder. +""" + +from argparse import Namespace + +from _pytest.capture import CaptureFixture +from pytest_mock import MockerFixture + +from merlin.cli.commands.example import ExampleCommand +from tests.fixture_types import FixtureCallable + + +def test_example_parser_sets_func(create_parser: FixtureCallable): + """ + Ensure the `example` command sets the correct default function. + + Args: + create_parser: A fixture to help create a parser. + """ + command = ExampleCommand() + parser = create_parser(command) + args = parser.parse_args(["example", "list"]) + assert hasattr(args, "func") + assert args.func.__name__ == command.process_command.__name__ + assert args.workflow == "list" + assert args.path is None + + +def test_process_command_list(capsys: CaptureFixture, mocker: MockerFixture): + """ + Verify that `list_examples()` is called when workflow is `list`. + + Args: + capsys: PyTest capsys fixture. + mocker: PyTest mocker fixture. + """ + mock_list = mocker.patch("merlin.cli.commands.example.list_examples", return_value="example_a\nexample_b") + args = Namespace(workflow="list", path=None) + + ExampleCommand().process_command(args) + + captured = capsys.readouterr() + assert "example_a" in captured.out + mock_list.assert_called_once() + + +def test_process_command_setup(capsys: CaptureFixture, mocker: MockerFixture): + """ + Verify that banner_small is printed and setup_example is called with correct args. + + Args: + capsys: PyTest capsys fixture. + mocker: PyTest mocker fixture. + """ + mock_setup = mocker.patch("merlin.cli.commands.example.setup_example") + mocker.patch("merlin.cli.commands.example.banner_small", "FAKE_BANNER") + + args = Namespace(workflow="test-workflow", path="/some/path") + ExampleCommand().process_command(args) + + captured = capsys.readouterr() + assert "FAKE_BANNER" in captured.out + mock_setup.assert_called_once_with("test-workflow", "/some/path") diff --git a/tests/unit/cli/commands/test_info.py b/tests/unit/cli/commands/test_info.py new file mode 100644 index 000000000..d9425b1c4 --- /dev/null +++ b/tests/unit/cli/commands/test_info.py @@ -0,0 +1,44 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `info.py` file of the `cli/` folder. +""" + +from argparse import Namespace + +from pytest_mock import MockerFixture + +from merlin.cli.commands.info import InfoCommand +from tests.fixture_types import FixtureCallable + + +def test_info_parser_sets_func(create_parser: FixtureCallable): + """ + Ensure the `info` command sets the correct default function. + + Args: + parser: Parser with the `info` command and its subcommands registered. + """ + command = InfoCommand() + parser = create_parser(command) + args = parser.parse_args(["info"]) + assert hasattr(args, "func") + assert args.func.__name__ == command.process_command.__name__ + + +def test_info_process_command_calls_display(mocker: MockerFixture): + """ + Ensure that `process_command` calls `display.print_info` with the given args. + + Args: + mocker: PyTest mocker fixture. + """ + mock_print_info = mocker.patch("merlin.display.print_info") + cmd = InfoCommand() + dummy_args = Namespace(foo="bar") + cmd.process_command(dummy_args) + mock_print_info.assert_called_once_with(dummy_args) diff --git a/tests/unit/cli/commands/test_monitor_entry_point.py b/tests/unit/cli/commands/test_monitor_entry_point.py new file mode 100644 index 000000000..c008d50f4 --- /dev/null +++ b/tests/unit/cli/commands/test_monitor_entry_point.py @@ -0,0 +1,102 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `monitor.py` file of the `cli/` folder. +""" + +import logging +from argparse import Namespace + +from _pytest.capture import CaptureFixture +from pytest_mock import MockerFixture + +from merlin.cli.commands.monitor import MonitorCommand +from tests.fixture_types import FixtureCallable + + +def test_add_parser_sets_up_monitor_command(create_parser: FixtureCallable): + """ + Ensure the `monitor` command sets the correct default function. + + Args: + create_parser: A fixture to help create a parser. + """ + command = MonitorCommand() + parser = create_parser(command) + args = parser.parse_args(["monitor", "spec.yaml"]) + assert hasattr(args, "func") + assert args.func.__name__ == command.process_command.__name__ + assert args.specification == "spec.yaml" + assert args.steps == ["all"] + assert args.variables is None + assert args.task_server == "celery" + assert args.sleep == 60 + assert not args.no_restart + + +def test_process_command_all_steps(mocker: MockerFixture): + """ + Test the case when `args.steps == ['all']` -> uses Monitor.monitor_all_runs(). + + Args: + mocker: PyTest mocker fixture. + """ + mock_spec = mocker.Mock() + mocker.patch("merlin.cli.commands.monitor.get_merlin_spec_with_override", return_value=(mock_spec, None)) + mocker.patch("time.sleep") + + mock_monitor = mocker.Mock() + monitor_class = mocker.patch("merlin.cli.commands.monitor.Monitor", return_value=mock_monitor) + + command = MonitorCommand() + args = Namespace( + specification="spec.yaml", + steps=["all"], + variables=None, + task_server="celery", + sleep=5, + no_restart=False, + ) + command.process_command(args) + + monitor_class.assert_called_once_with(mock_spec, 5, "celery", False) + mock_monitor.monitor_all_runs.assert_called_once() + + +def test_monitor_process_command_with_specific_steps(mocker: MockerFixture, caplog: CaptureFixture): + """ + Test the case when `args.steps != ['all']` -> uses `check_merlin_status()` in a loop. + + Args: + mocker: PyTest mocker fixture. + caplog: PyTest caplog fixture. + """ + caplog.set_level(logging.INFO) + + mock_spec = mocker.Mock() + mock_get_spec = mocker.patch("merlin.cli.commands.monitor.get_merlin_spec_with_override", return_value=(mock_spec, None)) + mock_sleep = mocker.patch("time.sleep") + + # simulate 2 iterations + mock_check_status = mocker.patch("merlin.cli.commands.monitor.check_merlin_status", side_effect=[True, True, False]) + + command = MonitorCommand() + args = Namespace( + specification="workflow.yaml", + steps=["step1"], + variables=None, + task_server="celery", + sleep=5, + no_restart=False, + ) + command.process_command(args) + + mock_get_spec.assert_called_once_with(args) + assert mock_sleep.call_count == 3 # 1 before loop, 2 in loop + assert mock_check_status.call_count == 3 + assert "Monitor: found tasks in queues and/or tasks being processed" in caplog.text + assert "Monitor: ... stop condition met" in caplog.text diff --git a/tests/unit/cli/commands/test_purge.py b/tests/unit/cli/commands/test_purge.py new file mode 100644 index 000000000..c6f86bba4 --- /dev/null +++ b/tests/unit/cli/commands/test_purge.py @@ -0,0 +1,96 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `purge.py` file of the `cli/` folder. +""" + +import logging +from argparse import Namespace + +from _pytest.capture import CaptureFixture +from pytest_mock import MockerFixture + +from merlin.cli.commands.purge import PurgeCommand +from tests.fixture_types import FixtureCallable + + +def test_add_parser_sets_up_purge_command(create_parser: FixtureCallable): + """ + Ensure the `purge` command sets the correct default function. + + Args: + create_parser: A fixture to help create a parser. + """ + command = PurgeCommand() + parser = create_parser(command) + args = parser.parse_args(["purge", "workflow.yaml"]) + assert hasattr(args, "func") + assert args.func.__name__ == command.process_command.__name__ + assert args.specification == "workflow.yaml" + assert args.purge_force is False + assert args.purge_steps == ["all"] + assert args.variables is None + + +def test_process_command_executes_purge(mocker: MockerFixture, caplog: CaptureFixture): + """ + Ensure `process_command` calls `purge_tasks` with expected args when using --force and specific steps. + + Args: + mocker: PyTest mocker fixture. + caplog: PyTest caplog fixture. + """ + caplog.set_level(logging.INFO) + + mock_spec = mocker.Mock() + mock_spec.merlin = {"resources": {"task_server": "celery"}} + + mocker.patch("merlin.cli.commands.purge.get_merlin_spec_with_override", return_value=(mock_spec, None)) + purge_tasks_mock = mocker.patch("merlin.cli.commands.purge.purge_tasks", return_value="mock_return") + + args = Namespace( + specification="workflow.yaml", + purge_force=True, + purge_steps=["step1", "step2"], + variables=None, + ) + + command = PurgeCommand() + command.process_command(args) + + purge_tasks_mock.assert_called_once_with("celery", mock_spec, True, ["step1", "step2"]) + assert "Purge return = mock_return" in caplog.text + + +def test_process_command_with_defaults(mocker: MockerFixture, caplog: CaptureFixture): + """ + Ensure `process_command` uses default values and purges correctly without --force or custom steps. + + Args: + mocker: PyTest mocker fixture. + caplog: PyTest caplog fixture. + """ + caplog.set_level(logging.INFO) + + mock_spec = mocker.Mock() + mock_spec.merlin = {"resources": {"task_server": "celery"}} + + mocker.patch("merlin.cli.commands.purge.get_merlin_spec_with_override", return_value=(mock_spec, None)) + purge_tasks_mock = mocker.patch("merlin.cli.commands.purge.purge_tasks", return_value="ok") + + args = Namespace( + specification="spec.yaml", + purge_force=False, + purge_steps=["all"], + variables=None, + ) + + command = PurgeCommand() + command.process_command(args) + + purge_tasks_mock.assert_called_once_with("celery", mock_spec, False, ["all"]) + assert "Purge return = ok" in caplog.text diff --git a/tests/unit/cli/commands/test_query_workers.py b/tests/unit/cli/commands/test_query_workers.py new file mode 100644 index 000000000..09152b3c2 --- /dev/null +++ b/tests/unit/cli/commands/test_query_workers.py @@ -0,0 +1,120 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `query_workers.py` file of the `cli/` folder. +""" + +import logging +from argparse import Namespace + +from _pytest.capture import CaptureFixture +from pytest_mock import MockerFixture + +from merlin.cli.commands.query_workers import QueryWorkersCommand +from tests.fixture_types import FixtureCallable + + +def test_add_parser_sets_up_query_workers_command(create_parser: FixtureCallable): + """ + Ensure the `query-workers` command sets the correct default function. + + Args: + create_parser: A fixture to help create a parser. + """ + command = QueryWorkersCommand() + parser = create_parser(command) + args = parser.parse_args(["query-workers"]) + assert hasattr(args, "func") + assert args.func.__name__ == command.process_command.__name__ + assert args.task_server == "celery" + assert args.spec is None + assert args.queues is None + assert args.workers is None + + +def test_process_command_without_spec(mocker: MockerFixture): + """ + Ensure `process_command` calls `query_workers` directly if no spec is provided. + + Args: + mocker: PyTest mocker fixture. + """ + query_workers_mock = mocker.patch("merlin.cli.commands.query_workers.query_workers") + + args = Namespace( + task_server="celery", + spec=None, + queues=["q1", "q2"], + workers=["worker1", "worker2"], + ) + + cmd = QueryWorkersCommand() + cmd.process_command(args) + + query_workers_mock.assert_called_once_with("celery", [], ["q1", "q2"], ["worker1", "worker2"]) + + +def test_process_command_with_spec(mocker: MockerFixture, caplog: CaptureFixture): + """ + Ensure `process_command` loads worker names from spec and passes them to `query_workers`. + + Args: + mocker: PyTest mocker fixture. + caplog: PyTest caplog fixture. + """ + caplog.set_level(logging.DEBUG) + + mock_spec = mocker.Mock() + mock_spec.get_worker_names.return_value = ["foo", "bar"] + + mocker.patch("merlin.cli.commands.query_workers.verify_filepath", return_value="some/path/spec.yaml") + mocker.patch("merlin.cli.commands.query_workers.MerlinSpec.load_specification", return_value=mock_spec) + query_workers_mock = mocker.patch("merlin.cli.commands.query_workers.query_workers") + + args = Namespace( + task_server="celery", + spec="workflow.yaml", + queues=None, + workers=None, + ) + + cmd = QueryWorkersCommand() + cmd.process_command(args) + + query_workers_mock.assert_called_once_with("celery", ["foo", "bar"], None, None) + assert "Searching for the following workers to stop" in caplog.text + + +def test_process_command_logs_warning_for_unexpanded_worker(mocker: MockerFixture, caplog: CaptureFixture): + """ + Ensure a warning is logged if a worker name from the spec contains `$`. + + Args: + mocker: PyTest mocker fixture. + caplog: PyTest caplog fixture. + """ + caplog.set_level(logging.WARNING) + + mock_spec = mocker.Mock() + mock_spec.get_worker_names.return_value = ["$ENV_VAR", "actual_worker"] + + mocker.patch("merlin.cli.commands.query_workers.verify_filepath", return_value="workflow.yaml") + mocker.patch("merlin.cli.commands.query_workers.MerlinSpec.load_specification", return_value=mock_spec) + query_workers_mock = mocker.patch("merlin.cli.commands.query_workers.query_workers") + + args = Namespace( + task_server="celery", + spec="workflow.yaml", + queues=None, + workers=None, + ) + + cmd = QueryWorkersCommand() + cmd.process_command(args) + + assert "Worker '$ENV_VAR' is unexpanded. Target provenance spec instead?" in caplog.text + query_workers_mock.assert_called_once_with("celery", ["$ENV_VAR", "actual_worker"], None, None) diff --git a/tests/unit/cli/commands/test_queue_info.py b/tests/unit/cli/commands/test_queue_info.py new file mode 100644 index 000000000..19ab9f23a --- /dev/null +++ b/tests/unit/cli/commands/test_queue_info.py @@ -0,0 +1,148 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `queue_info.py` file of the `cli/` folder. +""" + +from argparse import Namespace + +import pytest +from _pytest.capture import CaptureFixture +from pytest_mock import MockerFixture + +from merlin.cli.commands.queue_info import QueueInfoCommand +from tests.fixture_types import FixtureCallable + + +def test_add_parser_sets_up_queue_info_command(create_parser: FixtureCallable): + """ + Ensure the `queue-info` command sets the correct default function. + + Args: + create_parser: A fixture to help create a parser. + """ + command = QueueInfoCommand() + parser = create_parser(command) + args = parser.parse_args(["queue-info"]) + assert hasattr(args, "func") + assert args.func.__name__ == command.process_command.__name__ + assert args.task_server == "celery" + assert args.dump is None + assert args.specific_queues is None + assert args.specification is None + assert args.steps == ["all"] + assert args.variables is None + + +def test_process_command_no_spec_valid_args(mocker: MockerFixture, capsys: CaptureFixture): + """ + Test `process_command` executes successfully without a spec file. + + Args: + mocker: PyTest mocker fixture. + capsys: PyTest capsys fixture. + """ + query_mock = mocker.patch( + "merlin.cli.commands.queue_info.query_queues", + return_value={ + "queue1": {"jobs": 5, "consumers": 2}, + "queue2": {"jobs": 3, "consumers": 1}, + }, + ) + + args = Namespace(specification=None, steps=["all"], variables=None, dump=None, specific_queues=None, task_server="celery") + + cmd = QueueInfoCommand() + cmd.process_command(args) + + query_mock.assert_called_once_with("celery", None, ["all"], None) + + output = capsys.readouterr().out + assert "queue1" in output and "queue2" in output + + +def test_process_command_with_spec(mocker: MockerFixture): + """ + Test `process_command` behavior when a spec file is provided. + + Args: + mocker: PyTest mocker fixture. + """ + mock_spec = mocker.Mock() + mock_get_spec = mocker.patch( + "merlin.cli.commands.queue_info.get_merlin_spec_with_override", return_value=(mock_spec, None) + ) + query_mock = mocker.patch("merlin.cli.commands.queue_info.query_queues", return_value={}) + + args = Namespace( + specification="workflow.yaml", steps=["all"], variables=None, dump=None, specific_queues=None, task_server="celery" + ) + + cmd = QueueInfoCommand() + cmd.process_command(args) + + mock_get_spec.assert_called_once_with(args) + query_mock.assert_called_once_with("celery", mock_spec, ["all"], None) + + +def test_process_command_dumps_queue_info(mocker: MockerFixture): + """ + Test that queue information is correctly dumped to a file when `--dump` is provided. + + Args: + mocker: PyTest mocker fixture. + """ + queue_data = { + "queue1": {"jobs": 10, "consumers": 4}, + } + mocker.patch("merlin.cli.commands.queue_info.query_queues", return_value=queue_data) + dump_mock = mocker.patch("merlin.cli.commands.queue_info.dump_queue_info") + + args = Namespace( + specification=None, steps=["all"], variables=None, dump="output.json", specific_queues=None, task_server="celery" + ) + + cmd = QueueInfoCommand() + cmd.process_command(args) + + dump_mock.assert_called_once_with("celery", queue_data, "output.json") + + +def test_process_command_raises_on_bad_dump_extension(): + """ + Test that an unsupported file extension for `--dump` raises a ValueError. + """ + args = Namespace( + specification=None, steps=["all"], variables=None, dump="badfile.txt", specific_queues=None, task_server="celery" + ) + + with pytest.raises(ValueError, match="Unsupported file type"): + QueueInfoCommand().process_command(args) + + +def test_process_command_raises_on_steps_without_spec(): + """ + Test that using `--steps` without `--spec` raises a ValueError. + """ + args = Namespace( + specification=None, steps=["step1", "step2"], variables=None, dump=None, specific_queues=None, task_server="celery" + ) + + with pytest.raises(ValueError, match="--steps argument MUST be used with the --specification"): + QueueInfoCommand().process_command(args) + + +def test_process_command_raises_on_vars_without_spec(): + """ + Test that using `--vars` without `--spec` raises a ValueError. + """ + args = Namespace( + specification=None, steps=["all"], variables=["VAR=1"], dump=None, specific_queues=None, task_server="celery" + ) + + with pytest.raises(ValueError, match="--vars argument MUST be used with the --specification"): + QueueInfoCommand().process_command(args) diff --git a/tests/unit/cli/commands/test_restart.py b/tests/unit/cli/commands/test_restart.py new file mode 100644 index 000000000..ddd62486f --- /dev/null +++ b/tests/unit/cli/commands/test_restart.py @@ -0,0 +1,127 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `restart.py` file of the `cli/` folder. +""" + +from argparse import Namespace +from pathlib import Path + +import pytest +from pytest_mock import MockerFixture + +from merlin.cli.commands.restart import RestartCommand +from tests.fixture_types import FixtureCallable, FixtureStr + + +def test_add_parser_sets_up_restart_command(create_parser: FixtureCallable): + """ + Ensure the `restart` command registers the correct arguments and sets the default function. + + Args: + create_parser: A fixture to help create a parser. + """ + command = RestartCommand() + parser = create_parser(command) + args = parser.parse_args(["restart", "some/dir"]) + assert hasattr(args, "func") + assert args.func.__name__ == command.process_command.__name__ + assert args.restart_dir == "some/dir" + assert args.run_mode == "distributed" + + +def test_process_command_with_valid_restart_spec(mocker: MockerFixture, cli_testing_dir: FixtureStr): + """ + Test `process_command` runs successfully when one expanded spec file is found. + + Args: + mocker: PyTest mocker fixture. + cli_testing_dir: The path to the temporary ouptut directory for cli tests. + """ + # Set up directory and mock spec file + cli_testing_dir = Path(cli_testing_dir) + restart_dir = cli_testing_dir / "workspace" + merlin_info_dir = restart_dir / "merlin_info" + merlin_info_dir.mkdir(parents=True) + spec_file = merlin_info_dir / "restart.expanded.yaml" + spec_file.write_text("fake spec content") + + # Mocks + mock_verify_dirpath = mocker.patch("merlin.cli.commands.restart.verify_dirpath", return_value=str(restart_dir)) + mock_verify_filepath = mocker.patch("merlin.cli.commands.restart.verify_filepath", return_value=str(spec_file)) + mock_study = mocker.patch("merlin.cli.commands.restart.MerlinStudy") + mock_run = mocker.patch("merlin.cli.commands.restart.run_task_server") + mock_log = mocker.patch("merlin.cli.commands.restart.LOG") + + args = Namespace(restart_dir=str(restart_dir), run_mode="distributed") + cmd = RestartCommand() + cmd.process_command(args) + + mock_verify_dirpath.assert_called_once_with(str(restart_dir)) + mock_verify_filepath.assert_called_once_with(str(spec_file)) + mock_study.assert_called_once_with(str(spec_file), restart_dir=str(restart_dir)) + mock_run.assert_called_once_with(mock_study.return_value, "distributed") + mock_log.info.assert_called_once() + + +def test_process_command_initializes_config_in_local_mode(mocker: MockerFixture, cli_testing_dir: FixtureStr): + """ + Test that `initialize_config` is called with `local_mode=True` when `--local` is passed. + + Args: + mocker: PyTest mocker fixture. + cli_testing_dir: The path to the temporary ouptut directory for cli tests. + """ + cli_testing_dir = Path(cli_testing_dir) + restart_dir = cli_testing_dir / "restart" + merlin_info = restart_dir / "merlin_info" + merlin_info.mkdir(parents=True) + spec_file = merlin_info / "spec.expanded.yaml" + spec_file.write_text("fake content") + + mocker.patch("merlin.cli.commands.restart.verify_dirpath", return_value=str(restart_dir)) + mocker.patch("merlin.cli.commands.restart.verify_filepath", return_value=str(spec_file)) + mocker.patch("glob.glob", return_value=[str(spec_file)]) + mocker.patch("merlin.cli.commands.restart.MerlinStudy") + mock_run = mocker.patch("merlin.cli.commands.restart.run_task_server") + mock_init = mocker.patch("merlin.cli.commands.restart.initialize_config") + + args = Namespace(restart_dir=str(restart_dir), run_mode="local") + RestartCommand().process_command(args) + + mock_init.assert_called_once_with(local_mode=True) + mock_run.assert_called_once() + + +def test_process_command_raises_if_no_spec_found(mocker: MockerFixture): + """ + Test `process_command` raises ValueError when no matching spec file is found. + + Args: + mocker: PyTest mocker fixture. + """ + mocker.patch("merlin.cli.commands.restart.verify_dirpath", return_value="fake_dir") + mocker.patch("glob.glob", return_value=[]) + + args = Namespace(restart_dir="fake_dir", run_mode="distributed") + with pytest.raises(ValueError, match="does not match any provenance spec file"): + RestartCommand().process_command(args) + + +def test_process_command_raises_if_multiple_specs_found(mocker: MockerFixture): + """ + Test `process_command` raises ValueError when multiple spec files are found. + + Args: + mocker: PyTest mocker fixture. + """ + mocker.patch("merlin.cli.commands.restart.verify_dirpath", return_value="fake_dir") + mocker.patch("glob.glob", return_value=["file1.yaml", "file2.yaml"]) + + args = Namespace(restart_dir="fake_dir", run_mode="distributed") + with pytest.raises(ValueError, match="matches more than one provenance spec file"): + RestartCommand().process_command(args) diff --git a/tests/unit/cli/commands/test_run.py b/tests/unit/cli/commands/test_run.py new file mode 100644 index 000000000..2d9d66d0e --- /dev/null +++ b/tests/unit/cli/commands/test_run.py @@ -0,0 +1,159 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `run.py` file of the `cli/` folder. +""" + +from argparse import Namespace + +import pytest +from pytest_mock import MockerFixture + +from merlin.cli.commands.run import RunCommand +from tests.fixture_types import FixtureCallable + + +def test_add_parser_sets_up_run_command(create_parser: FixtureCallable): + """ + Ensure the `run` command sets the correct defaults and required arguments. + + Args: + create_parser: A fixture to help create a parser. + """ + command = RunCommand() + parser = create_parser(command) + args = parser.parse_args(["run", "study.yaml"]) + assert hasattr(args, "func") + assert args.func.__name__ == command.process_command.__name__ + assert args.specification == "study.yaml" + assert args.variables is None + assert args.samples_file is None + assert args.dry is False + assert args.no_errors is False + assert args.pgen_file is None + assert args.pargs == [] + assert args.run_mode == "distributed" + + +def test_process_command_runs_study_successfully(mocker: MockerFixture): + """ + Test full run command flow with mock MerlinStudy and MerlinDatabase. + + Args: + mocker: PyTest mocker fixture. + """ + mock_filepath = "/abs/path/study.yaml" + mocker.patch("merlin.cli.commands.run.verify_filepath", return_value=mock_filepath) + mocker.patch("merlin.cli.commands.run.parse_override_vars", return_value={"FOO": "bar"}) + + mock_study = mocker.Mock() + mock_study.expanded_spec.name = "study" + mock_study.workspace = "/some/workspace" + mock_study.expanded_spec.get_queue_list.return_value = ["queue1"] + mock_study.expanded_spec.get_task_queues.return_value = {"step1": "queue1"} + mock_study.expanded_spec.get_worker_step_map.return_value = {"workerA": ["step1"]} + mock_merlin_study = mocker.patch("merlin.cli.commands.run.MerlinStudy", return_value=mock_study) + + mock_db_instance = mocker.Mock() + mock_run_entity = mocker.Mock() + mock_logical_worker = mocker.Mock() + mock_run_entity.get_id.return_value = 1 + mock_logical_worker.get_id.return_value = 2 + mock_db_instance.create.side_effect = [mock_run_entity, mock_logical_worker] + mock_db = mocker.patch("merlin.cli.commands.run.MerlinDatabase", return_value=mock_db_instance) + + run_task_mock = mocker.patch("merlin.cli.commands.run.run_task_server") + + args = Namespace( + specification="study.yaml", + variables=["FOO=bar"], + samples_file=None, + dry=False, + no_errors=False, + pgen_file=None, + pargs=[], + run_mode="distributed", + ) + + RunCommand().process_command(args) + + mock_merlin_study.assert_called_once() + mock_db.assert_called_once() + run_task_mock.assert_called_once_with(mock_study, "distributed") + mock_logical_worker.add_run.assert_called_once_with(1) + mock_run_entity.add_worker.assert_called_once_with(2) + + +def test_process_command_with_local_mode_initializes_config(mocker: MockerFixture): + """ + Ensure local mode initializes local config and still runs task server. + + Args: + mocker: PyTest mocker fixture. + """ + # Mocks + mocker.patch("merlin.cli.commands.run.verify_filepath", return_value="study.yaml") + mocker.patch("merlin.cli.commands.run.parse_override_vars", return_value={}) + + mock_study = mocker.Mock() + mock_expanded_spec = mocker.Mock() + mock_expanded_spec.name = "study" + mock_expanded_spec.get_queue_list.return_value = ["q"] + mock_expanded_spec.get_task_queues.return_value = {"step1": "q"} + mock_expanded_spec.get_worker_step_map.return_value = {"workerA": ["step1"]} + mock_study.expanded_spec = mock_expanded_spec + mock_study.workspace = "/workspace" + mocker.patch("merlin.cli.commands.run.MerlinStudy", return_value=mock_study) + + mock_db = mocker.Mock() + mock_run_entity = mocker.Mock(get_id=mocker.Mock(return_value=1)) + mock_logical_worker = mocker.Mock(get_id=mocker.Mock(return_value=2)) + mock_db.create.side_effect = [mock_run_entity, mock_logical_worker] + mocker.patch("merlin.cli.commands.run.MerlinDatabase", return_value=mock_db) + + mock_initialize = mocker.patch("merlin.cli.commands.run.initialize_config") + mocker.patch("merlin.cli.commands.run.run_task_server") + + args = Namespace( + specification="study.yaml", + variables=None, + samples_file=None, + dry=False, + no_errors=False, + pgen_file=None, + pargs=[], + run_mode="local", + ) + + RunCommand().process_command(args) + + mock_initialize.assert_called_once_with(local_mode=True) + + +def test_process_command_raises_on_pargs_without_pgen(mocker: MockerFixture): + """ + Ensure a ValueError is raised when --pargs is given without --pgen. + + Args: + mocker: PyTest mocker fixture. + """ + mocker.patch("merlin.cli.commands.run.verify_filepath", return_value="study.yaml") + mocker.patch("merlin.cli.commands.run.parse_override_vars", return_value={}) + + args = Namespace( + specification="study.yaml", + variables=None, + samples_file=None, + dry=False, + no_errors=False, + pgen_file=None, + pargs=["foo"], + run_mode="distributed", + ) + + with pytest.raises(ValueError, match="Cannot use the 'pargs' parameter without specifying a 'pgen'!"): + RunCommand().process_command(args) diff --git a/tests/unit/cli/commands/test_run_workers.py b/tests/unit/cli/commands/test_run_workers.py new file mode 100644 index 000000000..8729822c5 --- /dev/null +++ b/tests/unit/cli/commands/test_run_workers.py @@ -0,0 +1,106 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `run_workers.py` file of the `cli/` folder. +""" + +from argparse import Namespace + +from _pytest.capture import CaptureFixture +from pytest_mock import MockerFixture + +from merlin.cli.commands.run_workers import RunWorkersCommand +from tests.fixture_types import FixtureCallable + + +def test_add_parser_sets_up_run_workers_command(create_parser: FixtureCallable): + """ + Test that the `run-workers` command registers all expected arguments and sets the correct defaults. + + Args: + create_parser: A fixture to help create a parser. + """ + command = RunWorkersCommand() + parser = create_parser(command) + args = parser.parse_args(["run-workers", "workflow.yaml"]) + assert hasattr(args, "func") + assert args.func.__name__ == command.process_command.__name__ + assert args.specification == "workflow.yaml" + assert args.worker_args == "" + assert args.worker_steps == ["all"] + assert args.worker_echo_only is False + assert args.variables is None + assert args.disable_logs is False + + +def test_process_command_launches_workers_and_creates_logical_workers(mocker: MockerFixture): + """ + Test `process_command` launches workers and creates logical worker entries in normal mode. + + Args: + mocker: PyTest mocker fixture. + """ + mock_spec = mocker.Mock() + mock_spec.get_task_queues.return_value = {"step1": "queue1", "step2": "queue2"} + mock_spec.get_worker_step_map.return_value = {"workerA": ["step1", "step2"]} + + mock_get_spec = mocker.patch( + "merlin.cli.commands.run_workers.get_merlin_spec_with_override", return_value=(mock_spec, "workflow.yaml") + ) + mock_launch = mocker.patch("merlin.cli.commands.run_workers.launch_workers", return_value="launched") + mock_db = mocker.patch("merlin.cli.commands.run_workers.MerlinDatabase") + mock_log = mocker.patch("merlin.cli.commands.run_workers.LOG") + + args = Namespace( + specification="workflow.yaml", + worker_args="--concurrency=4", + worker_steps=["step1"], + worker_echo_only=False, + variables=None, + disable_logs=False, + ) + + RunWorkersCommand().process_command(args) + + mock_get_spec.assert_called_once_with(args) + mock_db.return_value.create.assert_called_once_with("logical_worker", "workerA", {"queue1", "queue2"}) + mock_launch.assert_called_once_with(mock_spec, ["step1"], "--concurrency=4", False, False) + mock_log.info.assert_called_once_with("Launching workers from 'workflow.yaml'") + mock_log.debug.assert_called_once_with("celery command: launched") + + +def test_process_command_echo_only_mode_prints_command(mocker: MockerFixture, capsys: CaptureFixture): + """ + Test `process_command` prints the launch command and initializes config in echo-only mode. + + Args: + mocker: PyTest mocker fixture. + capsys: PyTest capsys fixture. + """ + mock_spec = mocker.Mock() + mock_spec.get_task_queues.return_value = {} + mock_spec.get_worker_step_map.return_value = {} + + mocker.patch("merlin.cli.commands.run_workers.get_merlin_spec_with_override", return_value=(mock_spec, "file.yaml")) + mocker.patch("merlin.cli.commands.run_workers.initialize_config") + mocker.patch("merlin.cli.commands.run_workers.MerlinDatabase") + mock_launch = mocker.patch("merlin.cli.commands.run_workers.launch_workers", return_value="echo-cmd") + + args = Namespace( + specification="spec.yaml", + worker_args="--autoscale=2,10", + worker_steps=["all"], + worker_echo_only=True, + variables=None, + disable_logs=False, + ) + + RunWorkersCommand().process_command(args) + + captured = capsys.readouterr() + assert "echo-cmd" in captured.out + mock_launch.assert_called_once_with(mock_spec, ["all"], "--autoscale=2,10", False, True) diff --git a/tests/unit/cli/commands/test_server.py b/tests/unit/cli/commands/test_server.py new file mode 100644 index 000000000..5a62e8f03 --- /dev/null +++ b/tests/unit/cli/commands/test_server.py @@ -0,0 +1,132 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `server.py` file of the `cli/` folder. +""" + +import os +from argparse import Namespace + +import pytest +from pytest_mock import MockerFixture + +from merlin.cli.commands.server import ServerCommand +from tests.fixture_types import FixtureCallable + + +def test_add_parser_sets_up_server_command(create_parser: FixtureCallable): + """ + Test that the `server` command parser sets up the expected defaults and subcommands. + + Args: + create_parser: A fixture to help create a parser. + """ + command = ServerCommand() + parser = create_parser(command) + args = parser.parse_args(["server", "init"]) + assert hasattr(args, "func") + assert args.func.__name__ == command.process_command.__name__ + assert args.commands == "init" + + +def test_process_command_calls_init(mocker: MockerFixture): + """ + Ensure `init` subcommand calls `init_server`. + + Args: + mocker: PyTest mocker fixture. + """ + mock = mocker.patch("merlin.cli.commands.server.init_server") + ServerCommand().process_command(Namespace(commands="init")) + mock.assert_called_once() + + +def test_process_command_calls_start(mocker: MockerFixture): + """ + Ensure `start` subcommand calls `start_server`. + + Args: + mocker: PyTest mocker fixture. + """ + mock = mocker.patch("merlin.cli.commands.server.start_server") + ServerCommand().process_command(Namespace(commands="start")) + mock.assert_called_once() + + +def test_process_command_calls_stop(mocker: MockerFixture): + """ + Ensure `stop` subcommand calls `stop_server`. + + Args: + mocker: PyTest mocker fixture. + """ + mock = mocker.patch("merlin.cli.commands.server.stop_server") + ServerCommand().process_command(Namespace(commands="stop")) + mock.assert_called_once() + + +def test_process_command_calls_status(mocker: MockerFixture): + """ + Ensure `status` subcommand calls `status_server`. + + Args: + mocker: PyTest mocker fixture. + """ + mock = mocker.patch("merlin.cli.commands.server.status_server") + ServerCommand().process_command(Namespace(commands="status")) + mock.assert_called_once() + + +def test_process_command_calls_restart(mocker: MockerFixture): + """ + Ensure `restart` subcommand calls `restart_server`. + + Args: + mocker: PyTest mocker fixture. + """ + mock = mocker.patch("merlin.cli.commands.server.restart_server") + ServerCommand().process_command(Namespace(commands="restart")) + mock.assert_called_once() + + +def test_process_command_calls_config(mocker: MockerFixture): + """ + Ensure `config` subcommand calls `config_server` with the provided args. + + Args: + mocker: PyTest mocker fixture. + """ + mock = mocker.patch("merlin.cli.commands.server.config_server") + args = Namespace(commands="config", ip="127.0.0.1", port=8888) + ServerCommand().process_command(args) + mock.assert_called_once_with(args) + + +def test_process_command_sets_lc_all_if_missing(mocker: MockerFixture): + """ + Ensure LC_ALL is set to 'C' if it's missing in the environment. + + Args: + mocker: PyTest mocker fixture. + """ + mocker.patch("merlin.cli.commands.server.start_server") + mocker.patch.dict("os.environ", {}, clear=True) + args = Namespace(commands="start") + ServerCommand().process_command(args) + assert os.environ["LC_ALL"] == "C" + + +def test_process_command_raises_if_lc_all_invalid(mocker: MockerFixture): + """ + Ensure a ValueError is raised if LC_ALL is set to a value other than 'C'. + + Args: + mocker: PyTest mocker fixture. + """ + mocker.patch.dict("os.environ", {"LC_ALL": "en_US.UTF-8"}) + with pytest.raises(ValueError, match="LC_ALL.*must be set to 'C'"): + ServerCommand().process_command(Namespace(commands="start")) diff --git a/tests/unit/cli/commands/test_status.py b/tests/unit/cli/commands/test_status.py new file mode 100644 index 000000000..4b239d9ce --- /dev/null +++ b/tests/unit/cli/commands/test_status.py @@ -0,0 +1,166 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `status.py` file of the `cli/` folder. +""" + +from argparse import Namespace + +import pytest + +from merlin.cli.commands.status import DetailedStatusCommand, StatusCommand +from tests.fixture_types import FixtureCallable + + +class TestStatusCommand: + """ + Tests for the `StatusCommand` object. + """ + + def test_add_parser_sets_up_status_command(self, create_parser: FixtureCallable): + """ + Test that the `status` command parser sets up the expected defaults and subcommands. + + Args: + create_parser: A fixture to help create a parser. + """ + command = StatusCommand() + parser = create_parser(command) + args = parser.parse_args(["status", "study.yaml"]) + assert hasattr(args, "func") + assert args.func.__name__ == command.process_command.__name__ + assert args.spec_or_workspace == "study.yaml" + assert not args.cb_help + assert args.dump is None + assert not args.no_prompts + assert args.task_server == "celery" + assert args.output_path is None + assert not args.detailed + + def test_process_command_invalid_task_server(self): + args = Namespace( + spec_or_workspace="path/to/spec.yaml", + task_server="dask", + dump=None, + detailed=False, + ) + + with pytest.raises(ValueError, match="only supported task server is celery"): + StatusCommand().process_command(args) + + def test_process_command_invalid_dump_extension(self): + args = Namespace( + spec_or_workspace="path/to/spec.yaml", + task_server="celery", + dump="invalid.txt", + detailed=False, + ) + + with pytest.raises(ValueError, match="must end with .csv or .json"): + StatusCommand().process_command(args) + + def test_process_command_with_valid_spec_file_and_status(self, mocker): + mock_verify_filepath = mocker.patch("merlin.cli.commands.status.verify_filepath", return_value="path/to/spec.yaml") + mock_get_spec = mocker.patch("merlin.cli.commands.status.get_spec_with_expansion") + mock_status_obj = mocker.Mock() + mock_status_cls = mocker.patch("merlin.cli.commands.status.Status", return_value=mock_status_obj) + + args = Namespace( + spec_or_workspace="path/to/spec.yaml", + task_server="celery", + dump=None, + detailed=False, + ) + + StatusCommand().process_command(args) + + mock_verify_filepath.assert_called_once_with("path/to/spec.yaml") + mock_get_spec.assert_called_once_with("path/to/spec.yaml") + mock_status_cls.assert_called_once() + mock_status_obj.display.assert_called_once() + + def test_process_command_dumps_status(self, mocker): + mocker.patch("merlin.cli.commands.status.verify_filepath", return_value="spec.yaml") + mocker.patch("merlin.cli.commands.status.get_spec_with_expansion") + mock_status_obj = mocker.Mock() + mocker.patch("merlin.cli.commands.status.Status", return_value=mock_status_obj) + + args = Namespace( + spec_or_workspace="spec.yaml", + task_server="celery", + dump="status.json", + detailed=False, + ) + + StatusCommand().process_command(args) + + mock_status_obj.dump.assert_called_once() + + def test_process_command_invalid_path_logs_error(self, mocker): + mocker.patch("merlin.cli.commands.status.verify_filepath", side_effect=ValueError) + mocker.patch("merlin.cli.commands.status.verify_dirpath", side_effect=ValueError) + mock_log = mocker.patch("merlin.cli.commands.status.LOG") + + args = Namespace( + spec_or_workspace="badpath", + task_server="celery", + dump=None, + detailed=False, + ) + + result = StatusCommand().process_command(args) + assert result is None + mock_log.error.assert_called_once_with("The file or directory path badpath does not exist.") + + +class TestDetailedStatusCommand: + """ + Tests for the `DetailedStatusCommand` object. + """ + + def test_add_parser_sets_up_detailed_status_command(self, create_parser: FixtureCallable): + """ + Test that the `detailed-status` command parser sets up the expected defaults and subcommands. + + Args: + create_parser: A fixture to help create a parser. + """ + command = DetailedStatusCommand() + parser = create_parser(command) + args = parser.parse_args(["detailed-status", "study.yaml"]) + assert hasattr(args, "func") + assert args.func.__name__ == command.process_command.__name__ + assert args.spec_or_workspace == "study.yaml" + assert args.dump is None + assert args.task_server == "celery" + assert args.output_path is None + assert args.layout == "default" + assert args.steps == ["all"] + assert not args.disable_pager + assert not args.disable_theme + assert not args.no_prompts + assert args.detailed + + def test_process_command_with_valid_workspace_and_detailed_status(self, mocker): + mock_verify_filepath = mocker.patch("merlin.cli.commands.status.verify_filepath", side_effect=ValueError) + mock_verify_dirpath = mocker.patch("merlin.cli.commands.status.verify_dirpath", return_value="path/to/workspace") + mock_detailed_obj = mocker.Mock() + mock_detailed_cls = mocker.patch("merlin.cli.commands.status.DetailedStatus", return_value=mock_detailed_obj) + + args = Namespace( + spec_or_workspace="path/to/workspace", + task_server="celery", + dump=None, + detailed=True, + ) + + StatusCommand().process_command(args) + + mock_verify_filepath.assert_called_once() + mock_verify_dirpath.assert_called_once_with("path/to/workspace") + mock_detailed_cls.assert_called_once_with(args, False, "path/to/workspace") + mock_detailed_obj.display.assert_called_once() diff --git a/tests/unit/cli/commands/test_stop_workers.py b/tests/unit/cli/commands/test_stop_workers.py new file mode 100644 index 000000000..970a0219f --- /dev/null +++ b/tests/unit/cli/commands/test_stop_workers.py @@ -0,0 +1,102 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `stop_workers.py` file of the `cli/` folder. +""" + +from argparse import Namespace + +from _pytest.capture import CaptureFixture +from pytest_mock import MockerFixture + +from merlin.cli.commands.stop_workers import StopWorkersCommand +from tests.fixture_types import FixtureCallable + + +def test_add_parser_sets_up_stop_workers_command(create_parser: FixtureCallable): + """ + Ensure the stop-workers command parser sets correct defaults and accepts args. + + Args: + create_parser: A fixture to help create a parser. + """ + command = StopWorkersCommand() + parser = create_parser(command) + args = parser.parse_args(["stop-workers", "--task_server", "celery", "--queues", "queue1", "queue2"]) + assert hasattr(args, "func") + assert args.func.__name__ == command.process_command.__name__ + assert args.task_server == "celery" + assert args.queues == ["queue1", "queue2"] + assert args.workers is None + assert args.spec is None + + +def test_process_command_calls_stop_workers_no_spec(mocker: MockerFixture): + """ + Ensure stop_workers is called when no spec is provided. + + Args: + mocker: PyTest mocker fixture. + """ + mocker.patch("merlin.cli.commands.stop_workers.banner_small", "BANNER") + mock_stop = mocker.patch("merlin.cli.commands.stop_workers.stop_workers") + + args = Namespace(spec=None, task_server="celery", queues=["q1"], workers=["worker1"]) + StopWorkersCommand().process_command(args) + + mock_stop.assert_called_once_with("celery", [], ["q1"], ["worker1"]) + + +def test_process_command_with_spec_and_worker_names(mocker: MockerFixture): + """ + Test loading a spec file, getting worker names, and calling stop_workers with them. + + Args: + mocker: PyTest mocker fixture. + """ + mocker.patch("merlin.cli.commands.stop_workers.banner_small", "BANNER") + mock_stop = mocker.patch("merlin.cli.commands.stop_workers.stop_workers") + mock_verify = mocker.patch("merlin.cli.commands.stop_workers.verify_filepath", return_value="study.yaml") + + mock_spec = mocker.patch("merlin.cli.commands.stop_workers.MerlinSpec") + mock_spec.load_specification.return_value.get_worker_names.return_value = ["worker.alpha", "worker.beta"] + + args = Namespace( + spec="study.yaml", + task_server="celery", + queues=None, + workers=None, + ) + StopWorkersCommand().process_command(args) + + mock_verify.assert_called_once_with("study.yaml") + mock_spec.load_specification.assert_called_once_with("study.yaml") + mock_stop.assert_called_once_with("celery", ["worker.alpha", "worker.beta"], None, None) + + +def test_process_command_logs_warning_on_unexpanded_worker(mocker: MockerFixture, caplog: CaptureFixture): + """ + Ensure warning is logged if unexpanded worker names are detected. + + Args: + mocker: PyTest mocker fixture. + caplog: PyTest caplog fixture. + """ + caplog.set_level("WARNING", logger="merlin") + + mocker.patch("merlin.cli.commands.stop_workers.banner_small", "BANNER") + mock_stop = mocker.patch("merlin.cli.commands.stop_workers.stop_workers") + mocker.patch("merlin.cli.commands.stop_workers.verify_filepath", return_value="spec.yaml") + + mock_spec = mocker.patch("merlin.cli.commands.stop_workers.MerlinSpec") + mock_spec.load_specification.return_value.get_worker_names.return_value = ["worker.1", "worker.$step"] + + args = Namespace(spec="spec.yaml", task_server="celery", queues=None, workers=None) + StopWorkersCommand().process_command(args) + + assert any("is unexpanded" in record.message for record in caplog.records) + mock_stop.assert_called_once_with("celery", ["worker.1", "worker.$step"], None, None) diff --git a/tests/unit/cli/conftest.py b/tests/unit/cli/conftest.py new file mode 100644 index 000000000..8ec423d67 --- /dev/null +++ b/tests/unit/cli/conftest.py @@ -0,0 +1,56 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Fixtures for files in this `cli/` test directory. +""" + +from argparse import ArgumentParser + +import pytest + +from merlin.cli.commands.command_entry_point import CommandEntryPoint +from tests.fixture_types import FixtureCallable, FixtureStr + + +@pytest.fixture(scope="session") +def cli_testing_dir(create_testing_dir: FixtureCallable, temp_output_dir: FixtureStr) -> FixtureStr: + """ + Fixture to create a temporary output directory for tests related to testing the + `cli` directory. + + Args: + create_testing_dir: A fixture which returns a function that creates the testing directory. + temp_output_dir: The path to the temporary ouptut directory we'll be using for this test run. + + Returns: + The path to the temporary testing directory for tests of files in the `cli` directory. + """ + return create_testing_dir(temp_output_dir, "cli_testing") + + +@pytest.fixture +def create_parser() -> FixtureCallable: + """ + A fixture to help create a parser for any command. + + Returns: + A function that creates a parser. + """ + + def _create_parser(cmd: CommandEntryPoint) -> ArgumentParser: + """ + Returns an `ArgumentParser` configured with the `cmd` command and its subcommands. + + Returns: + Parser with the `cmd` command and its subcommands registered. + """ + parser = ArgumentParser() + subparsers = parser.add_subparsers(dest="main_command") + cmd.add_parser(subparsers) + return parser + + return _create_parser diff --git a/tests/unit/cli/test_argparse_main.py b/tests/unit/cli/test_argparse_main.py new file mode 100644 index 000000000..5bfd1bd0d --- /dev/null +++ b/tests/unit/cli/test_argparse_main.py @@ -0,0 +1,103 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `argparse_main.py` file. +""" + +from argparse import ArgumentParser + +import pytest +from _pytest.capture import CaptureFixture +from pytest_mock import MockerFixture + +from merlin import VERSION +from merlin.cli.argparse_main import DEFAULT_LOG_LEVEL, HelpParser, build_main_parser +from tests.fixture_types import FixtureList + + +@pytest.fixture +def mock_all_commands(mocker: MockerFixture) -> FixtureList: + """ + Patch `ALL_COMMANDS` with dummy `CommandEntryPoint`-like objects + that define a working `add_parser` method. + + Args: + mocker: PyTest mocker fixture. + + Returns: + A list of `DummyCommand` objects to be used for tests. + """ + + class DummyCommand: + def __init__(self, name): + self.name = name + + def add_parser(self, subparsers): + subparsers.add_parser(self.name) + + dummy_commands = [DummyCommand("run"), DummyCommand("purge")] + mocker.patch("merlin.cli.argparse_main.ALL_COMMANDS", dummy_commands) + return dummy_commands + + +def test_help_parser_error(mocker: MockerFixture, capsys: CaptureFixture): + """ + Test that HelpParser.error prints help and exits with code 2. + + Args: + mocker: PyTest mocker fixture. + capsys: PyTest capsys fixture. + """ + mock_exit = mocker.patch("sys.exit", side_effect=SystemExit(2)) + + parser = HelpParser(prog="test") + with pytest.raises(SystemExit) as e: + parser.error("test error") + + captured = capsys.readouterr() + assert "error: test error" in captured.err + assert e.value.code == 2 + mock_exit.assert_called_once_with(2) + + +def test_build_main_parser_adds_all_commands(mock_all_commands: FixtureList): + """ + Test that build_main_parser adds subcommands from ALL_COMMANDS. + + Args: + mock_all_commands: A list of `DummyCommand` objects to be used for tests. + """ + parser = build_main_parser() + assert isinstance(parser, ArgumentParser) + + # Commands from mock are "run" and "purge", so we should be able to parse those + args = parser.parse_args(["--level", "DEBUG", "run"]) + assert args.level == "DEBUG" + assert args.subparsers == "run" + + args = parser.parse_args(["purge"]) + assert args.level == DEFAULT_LOG_LEVEL + assert args.subparsers == "purge" + + +def test_version_flag_prints_version_and_exits(mocker: MockerFixture, capsys: CaptureFixture, mock_all_commands: FixtureList): + """ + Test that --version prints the correct version and exits. + + Args: + mocker: PyTest mocker fixture. + capsys: PyTest capsys fixture. + mock_all_commands: A list of `DummyCommand` objects to be used for tests. + """ + mocker.patch("sys.exit", side_effect=SystemExit(0)) + parser = build_main_parser() + + with pytest.raises(SystemExit): + parser.parse_args(["--version"]) + + output = capsys.readouterr() + assert VERSION in output.out diff --git a/tests/unit/cli/test_cli_utils.py b/tests/unit/cli/test_cli_utils.py new file mode 100644 index 000000000..36b1633c6 --- /dev/null +++ b/tests/unit/cli/test_cli_utils.py @@ -0,0 +1,92 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `utils.py` file of the `cli/` folder. +""" + +from argparse import Namespace + +import pytest +from pytest_mock import MockerFixture + +from merlin.cli.utils import get_merlin_spec_with_override, parse_override_vars + + +class TestParseOverrideVars: + def test_returns_none_if_input_is_none(self): + """Should return None when the input variable list is None.""" + assert parse_override_vars(None) is None + + def test_parses_valid_string_and_int_values(self): + """Should parse valid KEY=value strings into a dictionary with proper types.""" + input_vars = ["FOO=bar", "COUNT=42"] + expected = {"FOO": "bar", "COUNT": 42} + assert parse_override_vars(input_vars) == expected + + def test_raises_if_missing_equal_sign(self): + """Should raise ValueError if '=' is missing in a variable assignment.""" + with pytest.raises(ValueError, match="requires '=' operator"): + parse_override_vars(["FOO42"]) + + def test_raises_if_multiple_equal_signs(self): + """Should raise ValueError if multiple '=' characters are present in an assignment.""" + with pytest.raises(ValueError, match="ONE '=' operator"): + parse_override_vars(["FOO=bar=baz"]) + + def test_raises_if_invalid_key(self): + """Should raise ValueError if the variable name is invalid (e.g., includes '$').""" + with pytest.raises(ValueError, match="valid variable names"): + parse_override_vars(["$FOO=bar"]) + + def test_raises_if_reserved_key(self, mocker: MockerFixture): + """ + Should raise ValueError if the key is in the set of reserved variable names. + + Args: + mocker: PyTest mocker fixture. + """ + mocker.patch("merlin.cli.utils.RESERVED", {"FOO"}) + with pytest.raises(ValueError, match="Cannot override reserved word"): + parse_override_vars(["FOO=bar"]) + + def test_leaves_string_if_not_int(self): + """Should keep string values as-is if they are not integers.""" + input_vars = ["FOO=bar"] + assert parse_override_vars(input_vars)["FOO"] == "bar" + + def test_converts_string_number_to_int(self): + """Should convert string values that represent integers into actual int type.""" + input_vars = ["COUNT=123"] + result = parse_override_vars(input_vars) + assert isinstance(result["COUNT"], int) + assert result["COUNT"] == 123 + + +class TestGetMerlinSpecWithOverride: + def test_returns_spec_and_filepath(self, mocker: MockerFixture): + """ + Should return a parsed MerlinSpec and verified filepath, using all helper functions. + + Args: + mocker: PyTest mocker fixture. + """ + fake_args = Namespace(specification="path/to/spec.yaml", variables=["FOO=bar"]) + fake_filepath = "expanded/path/to/spec.yaml" + fake_spec = mocker.Mock(name="MerlinSpec") + + mock_verify = mocker.patch("merlin.cli.utils.verify_filepath", return_value=fake_filepath) + mock_override = mocker.patch("merlin.cli.utils.parse_override_vars", return_value={"FOO": "bar"}) + mock_get_spec = mocker.patch("merlin.cli.utils.get_spec_with_expansion", return_value=fake_spec) + + spec, path = get_merlin_spec_with_override(fake_args) + + mock_verify.assert_called_once_with("path/to/spec.yaml") + mock_override.assert_called_once_with(["FOO=bar"]) + mock_get_spec.assert_called_once_with(fake_filepath, override_vars={"FOO": "bar"}) + + assert spec is fake_spec + assert path == fake_filepath diff --git a/tests/unit/monitor/test_celery_monitor.py b/tests/unit/monitor/test_celery_monitor.py new file mode 100644 index 000000000..b82d98de9 --- /dev/null +++ b/tests/unit/monitor/test_celery_monitor.py @@ -0,0 +1,196 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `celery_monitor.py` module. +""" + +from unittest.mock import MagicMock + +import pytest +from _pytest.capture import CaptureFixture +from pytest_mock import MockerFixture + +from merlin.exceptions import NoWorkersException +from merlin.monitor.celery_monitor import CeleryMonitor + + +@pytest.fixture +def monitor() -> CeleryMonitor: + """ + Fixture to provide a CeleryMonitor instance. + + Returns: + An instance of the `CeleryMonitor` object. + """ + return CeleryMonitor() + + +def test_wait_for_workers_success(mocker: MockerFixture, monitor: CeleryMonitor): + """ + Test `wait_for_workers` succeeds when a worker is found. + + Args: + mocker: PyTest mocker fixture. + monitor: An instance of the `CeleryMonitor` object. + """ + mock_get_workers = mocker.patch("merlin.monitor.celery_monitor.get_workers_from_app", return_value=["worker1@node"]) + + monitor.wait_for_workers(["worker1"], sleep=1) + + assert mock_get_workers.call_count <= 10 + + +def test_wait_for_workers_timeout(mocker: MockerFixture, monitor: CeleryMonitor): + """ + Test `wait_for_workers` raises exception if workers never appear. + + Args: + mocker: PyTest mocker fixture. + monitor: An instance of the `CeleryMonitor` object. + """ + mocker.patch("merlin.monitor.celery_monitor.get_workers_from_app", return_value=[]) + mocker.patch("time.sleep") + + with pytest.raises(NoWorkersException): + monitor.wait_for_workers(["worker1"], sleep=0) + + +def test_check_workers_processing_active(mocker: MockerFixture, monitor: CeleryMonitor): + """ + Test `check_workers_processing` returns True if matching task is active. + + Args: + mocker: PyTest mocker fixture. + monitor: An instance of the `CeleryMonitor` object. + """ + mock_inspect = MagicMock() + mock_inspect.active.return_value = {"worker1": [{"delivery_info": {"routing_key": "queue1"}}]} + mocker.patch("merlin.celery.app.control.inspect", return_value=mock_inspect) + + result = monitor.check_workers_processing(["queue1"]) + assert result is True + + +def test_check_workers_processing_inactive(mocker: MockerFixture, monitor: CeleryMonitor): + """ + Test `check_workers_processing` returns False if no tasks match queues. + + Args: + mocker: PyTest mocker fixture. + monitor: An instance of the `CeleryMonitor` object. + """ + mock_inspect = MagicMock() + mock_inspect.active.return_value = {"worker1": [{"delivery_info": {"routing_key": "other_queue"}}]} + mocker.patch("merlin.celery.app.control.inspect", return_value=mock_inspect) + + result = monitor.check_workers_processing(["queue1"]) + assert result is False + + +def test_check_workers_processing_none(mocker: MockerFixture, monitor: CeleryMonitor): + """ + Test `check_workers_processing` returns False if active() returns None. + + Args: + mocker: PyTest mocker fixture. + monitor: An instance of the `CeleryMonitor` object. + """ + mock_inspect = MagicMock() + mock_inspect.active.return_value = None + mocker.patch("merlin.celery.app.control.inspect", return_value=mock_inspect) + + result = monitor.check_workers_processing(["queue1"]) + assert result is False + + +# TODO will need to update this once the functionality is flushed out +def test_restart_workers_logs(monitor: MockerFixture, caplog: CaptureFixture): + """ + Test `_restart_workers` logs restart attempts. + + Args: + mocker: PyTest mocker fixture. + caplog: PyTest caplog fixture. + """ + caplog.set_level("INFO") + monitor._restart_workers(["worker1"]) + assert "Attempting to restart" in caplog.text + + +def test_get_dead_workers_some_unresponsive(mocker: MockerFixture, monitor: CeleryMonitor): + """ + Test `_get_dead_workers` returns only unresponsive workers. + + Args: + mocker: PyTest mocker fixture. + monitor: An instance of the `CeleryMonitor` object. + """ + mock_ping_response = [{"worker1": {"ok": "pong"}}, {"worker2": {"ok": "not_pong"}}] + mocker.patch("merlin.celery.app.control.ping", return_value=mock_ping_response) + + dead = monitor._get_dead_workers(["worker1", "worker2"]) + assert dead == {"worker2"} + + +def test_run_worker_health_check_triggers_restart(mocker: MockerFixture, monitor: CeleryMonitor): + """ + Test `run_worker_health_check` calls _restart_workers for dead workers. + + Args: + mocker: PyTest mocker fixture. + monitor: An instance of the `CeleryMonitor` object. + """ + mocker.patch.object(monitor, "_get_dead_workers", return_value={"worker1"}) + mock_restart = mocker.patch.object(monitor, "_restart_workers") + + monitor.run_worker_health_check(["worker1", "worker2"]) + mock_restart.assert_called_once_with({"worker1"}) + + +def test_run_worker_health_check_all_healthy(mocker: MockerFixture, monitor: CeleryMonitor): + """ + Test `run_worker_health_check` skips restart if all workers are healthy. + + Args: + mocker: PyTest mocker fixture. + monitor: An instance of the `CeleryMonitor` object. + """ + mocker.patch.object(monitor, "_get_dead_workers", return_value=set()) + mock_restart = mocker.patch.object(monitor, "_restart_workers") + + monitor.run_worker_health_check(["worker1", "worker2"]) + mock_restart.assert_not_called() + + +def test_check_tasks_active(mocker: MockerFixture, monitor: CeleryMonitor): + """ + Test `check_tasks` returns True if there are jobs in the queues. + + Args: + mocker: PyTest mocker fixture. + monitor: An instance of the `CeleryMonitor` object. + """ + mock_run = MagicMock() + mock_run.get_queues.return_value = ["queue1"] + mocker.patch("merlin.monitor.celery_monitor.query_celery_queues", return_value={"queue1": {"jobs": 5}}) + + assert monitor.check_tasks(mock_run) is True + + +def test_check_tasks_inactive(mocker: MockerFixture, monitor: CeleryMonitor): + """ + Test `check_tasks` returns False if no jobs are found. + + Args: + mocker: PyTest mocker fixture. + monitor: An instance of the `CeleryMonitor` object. + """ + mock_run = MagicMock() + mock_run.get_queues.return_value = ["queue1"] + mocker.patch("merlin.monitor.celery_monitor.query_celery_queues", return_value={"queue1": {"jobs": 0}}) + + assert monitor.check_tasks(mock_run) is False diff --git a/tests/unit/monitor/test_monitor.py b/tests/unit/monitor/test_monitor.py new file mode 100644 index 000000000..682076ee1 --- /dev/null +++ b/tests/unit/monitor/test_monitor.py @@ -0,0 +1,227 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `monitor.py` module. +""" + +from unittest.mock import MagicMock + +import pytest +from _pytest.capture import CaptureFixture +from pytest_mock import MockerFixture +from redis.exceptions import TimeoutError as RedisTimeoutError + +from merlin.exceptions import RestartException +from merlin.monitor.monitor import Monitor + + +@pytest.fixture +def monitor(mocker: MockerFixture) -> Monitor: + """ + Fixture for `Monitor` with patched `MerlinDatabase` and `task_server_monitor`. + + Args: + mocker: PyTest mocker fixture. + + Returns: + A `Monitor` object with mocked properties. + """ + mock_spec = MagicMock(name="MockSpec") + mock_monitor = Monitor(spec=mock_spec, sleep=1, task_server="celery", no_restart=False) + mock_monitor.merlin_db = mocker.MagicMock(name="MockMerlinDB") + mock_monitor.task_server_monitor = mocker.MagicMock(name="MockTaskServerMonitor") + return mock_monitor + + +def test_monitor_all_runs_handles_completed_and_incomplete_runs(mocker: MockerFixture, monitor: Monitor): + """ + Test `monitor_all_runs` correctly handles a mix of completed and incomplete runs. + + Args: + mocker: PyTest mocker fixture. + monitor: A mocked Monitor instance. + """ + + # Set up two mock run objects + mock_run_1 = mocker.MagicMock() + mock_run_1.run_complete = True + mock_run_1.get_workspace.return_value = "ws1" + + mock_run_2 = mocker.MagicMock() + mock_run_2.run_complete = False + mock_run_2.get_workspace.return_value = "ws2" + + # Mock study that returns a list of run IDs + mock_study = mocker.MagicMock() + mock_study.get_runs.return_value = ["run1", "run2"] + + # Patch monitor_single_run so it doesn't run real logic + monitor.monitor_single_run = mocker.MagicMock() + + # Patch monitor.merlin_db.get so it returns appropriate values depending on the arguments + def mock_get(model, *args, **kwargs): + if model == "study": + return mock_study + elif model == "run": + run_id = args[0] + return {"run1": mock_run_1, "run2": mock_run_2}[run_id] + return mocker.MagicMock() + + monitor.merlin_db.get.side_effect = mock_get + + monitor.monitor_all_runs() + + monitor.monitor_single_run.assert_called_once_with(mock_run_2) + + +def test_check_task_activity_tasks_in_queue(mocker: MockerFixture, monitor: Monitor): + """ + Test that `_check_task_activity` returns True when there are tasks in the queues. + + Args: + mocker: PyTest mocker fixture. + monitor: A mocked Monitor instance. + """ + run = mocker.MagicMock() + monitor.task_server_monitor.check_tasks.return_value = True + result = monitor._check_task_activity(run) + assert result is True + + +def test_check_task_activity_workers_processing(mocker: MockerFixture, monitor: Monitor): + """ + Test that `_check_task_activity` returns True when workers are processing tasks. + + Args: + mocker: PyTest mocker fixture. + monitor: A mocked Monitor instance. + """ + run = mocker.MagicMock() + monitor.task_server_monitor.check_tasks.return_value = False + monitor.task_server_monitor.check_workers_processing.return_value = True + run.get_queues.return_value = ["queue1"] + result = monitor._check_task_activity(run) + assert result is True + + +def test_check_task_activity_inactive(mocker: MockerFixture, monitor: Monitor): + """ + Test that `_check_task_activity` returns False when no tasks are in the queue and no workers are active. + + Args: + mocker: PyTest mocker fixture. + monitor: A mocked Monitor instance. + """ + run = mocker.MagicMock() + monitor.task_server_monitor.check_tasks.return_value = False + monitor.task_server_monitor.check_workers_processing.return_value = False + result = monitor._check_task_activity(run) + assert result is False + + +def test_handle_transient_exception_logs_and_sleeps(mocker: MockerFixture, monitor: Monitor): + """ + Test that `_handle_transient_exception` logs the exception and sleeps for the specified interval. + + Args: + mocker: PyTest mocker fixture. + monitor: A mocked Monitor instance. + """ + mock_sleep = mocker.patch("time.sleep") + mock_exception = RedisTimeoutError("redis timed out") + monitor._handle_transient_exception(mock_exception) + mock_sleep.assert_called_once_with(monitor.sleep) + + +def test_monitor_single_run_completes_successfully(mocker: MockerFixture, monitor: Monitor): + """ + Test `monitor_single_run` completes without restarting when the run finishes + after one monitoring loop and there are no active tasks. + + Args: + mocker: PyTest mocker fixture. + monitor: A mocked Monitor instance. + """ + run = mocker.MagicMock() + run.get_workspace.return_value = "workspace1" + run.get_workers.return_value = ["w1"] + run.get_queues.return_value = ["q1"] + run.run_complete = False + + # run_complete toggles to True after one loop iteration + type(run).run_complete = mocker.PropertyMock(side_effect=[False, True]) + + monitor.task_server_monitor.check_tasks.return_value = False + monitor.task_server_monitor.check_workers_processing.return_value = False + monitor.restart_workflow = mocker.MagicMock() + monitor.task_server_monitor.run_worker_health_check = mocker.MagicMock() + monitor.task_server_monitor.wait_for_workers = mocker.MagicMock() + + mock_worker = mocker.MagicMock() + mock_worker.get_name.return_value = "worker-name" + monitor.merlin_db.get.return_value = mock_worker + + monitor.monitor_single_run(run) + + monitor.task_server_monitor.wait_for_workers.assert_called_once() + monitor.task_server_monitor.run_worker_health_check.assert_called_once() + monitor.restart_workflow.assert_not_called() + + +def test_restart_workflow_success(mocker: MockerFixture, monitor: Monitor): + """ + Test that `restart_workflow` successfully restarts a workflow when the subprocess call returns a zero exit code. + + Args: + mocker: PyTest mocker fixture. + monitor: A mocked Monitor instance. + """ + run = mocker.MagicMock() + run.get_workspace.return_value = "workspace" + + mocker.patch("merlin.monitor.monitor.verify_dirpath", return_value="workspace") + mock_subproc = mocker.patch("subprocess.run", return_value=mocker.Mock(returncode=0, stdout="ok", stderr="")) + + monitor.restart_workflow(run) + mock_subproc.assert_called_once() + + +def test_restart_workflow_failure(mocker: MockerFixture, monitor: Monitor): + """ + Test that `restart_workflow` raises a `RestartException` when the subprocess call fails. + + Args: + mocker: PyTest mocker fixture. + monitor: A mocked Monitor instance. + """ + run = mocker.MagicMock() + run.get_workspace.return_value = "workspace" + + mocker.patch("merlin.monitor.monitor.verify_dirpath", return_value="workspace") + mocker.patch("subprocess.run", return_value=mocker.Mock(returncode=1, stderr="fail", stdout="")) + + with pytest.raises(RestartException): + monitor.restart_workflow(run) + + +def test_restart_workflow_path_invalid(mocker: MockerFixture, monitor: Monitor, caplog: CaptureFixture): + """ + Test that `restart_workflow` logs a warning when the run's workspace path is invalid. + + Args: + mocker: PyTest mocker fixture. + monitor: A mocked Monitor instance. + caplog: PyTest caplog fixture. + """ + run = mocker.MagicMock() + run.get_workspace.return_value = "workspace" + + mocker.patch("merlin.monitor.monitor.verify_dirpath", side_effect=ValueError("bad path")) + + monitor.restart_workflow(run) + + assert "was not found. Ignoring the restart" in caplog.text diff --git a/tests/unit/monitor/test_monitor_factory.py b/tests/unit/monitor/test_monitor_factory.py new file mode 100644 index 000000000..66bcda4ec --- /dev/null +++ b/tests/unit/monitor/test_monitor_factory.py @@ -0,0 +1,63 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `monitor_factory.py` module. +""" + +import pytest + +from merlin.exceptions import MerlinInvalidTaskServerError +from merlin.monitor.celery_monitor import CeleryMonitor +from merlin.monitor.monitor_factory import MonitorFactory + + +@pytest.fixture +def factory() -> MonitorFactory: + """ + Fixture to provide a `MonitorFactory` instance. + + Returns: + An instance of the `MonitorFactory` object. + """ + return MonitorFactory() + + +def test_get_supported_task_servers(factory: MonitorFactory): + """ + Test that the correct list of supported task servers is returned. + + Args: + factory: An instance of the `MonitorFactory` object. + """ + supported = factory.get_supported_task_servers() + assert isinstance(supported, list) + assert "celery" in supported + assert len(supported) == 1 + + +def test_get_monitor_valid(factory: MonitorFactory): + """ + Test that get_monitor returns the correct monitor for a valid task server. + + Args: + factory: An instance of the `MonitorFactory` object. + """ + monitor = factory.get_monitor("celery") + assert isinstance(monitor, CeleryMonitor) + + +def test_get_monitor_invalid(factory: MonitorFactory): + """ + Test that get_monitor raises an error for an unsupported task server. + + Args: + factory: An instance of the `MonitorFactory` object. + """ + with pytest.raises(MerlinInvalidTaskServerError) as excinfo: + factory.get_monitor("invalid") + + assert "Task server unsupported by Merlin: invalid" in str(excinfo.value) diff --git a/tests/unit/monitor/test_task_server_monitor.py b/tests/unit/monitor/test_task_server_monitor.py new file mode 100644 index 000000000..ea73978d2 --- /dev/null +++ b/tests/unit/monitor/test_task_server_monitor.py @@ -0,0 +1,130 @@ +############################################################################## +# Copyright (c) Lawrence Livermore National Security, LLC and other Merlin +# Project developers. See top-level LICENSE and COPYRIGHT files for dates and +# other details. No copyright assignment is required to contribute to Merlin. +############################################################################## + +""" +Tests for the `task_server_monitor.py` module. +""" + +from typing import List +from unittest.mock import MagicMock + +import pytest + +from merlin.db_scripts.entities.run_entity import RunEntity +from merlin.monitor.task_server_monitor import TaskServerMonitor # Update with actual import path + + +# Create a concrete subclass of TaskServerMonitor for testing +class DummyTaskServerMonitor(TaskServerMonitor): + """ + A concrete test implementation of the TaskServerMonitor abstract base class. + + This class is used solely for unit testing and tracks calls made to its methods + without performing any real task monitoring logic. + """ + + def wait_for_workers(self, workers: List[str], sleep: int): + """ + Simulate waiting for workers by recording the input arguments. + + Args: + workers: List of worker names or IDs. + sleep: Number of seconds to wait between checks. + """ + self.wait_called = (workers, sleep) + + def check_workers_processing(self, queues: List[str]) -> bool: + """ + Simulate checking if workers are processing tasks by recording the queues. + + Args: + queues: List of queue names. + + Returns: + Always returns True to simulate active processing. + """ + self.check_processing_called = queues + return True + + def run_worker_health_check(self, workers: List[str]): + """ + Simulate a health check by recording the list of workers. + + Args: + workers: List of worker names or IDs to check. + """ + self.health_check_called = workers + + def check_tasks(self, run: RunEntity) -> bool: + """ + Simulate checking task status by recording the `RunEntity` instance. + + Args: + run: A mocked `RunEntity` instance. + + Returns: + Always returns False to simulate no active tasks. + """ + self.check_tasks_called = run + return False + + +@pytest.fixture +def monitor() -> DummyTaskServerMonitor: + """ + Pytest fixture that provides a `DummyTaskServerMonitor` instance for use in tests. + + Returns: + A dummy instantiation of a `TaskServerMonitor` subclass. + """ + return DummyTaskServerMonitor() + + +def test_wait_for_workers(monitor: DummyTaskServerMonitor): + """ + Test that `wait_for_workers` correctly stores the passed arguments. + + Args: + monitor: A dummy instantiation of a `TaskServerMonitor` subclass. + """ + monitor.wait_for_workers(["worker1", "worker2"], sleep=5) + assert monitor.wait_called == (["worker1", "worker2"], 5) + + +def test_check_workers_processing(monitor: DummyTaskServerMonitor): + """ + Test that `check_workers_processing` returns True and stores the input queues. + + Args: + monitor: A dummy instantiation of a `TaskServerMonitor` subclass. + """ + result = monitor.check_workers_processing(["queue1", "queue2"]) + assert result is True + assert monitor.check_processing_called == ["queue1", "queue2"] + + +def test_run_worker_health_check(monitor: DummyTaskServerMonitor): + """ + Test that `run_worker_health_check` correctly records the provided worker list. + + Args: + monitor: A dummy instantiation of a `TaskServerMonitor` subclass. + """ + monitor.run_worker_health_check(["worker1"]) + assert monitor.health_check_called == ["worker1"] + + +def test_check_tasks(monitor: DummyTaskServerMonitor): + """ + Test that `check_tasks` returns False and stores the RunEntity instance. + + Args: + monitor: A dummy instantiation of a `TaskServerMonitor` subclass. + """ + dummy_run = MagicMock(spec=RunEntity) + result = monitor.check_tasks(dummy_run) + assert result is False + assert monitor.check_tasks_called == dummy_run

Zuhoka%S7#yHV@ zgSP+W7a(_y1pT5xiXruMS6{V2nZw_3BcvAtryxe%21?~(*uV!H{%JcT4l+~NQbh4b!5m;(5-E#z#%!Pt2YpV-d3J)BocAVEC`^BP9&#n97)Sx;T3;5A^ zQek@DWWztWnfLlSb}@x6LN%rcID{D@fj`sG-PGX?2edr;4?Karwb)soSM`QF%iN@XLsBtKy=AyW0(JIL57H;kM$VEO>hTfd7_kDrp%wH zRX1-ejuX6Ua$0gVH9!`|RJt`?q)sn9WqJq?f)0Hk87%h3BNtC*S!t7vVC;@KXJ9BV z{i+qe(M%x~X?5&-Qgh@MmY^CX6VR-rfjIMN(R&APlSHTQ|Kquf76>qIYwHAO@diOr zVPr5S2YY6J^Is2;k|a#uGRPMG#8^OhRD%O9i>Z7)>zkZ~KyqB24qKt@I2e^*;cXHM0RquD)g93~3sLELb5LbLT&v+dC*m4Kd%w;^3 zl#gxRo9Eh=ibsOC41lFYe^44`d5wQs?)=Ok zQ{Wd8s~0OJR!$c~h+wd~)&8wY;1QZIE*lEsz(4W?a?+uNzV~Ld>;VXP3sP&u8wM-9 zx`#?*nicm;r^B>$^a}AG(}t??Hvk=Vl8h5y z<&1Fo|JpP#H!n9YeKrJ^d3KhC?}|6brwJvSGsd*iR%(LB!Hju8ffN)MZlcn}Gz@0t z&N2QdI_ppQ_uFEMd<|0X%XTi?xa7MWGaEEx+4x@e?eDs`2Q&kqN`yk-pXa6t=ralN z9lIEwlQ814`@^c)2r)h2O_-ShJNCLypyVccyemS85ro+1sur#C2Kl=TgZm=uHXqni zerHT3_Lt^#@X9lNDfW2W;p`3nwGB1_=rIHm76F>7k$^MI;>4}&5vVRv+Q#KjkQ?L2 zjH+2H^v7H}5mnW^hl0l%RcMm&KZbx)p(p+g!t>@{HXujLu zp{{4grjZHCx+heC3*s~X(B@XH7yTLt5Y3Zp@bm`N7R5j6=#&XB-Y;7pZ-LoYV+9jyOYj_uO&`ZTjy^%GH8J z8PJ2gOflHG`RWW~RPCR}lcBoAPP8-jP^uCh@eNv39Y**w>!e%UbAU^Xs+@+ojn^s> zSfe0F4)47Iwedlx22TL3b}!XF>#~5Esb{Zhv(*wh;NB%1CHG~krT@foZ25&KQfW7j zcubpSL@8~&;%)QYyIS2`dXfc5UOfb3O!<>X88TUgdZtGUP?B5C{MGxLFCeo6jx_*ME_y@uMJuSM7~WmGovEygH1nN@OSc*wxmQ4kLI~=2)-DiKwX-!# z?$62pYQdW<%DC9gdx)5#xGJJfSQD1UJ(PllmSRFjB{wt&_T$k*L`StekiUxI{?#j{&?KI?kJb6>V`2JS9> z{ly{gU1*hhHJ3JFQf&il^VP6(dX?W1>&NsLgd6YKKqPQwr($CWrC#BCmizE$_fotU z6RJ;qQgzLNH$EU0;VnB*Gn;Y5dcnQ8DDw|z_FFgdPg<_|Is@mULueiA0rb$AXn9R? zk@+Zdxwm{Wb9q(lN3@dyc|nj%E1a=_xg5Cy=~#$ir2z?Q;Vqt-xPQAevp(XY1yII9 z4#NBl5gZslaSv(-vRuipm<%iNnC{~3*E8F(YK>$Z0n7_n-vF~6x`o;`tjm4R0)kPw zCHBx|jS`daAe|wy`OfTQ9bKI23*2TMsJ`%Cb}b&vWDN*=Jph!YCa@l!;)zU7+;WKK zurT8ybDtU(!DkOadghMKNBA#hC-z*A@K#I3u?`{UrCr^m5%48}Urb-hH9gu7m6&wq z?IzO2>=8~cs@JdZYbBnlQ!Op2`ZM~x{q_Sv;)0CtScO=6W}$FMJm@$B z56o3X%X?0m@~3Fn0?{GVLY64c5*~=4`&X4EFRoJc_gpp~+4xn<_PV%c$R7im?%`~? zvpOKp3}%QG$_2l6hN^5&y`>iXSX~l$v^Pvq5^O}Ef27d~AaZ@sZ(1=8dWL@cy}rp; z4nM37(I(bc&3A`6p*xlbmp-^uZE0zhy~<;_fvvLY+}V&Vr~9-fUY@l%@F>xXr{H?o zb*>~}7rIP+@&@&(nH~$HpCEuxNNwe5FbK2I8SNU8F)sNKZf}s=z5lN@A`d{LMVpG# zQ>vlixeu@575qK>@PTUAw);i!(xU=DH<$;pd=I;+|?~bwK;89uLIcKub;ir~Y&7 z8;j86aUZ+wj%mfWW_~|jq|N^(=$j|~JMO!9UwSk^OLppg!6(pr*@cECA69sr%4#W^ zMrF3yTO($UI9FpgWe=-BTrtmF0Xxbkz{nr~un~Oh2^rfgln<2v2f4-OMnb`nhtwst z_=iK*^9sZoGgG&-paI-O+u{1az^bJdHcSDJ<4Bp%m}lR8{sIi3;mED9jbxXBicw%K zpaPznjezK*P==M@>d;??EC($0 zfDchkt%R#)eb@mL8^iPL#^VY9$MmP3c}=o{e%0Gyv9UbEhH5*yLeORpa!Uv4YsUlk zBp70NpFhSGu=yj6;|Is%(T_Hl6fm)|y$cfg38=svK&=ffXv>1X94k?V7h}@#LV_ZI zw}d8veqOV*1lm+B2*3abb$Gm_l3)f2G0Yz=s9DCNg#wXAi?1gpG_^ zBw2)n_0&Oc>~ngE`?-+&rzgdUVjJ_p`VWa$%&rU~_vCBQk({6aAUGYNGbG zc6L-kz*E%KTfHCblJ-ou(uQ=f%zOp;88*S(NBf{XfrXj?Ae8`vtv1(gj92H|koGnv z3T!S}@f#V92kgu@OR_r|@SOt;KLOVuc{$bh-_q~r2Ts-1Z$g3)gUcl7cdtl88Tx?j z`wz?QI1~4~z2BE1QXrkD@MqC~cfwl`*oAh2f>AItj3tsiA1u_8$8{s*xYvl25mI~| zwA#|}9PF<$mH((%NTuEj2cU;>1f-)b!)OOU@|xNUn78U!C@)U&=Ts-ed$DHzJti?Zcvp|ssosLfyQf4KBxDyJ2)m| zriIO=ANM9ypZHxoC;HJN$hC%rLyV^umOuZf=rtWm-A}V2fd$4ibO6^uH~106c|_Cq zkCBm@B#!Q#0%dawRP&WV-`jg7H+&rg3~WUT;C#))se!!lfM$otBqVsV zB(LKTYrL9^KiL?wq3p<~eUV4}Ky(9un?&6xw0BhDu$Hke&yN9tKdZFsx-?NLw@`7j zUHuMNFSU?+fbkdX>vL2ZL@Et}1bATb>A&I97$(38ppGI-w0#{Yr@q>rQAQ`co_;~YGn7uXh;*eVS9Ru2w;M5rb{D~GG z?(MZ%@fXsbJ4pn}<#`=K zn`zI^yV_6gdA4#qM*UqZdFo;T<%=Dx1Mb}Mpe|={8ep1@&F*MF?mX?ygUKrX-^u#t zV0B?bMOLlnVOZ6ppR$5iE4%&kHz}Y;SwJ8v2@>DmN(ZabiIQNYa-H>12mHD1=2-oV9&^*7A^4m{d$J&V8cJznQ3W{BFBC zOC9RbRvS2wu)h53A{+=0WU|0Kfj|Zo*jO}@UaLd*Rsqvfrj}26yhVy*UhHObt~gjG zi|16%QXKBhfjOjw2VUnQn6bzs+F%11yc#JsW@93-J~l@5@Z?@aAW+d%7!)J~HO(CX zdGs<%U(owW18aH%w4qJ<=60B1{uLu+vPYN9aTbmPA;UtwDAkbKr<&aOD*|Y(n(KL(Uay6{)_8tt{d8}`&i)`fVe4=5~e z03iYT(gPCYoX9a?3sn>FS(ctt5z_XC$E^e!t-#m=dx13PsA;rA91@T|>jMnLKnXzK zy{w*Jl0L>DDg?%%cO@*v7vdz-LBmz}8@w@PqcW#hK`$rhw*gT68hXP{jZY?GkS^8{Iy65qq|ddJRkf{3c( z?w=-A*Y!xbg&y4b%kfmhAesbt?1~UiPq9#?S5%1cl?I5KeFrxNNW|W>9S>3D9eR78 zdgFLd#nn(qHJF$5xmO+vh(6+F@g>FR=Q|i)WDG6I3!ZrZbXcP`YzBR6kA_JX&3F#pXYU)k>N-_X;_*0SYE-Du zls%*Raaq3z^I)ZM{XZxBgpev%dcL@+^l1q89_P6l>$u=#ZU5zsV~E)u+&@Al`FKw>M`KrVy6t{-?En6qI4wWz!zrk_+PMKp^X37ue%2)*aH-flqf!Cmh(em-yNWn> zZ}004M)KZ&{+(gswIF@FZKTHba5ss+#jxf65katEB`FIF3yz>c%O0|vC_y!Y!}GR) zr9AUCa(!dNj!)3)D6$M?S_=>!PV~|49=}l=FDmB+R0$36MzPkBVo zUJbg{8IQ=4+VRM1e9s}fuzHFyuyoUjD~J2ZTG@y&MP&A=h5WYe1c$ANoH49 z*MA3K4|i#j^;xz~Q(U$Rfch7E1eSX}j-6p|tBd5mBN0XVXb@zt7g=Ib?wk}2fux>B z;A1?2KcseU3c4rbqZ za7zVn4LzCs8cL%LQMPGYwOk!AA6Nd2ID)#o)cuss0SsYNhVe!uNWqS@si)?LsTnN{ zmkL6%4HJx`u7rBV1E#k449(;P~+Eq0p=kXdtDrM@2 zdy0+jj*jp{{g8Xx6T7{=ja&QOn|?f%UQJC6H&6s_B4`;b0kHf->q|X)mY+z#kVjAf zNf0B{!i&=IQNDTPVh^1dZW$OHd?O1zDgu1(5ln*bV3YZvy}WG6U%%cuoW%$xF-z>m z=a#nCCexHSaOmDazku`D{~Wwhzd^PhejqzL`_|si zho`?Nq?)~a`4Sf$b5>zhn~8J0`f)*YjB>l!fi^~RLW?8>`5;3%PfumNVlng~W6Qf( z4J!lowrP68z(hNbRUz1@UCwll3b|JN2ZDiPL!&4{}M{Aor z*!&fhttF4HgY6U!$zD=?{2f>+gefJF0x>u=kOm!d*0vxLlaiKYuyHWIO7PXR_pM-W zUlOHG2I~~yAHgFJiKjw@13_g3utTvx>$+dzGQ9XBlrB;={;_}SOsoaalkf|355ylfmC z_hA}tn}2>dN{qC%warAw^s3M{Xx(D7+Vu;UR4J$Dy>_t3Mi}T_2bSqQs53_hX=!6I ztVELs<|x~1LzVApp8o* zx1gjH3@beKh(Q~Ge6|sJHT#qu=ytr+Z466Y1KL43f^_IIpC79gNQ^^UEM5z27l7+K z7F>o)uCf^pL3+TTDlIxzdd%}}{aMKGRelRu>R}_Wkpn_yBqd34u?_4?9y~^^V9$qV zheyW%yI&q118e67Go4_yYYHFcNROC<^0(mY4c|TgD?dk}9Xr3azHUqI>Cryi2H0{$ zK#xTTyzz9eEHDHO%VAQjC_dx*0RQ!sbwCSrFlVu#w>AsvBg%SS^X)R|JPzF3+Z(bE zzjSWWZVBkx;?G4D^)~IXa8Sy?PHO>Z3W)%%upU6t`;hlD%iVJ2B}@99J*sTU%F8DP z#R#cosgei({nvtNfrEwR4hX!6;?Ad@p$agNV5;J-1yj14_TVIYo{1Pv+@&To+P+t| z%x$WLMtP5gdVdXKU*B6Szh>g!?Ojrf3Tkf!WVojc40*4!9(ut-)eXVSz#5^#!NigW zuzz;~E5tfI%gq(>c`wIpnF;st<42l2_9Z(-HDpUj$)+Lu)1!QXhd$g4FOhfud57r= z0At{vp1-xJ{{1Y}Pdm%aTG$BASWFXJrp5jE@q;Gs6wqS>#x6;cIB~VlhkE2eDh&Ht z?7{T_vk3qK)u^!nrlZTRe%w^xk7V_mSJ^e9+K9X>jiUce0X$tGNY3rO$t=g%R3 za~QPjYdb(@@0icY$essN7p;zC`9Kht>xjurI@cab3g`ah$#9!`kQ8HJR2`5pV3g?3 zVWMLJTL|3GnbLhSa2g6;1Z6;d`p9>{Ayza#+Ll_WdWZYj!;j_Vj|VoEnLdDHEUe8Z zK*6)&ih;zb!#x4KEKGbtkLL!X{WIi7$~1~r)}#Tfaq$kBSO9E>#Ze>2Pd1$LiV&Y9G_n7G<#|hPt%Ur0 zmxZ1Ep0Wok2S?`si}}SLBb^D--M~scUNi%H2buLk&1joTn)>21koIP@>-&6vi1iSo z75t-2a|r%ipk_PRF=U`SV8K8pd|-xPqkt<1@`})}rFrNCC13f4+k`8~Li_!L7G-4_ zP+AY|(FL@$;r%2o!h)Aw-FC;E0d)11Kl_FXJup!9glyYwgBFoe)R`@ocDhVhC%+5# z$)#(n=mK9;R#sL#Cv(eD?ypK`CsR%?ZB%aBscxaHPD8Pnk^~6ja8K+!3K5MgE!VbLq`KE zF7N$*!S$IdvNk?ovqNL%wwv6NQt^(Gd}+%@fW)%j_Kt2Y?Bbf|ZgVDXgzs(B48z3M z)M_1(vK@B(jj8vh>y!EQc>upyFEv_4?QaH5XB&5Ck03IgnfJVFXIrDtmr;rZ>=(Z| z7bvc+tiBnm?kg^z5WWM+gal*&ESjspVSG54;V4oiXijN&j4MM+VI}jpk!nf!UO$QX zbBa21*Vg#iR*+;pJHuEYqjV$PZzsd<`zy`4yHiTumuvUY^~u9S4vO2oo=e$*W-95Y zKfnJ~Bri{kovm3~T8&z)xn`VV%Cmp%BzR^|jTtB_MrHwQ(myhi5YdX!iJhODnxI(y z5F5q|KB4F^3Cm4OPG3TmXi+wlH20SlDOXB!HN57}bEZF9{ghG?enT$gGLASm`X1a4 z#tQjBv#Ghcg1~=rR1|9wpOal{U-DWTvg7J%JPqsb3iSV_)z`#eet-Tm*t|^6uv9PM z@p>iG&7PY;!&vbA7e~Y<04PwH<^)VHP{Y=zt>=eT)kbO6Nza+|kI>YWftWr@Xaa-byy(;Ii(6}Wdc+AGS-0}N?|9JmmKWRJr~+|Q zJ*Y}t3Un13+|dI0w8)5vOF<+E(x5GaZ9K?}i`~v{0mIlAe-lj>!25=2^Ru>nm;mt; zoTO1(3m_%|4+B@j3F_p$k$!z&Q}gl%U5VGgRT6XHfewzKnoZzgJ_cWh*`&RI4^Pq_ zJJ$E-4>d9dzKpV?-q_UCbjV)9(gU+n0zk!F9TPNiaHWnxW>YT&W?1rC^~exFSx~eX zn#{G;RcrF#puu4S&{76ojsDJwzP`TDUjXWU2mDbnWA%U;KIt0u)38)0a2qi9SFVBs z055%*Xo+Cc4GawQstdwarLN z5l_{r*CDqX==vwKb8{U7p`oEPg`?Cv(5J%}0h87o1*pNk-@hS)0IsY!d1WbJ&SBKu z+kaQW;OJ>8X8>^XfcUw<6;wuDj|T!V=dnUYZ4i?EGYB9Xa`)kNC@iq%G=#eZrd6m~<&?$>@WTNfQ18iYzLj1?7N5n7q2` zndVRT_aiVY6?O^~{P)kF!v78e`ekZl+IxX}T3YLulLQP8l_;KSkY-DQs+_=;`Uu<=);}04Wc(fK#eDLKRD~vg z|39mgo|FI!stSea^eC4uf2~vkcy_qf_c0SE2JPx9ItG#+j{rhAZuCVmj?$ReTDP`a z-Um)sv^VJ;N)VRaClqrg_&t(U5OaecDr{U^?HI)?EZ@n&mlC`D^M;AkjR^^!ix}SZ z$rIV?W`ATXT{tB9PUAH)C(n-q==%sj1+b{|o(<$g_e35$m!Q7YgW?T?yM4<>aAitJKfX>UBxH))5|V zgA|Dx4j8~bE1~r0d#}T7FMKTc7H4Uad@&>ds(!^1j#Mprasl z+Tl%A45_x%JMoHM&`g$p&&^D1hPIxl;i$;jHy54mUV9G~W?lyAf6|bq-7N42*;c`8 za?5Sr3Z}PsmSXWag1YmvLOsA)kLT139{dBMicVn7A=fV=GSW6(Xo@My1zHz6de%!V z2IGE1N?K#gl%Gc7uLlPRK(~7V}?@%;-*%f1?TiSOhQg zXl{1+&Y0w&sl@c7d~hILD-eBm5z;1m9C}a7f?%(^gN)pNR z+~1?FA`)a6VOhoUyH>%MPJVlC-PF`%_!{zD@hUcM7@fGK+WQGomr%iUmSan$6Iv+& z>Ba7y#;W~lQF)|r<3sbUn0=|m6)8{CI48TJKL!CC3Z?H}4|VyFFzPft$bJV5hd~%BpA9(7P)ez6 zeSQ7&($Z31^c*#KX5FjD6m_Ha!iz6u`QwQ%aACu(X*cHTG8N^>Mc!sQJ@9)q^3o_a zo>IC1w7#QUUETDo+==p~uU~&B78-}_?R*`c(11x~w{oTIp@Dh2ck9Yr|3o^@Nsp|s z`2IK*<|`*qO-B1|gA?1Xm(!lYKeSPK`S^7!8gpp?!su#PSS<_zu$KPbyirWhx zekg{+nb6b(?9ZNMdf7NGz3MV8j~snaCXM$06?ivSb4)n!kCVqU`z6kp0*n_9-!S}q z>h^aw#htR%SP{yTCp*!BF;0SNs94|W9JEmz>;oeFwfx#flHsyWtOnz}P0WuX7R1F7 zx#`4r!qxDMvw_|VG)pG854Ty?Kw(I&SKoNAmjS+K z_#N6(rXTs=!af%!oE&cDMXh6}>lMgFX|Xst&g$uYDW!lX2}>WmqZC>flU6uMUkdQ^ z!8@9@c3XALW#dY#)rM?C(=Acf6B8bDk!!2%9t!&)xSOiIBdE$*yz!0EZwP8V1d0+;H4c|{)Sigl5K_G&sA8}Z z@v(z3qB{JXK_%+gh{tZ7k9j?MO{ka?vtGCpFd8IfD7qk?y#LdqRnuvM|4S+AzVA+S z!DX1y*RNmig}__1Vgc-S3GhIVR4lbrMky?}janFm0Q=O5AHjn~tc&!A%xaJn6m~%~ z@zFh}kL_3YiJzNo4G_A4Rh1hYkSy+IZTIna28Q(GK}_c#7NBSaxlAvfw(h;PE>~Z?NyeyYFO2B# zj?#sKobf2}7RFHi3^W-|5RKa`oVAVj784(QDi)k*WJLI>(KquvYV56I(fUO9mRU(y z^Bp{I%+un9lhs|>oSl;@y+jtn;^4?)89~tP7hzI9=By(YH`A3`x?Q|F!@V+wTgNifLI@^}2WXzBy#*r|^tgF5moIV<>6FOLb`l>3l8dvJ=dni>i& za&4CzvJG28x)rzdzE`tsDq>L!Rubg*&L)G!rZRrH$FTH3Z3u?Yio;MrHE8huu;J$N zMru_lhem)uvHCYNSoGNMkh-L_Duo1 z7<+UgtkDG~@jgu_mJr;OrsH^(2bVT|A_iKYo)*X$9IC3S`VIggD}XDtwYA4BE`=>c zldP`rnLn9Zb#yEDTMx`4fV)?8QO zjd)p6VBpdU=CC!t=0DIxp!7jFQ~3x;7xDA+Gc>5LF#~lzK-BkK|BPWmovi-#-Mr12 zUN+Z}5z{)!G}4w5=L_uN4A6mHY9>ra`|&p&!PG>s#SPG{Z%%n|`F8t;rLg@d!n+!Q z_<0!>x;a6le5@cU1@;3>kWhPcN*nh_ zSxvXLPnt{fiz2V8>BN?Y&dUV24BQ#;e@M)m_w!3uD>expTo6R597#R8+v3s@T$d)} zD8{v8pe{WoAd%K{`mepwLc|Cs<+vF_n+-IJ38qJu0jOA5Ls?KL}LC{jI#)7@ts zd7J!XHk+m}vFkoec}&$6&H~D}^(Ed&_Ibfjy7Ww#k+}Lq=$$PtU28lKWBrt#&S(4r z_yE)k1u8#asMo+M|NW2DXh=G5EQ`gar>9f#4fq8sl&oO=2P7+iL;UxHC>rWFA$*Yp zZODVGhYn!q9&Vo_Lz|o4f1TwZHIViUqP-fUJ|rtC2{B_p*MV1mLjs_F>eU0)*z|Dc zBC$}$=HUNetWa|4;FU9hZ}#_pB_NtDi8~;fpkfVtMHS$RQ!feV1Z)ZI;5L=uzvD}7 z139-cFE6jl2f={T{RBb`c4}(s@5sXbhpq39r@DXtKjbc@C6UZ%D2_@gBP(^JVI+HJ z9y>%fbyL|&Bzq)#uVf1++mXzo2nmT}&)@Yrbl;!f_i>;9%6Y%f`~7-duWLT9=LJDW z0WueRUhMdQeABHv#rC6Bcql9^oNEWcz}6#5lp)W0+u^%gS4$iv|NBoQv~{gKnh)P5 z0jbp#j953gIdZW)*v8yjzC6xwwE|KEy*a<}|V@{Ldc^4A}UNJf+B@!93& z<*3-bdF!pm`uE{alCLA*5Ox%W&Df-_Xm=CJShvp7pSc~U*VhT|*C=AewC8OP!9DP( zbd0BjBB83Rnf9kBOma+D`OJkBt<~24eyD@2#%7&Ci@dqB5Bu3S9@*c&&tGM;>=htz z3{pYwe3$Q(IOJ5giqS~$EUwRWalyAY1?z-7DhArvoU5|wy;pCSGL&>$8q?7~-cm8! z(fY;gPD$A3{8MY$2}dd9o%zabOrh`sc;=X#oE-DXUl$#zRNc_aP|eVWBDn~?Ly*%7 z6Z`DyOc^`hztF2AH~IZZqQdgjwrx8-dY$e23ej@6kr#l9>O*qkxbVuyiqcsXI!pl_ z481aCZ>$w87_|hLiu|Pw{l1G3#XyeY3X&AM58DjTsWSaGkiP6+XjVsoXV{o&e^5Br z+HDMFLmB~0_G)>N;mzg#Nv6MHJv{1!1t>6kijWdRv9>%9t);`{(#tn;E~9}TA83PQ z-m{=F$6i-t5Cd36!;ivJ|BHq%wvtpTnr8ib!v7Hf;{S(Bni#Ey4O=e}V6U-vy39k8o=Jrqp?ygKAMv zFJwqI(?tB7v(=kL4E?joSKKY$$r$dOVr} zi9VZeUyt5?97fM>Y0yYPYfcSkC#M6IHYgA}5iAe{ar7U}PKNj3&w&%Qc@OzBXq^o} zY*4Vc`F|KPj3?X=oJI1-1`20Ay--2Jm?B#dek(=s0O#g296>k+vgF0}_xGE_nr?#Q zU;_~}q|FmHFqlj3pI=7+3A`j(eV#k_eDn6I#eUgTrsOqhOI1qC{RFWs`gnxE|7|Y6 zBw2X=N0p+Qf9y_B&K*SJvo~+vc>c3~htV%6pkILcFZn)|4The_$tAnC*I@eo6|!_d z*-B)f!yEkb=g)@DXV0GfQMY6yg3QmE#xt$TNB<;O$?y)m!V+~OM6{dVtZd*%lJ(`w zm#xz0_5XS%=vl|$_5SynFq>Eh87#ZL|A6LQQ}-Cv!X&=im&$-C-tX=ElzG=x8c=8nnIN5xmx%6>_rVpN5KJBw0jp z+tjXIyA~cE-Y&U$8nUZz$;uj$4c4Z_qw3`Um{3)r6DLlP&jVRPDA02_{j)F_Oe69Y z)Ft{QE<$O3>$h}^QLd6k6_~XO1N{7sZT`3NW3uue8@rAvKxdH`RbtRtf<616v!pA8 zF7|~|vgy`wQay|wp&&2s{3ci^|0$B_KW0+(@D{|&gCTCq2xB(g4qiQkl|Ybz%?H4{ z18fC>!8{m<;E;fu&<^fW_&;t`KGzFJ7GWRigSoa2^P~lU zmo!RBO6Vjq$Zt#BQZJZF8b#VRlt!l`9})c%Kv$l zj{^fS8CR}c`E&4FP?t@97$orjkpaBUKbo4122yWA19~(55laka>msyTpSGT^t^WiU zSKj}uWf-V;6cl!Jyx7V|OEAO8k4+Vb5Qq572(lr}&;Q3){dx z-v-G+iU+pyOV`$sBVPv4s;$cg0YCyUuPw7iels(4ga^?KuYWfV{TBIrlNHM#{0mSc z|8s6))yWqDeD5dZf_|B89UlC?;NajM(?5E4`~UZ2>H{#49f7kpl`S*;AoD-!O2WAf zObEg}JRUgE-HH1&n2F5J>?%5sIs;?HfEC04|0GP`lCNC<^ihDoRx_QU{Y-X zz}E-Wr6fR7sL*rzj(&C+@;8BSRMG&}A>XqUwK*s6Kqu+^65~w-S;$W-FzYgEE}`A6 zfxW7{9_b*XqoV@?@1YFy1v(J;F#i=CT4X{0`k;3UrU2$gfV}Qy6M2h&nto0sv;r+h zxK_VJC-T$C7P|M=>N#F3rKdzd%e4hGY`cr&*yJh2BnO}y#?6g)8K?09C^kPYe!rTm z1H>yH$`H0}F5DG?&-bnrior-|&^WznhEK6vXsV9{Pef#7Qzuzi$wK*f@Q{IZ#HRU> z=;PVtLv9ssYxpAzPf)cWRL^ht;kF&^02xB{QA&)U#z*nRmD#T!7TR@UB4r8Dj(o1h z%ZXIY-=B>Ck{;tKek0#%l83xZL?Caq0wqdz$-*(wJQ$>7(7o>+=>rO3?6eH-p{V^EKOJ;uIzh4Smf)u0JiGaViPG^C$Q zQ^}H*u&b-9YX=g)P$ws+_B(Q-A%LU>9|wPjDBKB)Uq>M1F=HxUE$`D=FPW{_7#n~Y z0oY(Gxw02xvx#(LL+*2v;2_AK0ya`Od}J@!Vq$qbtF;sP#`jKee*T@3B_0e2#2A+K zsWI{Or2%vXJfz4%2^t28xgcGP=(0H(Y_L8))M}vr2wv0%i5gijA^p5(-@c@=u`z(Q z9E@RHmW2>W%w^G&uL^^yG@+wAzY$yyjT0|0n+F46tHNI-=aKkA$(yH_P=0<*0WLuU zIX>jU?jHG0kxSlCvLipznWYnxL55Jpst2xSSes(zX1lm5V&FL;;NJ|ZM=o0QzQ}+e z3TZy2OY=>>009tM7uS}CZ^1ikMz6L2+(CXZ*?&yE>v9~Tap#6sEl7gY{i9sT_Lu$$ zcvnB#;$ZQ}A0ap$suW~>=L)S|00t-jH(u}0D$s+k(8##ueKMSdBKD>;#s&&j*9+?U z2-Ur1GK6Fx0*9c3Y#*{E%>kpKv17-ME&bn2SiC__g6MjkWY-t^*Eu-dOW2HT>5u*I z!%5fz7>t~WQ4RvjgjC4C#-tfga^#GR>g5K4>c6f|I03z{{euTw9!_w%2ffkS@?6h5 z_lj=`L|^(zzPcGGgUEue?O|*L^98$pKr1Kp zyw+?gX;frAV@fneR%FE`-tOxDu{m+kOMc%)F2?Lpu}_p+8@Z5F;WkR2)_!JSt(Jdj zurZ09XW4e?hd@m9oSkP|gAA}TI1PSB6E*NVKC zrZbzE+j0(dn9{}IvgLWXm~!@kmFgXx&CBKDd7wuw)n8L38`sVCI9aDEa+MIycp5)J~6E&;*$^6!IDueE7_<+O_sZTe``O(d!_MDKN2^FRT_p=i2rXO&q)`g zW%4D4fI5X(DK_(O9xmXsLf*Pzco>SKpG!4ZHJ zegqrV>}cNlp3vSoZnUFua6BqzBf{%qi2pm6H8~xJH4RLx4Dl8y?vBOvfcB zu6-){&nH0cNT4h5#QQX-XFk8dxshqk6?<-@cPMQ+1yxo6bG7vg6qMQIPNEb^M=-BT`Z zz^Ik#P+2gT3YS>@NWs<=cErjX!gvv#gF){`wNzRiHykWWt%|AFUp#pvdE0@fT6Upy zyqUySf_L}LmUnqWy_eoQTiH;Rcb-dJ+jsK4VRZi9@9X_J+J|nBO@Bj3KTT&ToT_fxVvc12eUrL@0Z)!k$PWUXi@PC$8PxL=!C>% zXf7=bo*$*iiMQ2Q2kf~=h$RXW3IZwSj69EH(sjvZ79hru7f#;o?M1y0CwSr!c9>I@ zjw2$mg78!5Th0yIZ14R%}HMS(ly%Mn)nr)+4eDJWw~Tgf21#SH@u?wc4Sls z>Szr^^f5y7b~#_QJOer39;8SyEt`wcey6LO!i4*l<9$fGrTTUV#hX`a8tWvwn6h&s z%K0%d%gtsCdjh@Qb`Gw+6t=uDUleXjuP0ufyQZ=$iw1-@^d+>h+(w%9FW=4_Mngc3 zhxmg^#y`M4d?iAybolA65R2=+-p`)kI0|fVm)~eog>&OPj8iz`;tO-k+}4eaU0E&e z;^HmR6^3n(-cw^&`Z}cXw#iXGhaaT%Iob-5y%!A~q0XaqxiCrej?-JeCX#4rzr zXZ@AtvO+uPxKVb+cHZ$6N1VunVn5Wy->RSVptmq|?4FR(vW-#czo&e#sWI>}o3zh` zdC_XSX`u#OP)7I=GvW2Bi_%f{PpUnU^eefe--c|Xb?&?MpZi^_R+YNCs<=HTzBnVC zMA!Oi$`O}v_Gow56T-|wpsqFx!9heSR7yHcWVR?Mek4b*3*L8{Ek}IGS<%N<2D9N|eJnyuf}i?Ha)@Vi#i_iD_gM915cqft`hpk3FdX1?wTeo5 z>bm%mmqq?wGhGY?KF$5DU9Xqsg+!K{r=rVs_-AaUjuJ+wo=e`oml{6(f|fL}1HeZbDu-H{I#oOab>#H}2+Zk50{2lt6&5W(iSlvn6eWP}i zXCM%NWTk$sIId_WTj6%HMS1LP&*yp-4LWoy^xqP_)YBT6TT|y&D(vSXM{-`}ga^`# ziO-w-S`u=vA2t5gYJHCPdsS9o>tX%FT{@o!!|Dp=pki5=8&3_|TVf3$2rM1tCE=U>EPj#s=9`PNm(_b2zE`8-e z6tbn%3n9cM*PF-fSMrSIE;`(&`r`sFL(Mm3cc9m`ilwGf87=}KVH&besgI-=5 z)q^66+b6geh6~NQ40H$u;R4~jh6V;KjZrXWp`*ZhE^=zPOW*PL+4)nwose6{U`$)7 zZ^qt+oF?X+BS|$M>`i_|{C;`;SBulZJ6u|CaGExzZY~odp*Z+^k~}QuETL9c%GGpc^$Qxxx){$P5(CL zi=U-oS5;vYgj}qy#M)wMbDh-$uyQr~!zC4}i#^=6###-0F;*^P3N>~!>&{PdJyaqC@l?K*11^)cwA_BJ_m`%` z-oE~-Z7%-E)LO>;t<9&Dg&)}6awB>=<#T0I#^27PwbF;$js|*el$A6{n+jAS_ zm58H(ZHk7!@o~>y;5ZVg+dozJiQ6A@T@5YB>pwql0j|P}UyDupdC|^;J%<#CZl7jW zvBw@649C?{Kd&vn)~7lw-6@zg-L3FmYnp9gk7&pjqe5Hld9i6W>nO$Ow1}?tNO0{i z5__Ob$nOJH>`D_9t4t=JJ4?@r4Jo=w|HY?T&iGPYZgLq<;j0~m+_)h{&(3&W>?0A< z1$o}rHdw>%=>UPIW#NIO?uwZsDy3G#ZwWgp#EN()iB@9B)_UAIU8vF{!9aM&aP-X4 zMfw=t-coK{a^{Rn-DsA`6v;Ipn`Jaf!JNzLX=71Tgh|?^SFX|Vz#d$YO1tu^Lay}M zl28C?KLPv5jJ7~Q$bDPeg~4QCDJ4QEGN^Z!8GiMp zDL*ZFL1(e@olu`}ANcAmr}j-;wN<>$to(Me_d-(nLY-P5F4_1LFWwQKF(YbAdzObi z9Z$zr>%yyM#cy*6b5}EWv1n|s-4bEi*M7uFj!VT{cI1QhDa*>;kkI0pDzAZAwKgZ zKi&zCMk@Yhv(@PJ!SYuh1fNF_g%%LdU}SkhZX{`Z(1UcpL>iCj9$$u}+N)u2&gBOO z=7=!U8l--A87XP_wPcJp)|9B%-kCz}=n&k@Ni$JvO1sJS^w!ioNhlWz}lB*jam zFYcyo?{)feYyU5vz5(TKG35@^!q1!-<->R%($d(1_Ibcvfy=r+woq=@vXrv>ZmqT0 z$KR?aO_P0HR~s)4glZ9pQVd1iF%4!%6>YViP|qv*v>xiZ+y91P>QJB(H`4`}{K8W^ z?cH_A0uHW+-V(sur6380X*xiW`=#VT(i1kPlTd*&M#nbeWVHZph$q1?m5DY##XQ4j zKk=34TN~N~P77S@Q6DXpg>GbjC=p*eGdH9ajiox0Fxy0Ian^^E7!r|NSZ28|;#>{Ika-LQ0 z4?h%F(h#eWD7ue5O}I2P&?Lh=(bp)~tH$njM-9W2hNHP&3@JPIB265CY-KK`3sK3pA)o?;%7E ztDu=1k_s8AHZ{7*_DworR(`|uX)R*==~kv2AGq_;Y&0EW@X=M^7lR-zMsz$DG)r{W z?R5ODZ%w3%2!;dS;ssoORHft0;#v<{s97Ce)cba_Rz1S%g`we)s%QKR2#DtE)zwIo zw^`CG8`NgJ_bogH-KR^b2YHpK)(2T?iq%i9h7<+V+K$qE82(nZ6reQRtpEqrXLu`Z zVl}$@;{_G(;NtYqPQmkLo$2+RfJr}|T!)a;P*4zd_^sQCa!gp2jc@wi%ii+p{1LeB z6Ov}M(x!7c?$c`VJvlZKB9(R~1%7i@Wr0LwxPdmc~WCXsvK=4U^LBX&uJCc(i@h0)mpIP7y z5>(*GG~|HoH%(IdTZzWIvfIYb&V0MI8R{NDpLlz}dW=Uh z0vHy|!QNcMo2W8@jQbvVZ;+v0=*-#--XSA}*#-IRbT2Fm>MtSxU7i`5C_kWvU(_<< z;aD~*I#u6URaNzGc3M@tPKDARS&}9o5aXF!SV)3QQ9ksbfE>W5a^_t50X!mb(7g;i zMva}&WcR08Ly(+@kS;hcMf{ZAjbtCk74M+ ze~0QbD9wTK?S=lT|MOnM&jPb2P|bqem!k-67rZ_1Ab$AsOdps(Kn7U~bpxgX+?W?m zJ^D2ASRENh<>SrEppD1mPo>JpaT`p!CgT(&J|PDKeXv<{T(Smfw4o9S=t-1#tT--_ zdqF+ai$_M!$W&~YS+!Vxd8ru&UyfxB3C!w%KL3W|ofzGnoDzjF+8ymNIvr4`fwCYr ziAPy0vi$c$yQGS8$4SF6-fixoh1q2R~Papl%! z;c9VO&c3CkitGGOr~^ZutX8{TkxN`)Wx3&B_^l!HerrjlY}UruQp7mh+Q`STwS~Q8 z#P1%Jeu?em_zZKCp;@0p(qIz<6#5YYpH|0o*1yGKx=ZalGtLPT+9w?d%pa57Na26I z*XQW1N8xBYZ^eBzl&m@i(s>>VcbHIYow5{-+EdinqbdtIr;9vi87ywcI1d__3J_D3 z-JBu)b-?&^wJt4Sn)v&`h6yD7x_KIb7yO|IlPnM9T`<0iss{zY04*BR|2uBSN92!y zt!>SFdk*Qmctr{yJV>dznGo4OG5kIr{v!aoeykw%ou^L%GhV%e1=!S&VJpEqfz4z zs^$H4Eg6ylLRuu!OQRwiBIm(9N>PU*Us`If*s9$-28kSU#_nz-Fl~IQO|G<|#d>R) zPl3xD+27yNx*=0ob|Yj;W4%8oYKG|h$*m4VL&VHE$KRzQm40RJO{VY1$2=vu%87YB zS*vef2G09htSwC^eGDF=Q?eSrRdnZmzrLs8($Z6Fui=Mr-M^xr1R5{+X8Jp{wu;`? zvb8K-R~g9XUd#I=-t$E8@lruSn_YgGVAoV>{Nhyev}1!tzTS&bmu54PmH=&d>bz}B zQB+=Wv@f`1E;>iMl}-h$X>5!o_T;o?4XrNGxE6M8%%+Ws6p}~-VWjRd_Y0xF9J@Z* z_CpRd_BJ>{&1fL4puQutBz%bXG6#AM3J`bdBE^72U-?F0dV>)y=@E1{c22KSyVL(w z&%I5UBbu2bs=FVTwR}B@9N6r_;yxUE#1&E%cO$3ltzAIA?n}SKBS@+faw6_=3YoOh;1^-wq_UkIun7C~PH8rBaksD6gTXuqaJ!nA1$RxW9gWRv zTYu8M;Ly4>&(og}KKZ?Mt;4F^POXq`r9v})%rtHxV5qo_@9FP$bSQB;I{ki1a!Yo3 z@($SV6h%t`7M|Zzi`T35x|${9d(z>w=pTz=i4hIwhji3_gAW@nD(kPF#%g(fZm_~l zTHxFt^DIDzL~OKdq~J4DaYQIScq=Oo8lYRi$eW>Jm;@cCu7b>uM0}}saktckxF{RP z4~f#A4;XfN(!a_&#myGDc1EEibv78&F%cw7?pjm7xE1i`rWS%ow zvwr6Igg^{FGsA{)jQmjWt`QCWr^F}FfLy3rp8+WOT<`~&SA?zQzSEm58@`jx{~$qm`JZaZ)l+7gV* zL(7aO9(Zc@JL0<1N52oz+2S6?FYYG(nl0v%HHta9?`8;lgabCbuz|qLP;gEu;H6*J z+HjOr#N6Sb?icRm5a|xvbEYdF)=T@`^6KdjLy=%8wC}PHi(xk++}Xg>B1qTlH`sN@d*L{O*3a z{*@(Q))x~tuF+-lY`B7hC9HZZ0p?7KgH+U^3!yY7YV zQ9%J??s<#2@tkA}vyn1=ofu(PKjVO*fu^2;MV8@*#SLwWL?NFc#d5jn!$YlCH1*G~ zth5KV&MhhZ^+C+7@l3O0jL0#y*FRdXI>K>t4`|4W07tagi-~HeH_{5#=6>GQ(;FGM z34@=&!9ho_+vItOUlluq4^jlPPC0x`H%g-}y*l(sk{3rSd1_an zSA1Q(!^X^S)zg$`9+bPzcQ?Id|CTP2tk}cNz&U$owtw1@dq6xqYo_hi@WRY~t!EB_ zGi`T<-Eb1g3Z4UlZ(sPv5+yBUaD6G}mu`x$n;Z8DDj%!12_c+oQHso?W(#p%ldD*s z9X+$Ft=9UqX`{~&OHpEgoQOuq<vMTr7CqqAL`Re z-7nEjfM@m{NfwlHcN3vBrK2entDWaAqFXu0k=rfHQu1n2M)4C{ zfg-M|d}%&S1kcFv{)VrX@!Z}ZB_FH z#Wd3XJvz9!BGZ~;-yybdl{an*b>T)X1^BXN;Z=OsQe0fwApf+1i#3O=>8RgZ?16{x zZt$%uBNPFi!O>#^xDsz*{AZyymwgKiH~8>9YtSgj1xt(4>qhZpy$7JPWbJh zrQ!hl+@;=02D*p&wc5-NGj+;We3iU92J;B24G9~TSMFBkX$~(8MV>BDg0Lkv^u!6H zbyghkL*W6!>0uaB6a;7vV>6dM0EJ^f-p!fa^VFUv=}*FEOjW$1O%bPL-XY*&lVCxzFaL(|}hC zNiV~`;5Rw#Acn*6wdk{uvNpb4wHL2e|HM!HGX08>JCcQ8=)GsC(%D(iHpkef_49)A zdM`Eo>CXOz$NNOXCgt3|U=L#b^l2~WQe#_^a*63`lo22e#ib9^lajb7_z^ZLdjk4F z+eexsKBB&E2zL|!%SVY(Jx$A9Y=vK+QWB1+i*uN4HhVd>mT=c(Ub>N#m*pG&a&Wq6 z{bSGNSQuFx(?ZU$jfjX-3i6r$HdlJ=$@zfJqfemT5|ny2M?{zQyKl#LD5LS6KLD!Bm@ z4WNA5cvNV6nI^>JFs)Y*(FSMjJ6E?F^w(Hv?Cdhdmbkx_U1Dhroi-qS(@fwT{C zoGc{pxs1Qe><1rG>2VA96}91bHST?VHEzlmWoq@UeWylSc?LG9?aNkz1C8Pm<6A`s z>>DbIUQBjqP?5Oly<)4I9JNconF--MQWdBYe5zGH`PpH`uMNi>Z@KMRq*pV{8ne{< zyofc+f6Zy2Do$wS;`!WJ8Cz|-u0hEz8LCdsuMCq0xNilfyO>)(9XKL-+ z28ULMF9Kq@cRLsKVtjk!TZD{T-pB=h^mEPpEITo!v3=dV#muMB_Tng{l7s?ml@9Or zZPWIQoB681f8SNlcH_|yZ>tnl@*q>H$S>TtqxJBdyk)D8XwX4{PFo zf7hE?i#B=he6HkI!*G|A<9=+lHQz2!cNKVkL!z;^v`Y)4?W-6SlqQonU%$;jMaxc! zk{sU0Lf@x3Fb<&nH|nH^(a2oj;=iPhuVJhbd3vB(AD3a!NXy`DaWyDeld6Dof&}n| zrRkLi^o0saRs21({LORWlXGY&?|HF=b{;1El(Tng;UxKFCUciy8f19yTt0@tFPG+I{&qcAZL|lbBXq?&pmmT zC*86W<1T$d!IkYcKCbDz&(&H#B@DhLgf$LZUiY00jj4Fc0ROa(goNBjj{5l9seA{{ z0n)*D%}1Z9>%E@OqbEy3KDPT;4vr?D`@uM4tL6VzO#c!tu5{*}-h}(G^@qNN!SKNG z6YSRNJ!ivoY)Pl~1$r%h4oOggQ^912OnM{a+{s~3@~ch%(w-T}-6Rb$7-JsE)3mpI zM_$wdLL0Pj+eMu4B0@Q-Kkd-l&I{v-%Hb2oDsWe$jJLhCnuv} zq@Z@5=h5|$Nm-It66fpuAQ8sGHv7SDC9%S8dnS&SPq$85S>EHWmKml3Wyy|pRA65aF6d|UgEGfX(%@UE-`NU=`r3($emQFm%4M};a$=9}pBh~Dlh9ns zX}qR362mKId|N|pVo1`x{JkCpF<#zS1m%4&{C|TgD=E1|!Qabs03+dF)7?NOKY>YK z(3}){g-VxGJ}-!xc+kJ{(j8hxpM?x%PXg2gx&Vf9Iiy!mIWR|iQ-&c7H)RpLv%rF}j| z*&^SS*iL9jM+VnQYUgAmYP#PofA-whdmNg*LO5Tmu>JB8*SJ}>eb50rujczm&z~h^ z7ig=~`+|TcAIijf}uf`^6y08XTK(4;AA5Z@IW>u^{ zVAs6wJ~PRG6S}xp=vLUjWitN7v~t+YZIyxM%xN6kiAbvmtDo-{XV<9JYUfn$mwg|7 zkTae+<9g`bx}?iIoc>=5)n3D#-x?jCuX0>Q%s*SI0(nDdhQ6)p=gO7uVchHkGf zLBP8L8E%xs@G!Dz36#Cw#u>n#wZ1XG-AE(PNT!et_}ed+)lxSC;pH2E9wG{Mz)K&% zw0+K8e_z`d@bfN|Y{I@Y;9?U?9Zy=F^YLi(Hm-25yZkco>DQZnM8DAFw;SIVYt+gX znhIMQ>sD$nMzgs8^3%winU2mW%j;2I67xF0=GRydWu(9#N-~e=m@(tV+Vj3-|2Qld z-RuxBBi^+X<4s*w%h;UQ5NKvTbW2;uDsZjy(@spRR)%(&Fon1^a6!L0$#Ysy*3=ru4ur<;ovb&A19!hJgn{fAi*K40re2Oy1$~w720q zHzT7JmDH(XOpzKtk{-09tf3bFAus2JnQ3vL$@%Dn;bl_6XLn^dprMUj73*{6Wj|BX z4oD;*r`wM2?Aah-;d<$x)3gnaXd4z)Nz^>jHT`a6u%s|Ht7>FL^hN&u+U5J+2+X;> zfu0_x2H9!fydDA3Ei2aM^C8QZ?ltkX>Tk%ph1a6ejC215XP2)8E42tqZD+NMVth4% zwJ!O3#ni8cTj60mV9NI5P*1AsZaf1b<+lU6%}}f~1GHs&gI5+_3#pt|6J{D(M&JEv zpM9$JxuRXgCRbMnj=^&zhe8yK7)3C40RJLWgRBs*UL-@8W7U8qG|{t)^WJ>u!!xGr zsD^nRfRecdWJhEhbX|D7d$+zkjB|{4a|~74 zc;>!)!>Q)x!?-Nrso4d{5J01Y9QwElP7>+718dqc7wJ$ zZIv0+NvG_CGPvVPDq10!@Q2`uwVee}Jehmt*A$r#rdhw)OgjV`>^_OxUWNd+H)c9S z;ygeyFihN1YW8+^iO-$^D-T7i_iA-&0QBBzF36pRXOUQe{B`&vH~eA1i6S$8 zwmC2_pT5qTwVFaDF-7!WA{9qiqLmgZ41T!WD!?c`p!Z8lk|7 z|1b2ALTeH3g*VRcmGHkR-PNiJ>k0WdkIgUm19bv$R%?*^CM1S%Z#nAJ$_s~9sVaMX z`KmTXv(_~k+9A`U9Qx-uYlXyxk6-o`bX#?xY-tL=V3s?F#uSA20J9FF2}DH`RCVah z(FbofoM0R%7QwdES3esy_Od+(+=={hcSiB>^>wQa;WC&6NZ0%=@tgSd(Ydv(vB-d~ zW4(m&8(Fu=aN$GBWx_z|3Z)N^5Cc>vkp^IKOTF^#J+ZH-gNO=LAHFv23?kluUH%Au zblwW^N;ieq!wYH?=a-+jFFqLqnya2JIE^yEt0%|b30zU_WV+qJjW|6BYf_+*mx9cG z^!KazoJL@cVAfgcxFdJm!{{zRRNC%yo<(SI<3GesmFE!8qCmgH&=^!VLp{7ruT8jdRVsr0 zRDYB!Ylc41(|C;JoPYsu)X#=33;+PE7KD}31kr5^pTYFNtTT|X&HD_#G+H{-Gp&}0uq;3|1 zk^?$(=H(IFj<^LXrDT{OW%$>hgIb0sJk<%WJtL7Ix(&3O39FenQ7Uu?rT}p=%te*GO^-(N9bm?FV5;XQdV<#^`NQes4JiO+*e)zp+;B0M`e3F; z(P?PPL0r#m4xn+#%A6T#++pZbOeB+$C1_pFqr_yl<5~5^Kq&4aAX!k;z$>_hX?&W% z-inHRhr#gu-pG3(-exy2=lZ+i%`sN0OI<@I7w`3+)Z(dqQFCK&P5KU_p6haP$Maab z_wslfc7Cj;>l2JL-j@@VR8Ynn@NrVe_G$2rZpX)0tMACF=oR0s+#A5lD{?R?^2m-G zSi4rv+suX8y0s_bj#-cvNp@}xU?#Ek%eEV(+M}y8W$)erP1xG?XO=V0<7*G!9o)mr ze9`>FQKxvvqv}v*VT$VHdlR4Qk1e;#N5X?mhjGlOvmoirt3ErUw)1BVFy>Jjy~)Vn z%|dl9CLAr>CD2Y{CQLG*Vg0$)3Gp)AkMwa8()avG-oI5{rd;OpVw%|x`P=1CVk%us zEi7c(nizQ1Uv%9`l1+qF$C!GfK+lp_E%%t{@5*hCig*01+LOWU<;>96d`j(ui2M`c z_{f4!vVz8d43xfh!80-{66>qb!eX3+jyw?Hn7`%pl}Kc zpk_fQ84;YhvrI)Dm1cMdH89MGGVKvFD=Rrn0ir)icD6o;m3{qK)Or5Tgz6s}9}E_Jls`#hxRS*`PrMKNU5Ee}%UhK7bx$@?O$^LT$k*$t4=Q1;vd5M7D-MNo=F# zszkI3E;BTkST;p7R$rNMM%xU~GoX}(r00LXCwEC15uis$`(~m7WirlJTa4#&6e2aP zY|1Yv7`B6w(d7i-+l`#{8418Eos@kH{AS^JC!aQFKH;;ybj9~l_;sqAB0pz z#mj&os03;zcXoDCHJgsLCz?P9QIxP9G$1EmN4cZNc`i#c`M~4k;s~^E#W)H~Y$u*I zq4S`H0ot5_CuEgyT9b;P=leD6H=}$Sc|FlAg@B8JD8gkGcY{XHjz{ssWQrsI6r!*_ za6*+z^8SE2GT5M7XmrE6+m~`zXUhg2WRNgn@7CAXOACvNxIENhz!!!;ol&Rw?)%60 zmuJWL5*&ah@P^@dH<+drJHK#vlzn20x|Z|)z)=Jj8yO)91E94!ioN*r1y7Z@tvP{N z)V-`!7=X*W!!LsCW;InSeHm~>EuDPHn=I;}4AmS?iViaG|F_BRoTFNHpOlu4b%DW} zxvl~Joed!7tuyp#KX@y1CGVInW0EldHG!x%5R3zS(t3bjb&%b-Z|EchKK2RLk`tX% zd^cIxcK}nV>D%U1I(3qALY9!gHlNswI*?w5ow!b` z{{V#iMt%TN)uPKk64VL!lOYxc)BR4+%1$011sD6N_+>-QyyF~FXC^f}GhrT!cub9d z&LvqJ3QQhIQFA=#PGkah)RR?0sVQu-6;TW|KpB!PmRh-sqZpBX@YqY}gFLH|A3!wc z?)v}^65>Joss#R?<0rB09m41*#-O=g*5)r*&##gutO-3(%axOB{jpF=hTPctVAO8j zyh*urW1nD6BqlG1s8kR`byy}0xrYP6`^J=u=@B7LQf zdFT~*f$thzhuNN1H|250TSxkhuLY%-`kMo>ywE+^tTrroK_t`}IL=V!qWnmh4^Rdk zuT0F$8pkUuB`E8BjXd>AWg{^GBUI2rLF=+q!zK3YZ+(vbGhk*_?(J?#SS_<(T{)2O z-KumN=p7_%O^{wRM|x2uXLoET!hlKZM|?o$Hh^6m=r(q#6Ld#lSqInb^Ea3f1c4v1 z+m$luY>JJC3MAVMkQW#mYq#4-$B?-P08eC#xhDmGL?cHJM@e2R4812b zXcn8W{#~q;{|rg|lk?=S`Qy>~;pOGs8QY2CJjL6GIh|*T*B~@l%nztq9RtG6Z%x!D zrfB(@WQY=-{r%AxQ}nx)R1H@Qkb`U~1%C?eAo8FB7Q7}nM$#_wpPQEI>F!$*x z-O{Pk>-UCW?oyNf=?fT24F6-Y0@DMz=GA(~4d?1Kjn zC^0{EKyl0rH{Owx3;fN94Wjm`Wj6;60<&Olz*#1(1Fk&zQY{%7QicMDutzidd!j|1 zr2zQpv3xK?A22d9F>4dXB*L__7z~!;+{U2Hzh0->sHz|7Qn~>;g?NA(qTO-e2ZJDU z4$(I0`oOvj>+@20Ec4wSS6z+XVR{WV26f$armGE!3c~jf`E1Xn=TSc9;e-e!&hD|c zaRk=QrCbo_jzbyx^6p@MBujS$9(jchT5KpiMJ~|TQOWNEg^PKU0v+N499ZpA!I({#%-!9-;1|n*CWMTg?A8qPE=99s9pK3N`XnX$jHdp z8D1NY^PS-ipy;641KWK5U|bFijvEP9qCMgJOjT+OBVmzj&dMWsqFP^m01p52U!|lL zVp|yb!GMf=XU9>{Egm%T{SN!Nvf`iJ0{Lf9&JVz4m*7wOz+ekk9RLsS#P6P-BEjMR z2r=&rflpoRkwKvNx&t`ws?tb{@P^#j8LN##d(BwRtM;FiREB?dBWT+65P=SJcH@L8 z>vLJ&i6$X8RZgL*Hl;oXFA6VASc1 zX$oZIV&$;R&~I}YEOT&VRx8~XxG=Oeq$|K^g+DRSuoTy@B)=C&76;bXuL@ z^Hp`#&7VQOiZ51^Kd^QMRwi&XQ2(9Uk;y9} zq|DE5-kr_iUTWMLEn-iD2@8*}CxM=5VK-yl_J*fFSR6T?pDi>yQApP01ubW#h57k# znh@Q>Sve$cpPwJln+OHp@SfY2d4So1aQW$O7-2zbeWvPr@!V7+1;(QX8ZP9s{?jlp zDEocsh2b+|*Oel{Ok*6BR-^rBZE0C(;FgAs3{`x(omTLRZnEDgw8#L>U6uo^1_1qa((|-mXlU=)zWsoF-yoF|EewZxL)eA!=aggwk!h^T*`J{gNC{e&ZKkrs z=c=BBi&p)S?%EIvyOQO(2dO8gKg%wP_$uV}2VW~LNjE}>e%NHE#QT%r?pKC&^2LVM zI?p&*X?xcBT+HRfKuolMd1ZyfG^t;$FCmJuTFIA(kMhc)jZyyv4sMi??FkAAQEqjY zUofO{b!Fl)v!(R40G8{>uLWBh;{KghytPyu+Ie;%B}EU^0EmY{!rv$O4r~Xjh2hw* zUoafLc98{>d?wV8XJ>!0r6p2-Ul!%3zOo2!q4CJBFp!35u##p2S_XF_Ne zA*F}90GO6^%v>r2f4z^2Ec!$lwn@me32*050L#2Wf(-Wem01?_%W^R3X6=$v>iV~A3+}3$9x)kS}PppM76_&Q(lg9v6ef7ZmUpz z&>f+IaUFu!|AJ#)*$pE+pRu{l5?8JB1L*gJVQ!_*phb9NT>-bDJ(55Rw0)EYkadM% zkDH={2hj;J$uE6~+}elNKrjegPU`TYonS{}`@g`WeYk1aeVV9=!AZKsIphk(%~0(12mxrpMjY1xY@*>;>S%SJcIE`S%_cn3$Jk=sM*3m4 zZ^E}62lsL)IzX$Xw7P}{9e+5^mtlCWBjpZg_TwH-iO}#AyR&v@3jVqGbZC-XjI8tg zM7_lhS|&`a5H5KUg)?_^sL!HD6(=5Gv8ck~)A2{0SEs>I!N^Dat8&UvOoUh15_g_D zbt&me)teq8bQFWYr17TJZ9cEq4qL}2zWDX;rG;%664!cLg3E+U{_|RPXY#uj82{tY z_ald%ABfO-tS$+HD`9!*EsP&LfboYgQDQR1{3yVTS}x~4VRu9L_JV|>B7rO(<#cJx zoGci0k97Q{rxI!>>qZ1`|J4=-8mgmk(~~iy*^cpGQV^>?5ZllDU*6?CF%$}e<^1@g zK4rM^WfGbinhd0@5ZG{uO?$H-1FGzIetYc2TJ{GA_f$=jgsTk&23%|@G09wH9bVs_ zc=IFp7f#D#Nzjt}Ga7T)4)}zO;K{f?5PuV>BPGd>?UCT zO|o?e#*Wj?{@S)4l)Mz5?*~;YKxx4LoH_v9HW)?$xh&W=V@P9T=zM||_*$^Cx4gi~ zRc)M+Ka8O{>bCcqhQWge3yxOIn{nA*-`cf!J5;r**2mZ#d!#qr?*b0H5EaW*vsKw# z=c&NRDd6v_5Ez8819R^#03w)><-`PYOMOQT1Unie{n}o&)N^Bu1|C$be7O$^2I7fu zZ&(o+aYXviK+WCH`0`Ab@bXOyneP**=7aa8rc(vP|qaeY;gC>C&f z&KvNPfUT?+Le;lo@A#(4=oE;z^$45Y$mVbY&-9_y)CH^22ywuwxMEm_*u|86{j42FU zX%Lqy7A!%?acGhcC`&ygXxSo??QS{jjum=Q^ehU(oJo{1sqWY9B!d|i-ysDiAsX!LWw=jCZ#zDZXUrHDI zNT0ZUr-ZHd!nf%stiwYiL!yrS!Cst-Ebg;#(wc5*%P*NVR#wG#vgmNxh*S>?x5_o-sl^+?vNZ5GRDeFlz#6F7?k!-BFhQ40QDVBs($x2)T51nt(f z7oF?4lkhS@M(W;f5AUlyhO5y}7e@X#xC%> zbFDRLoW3q@J3kX2QzBU9V&eKGWUTqnhNilflk>YV|Fk=Kj?J&mkF}q1;VApnx{h6o zN{YUbMIUM$?H~DNxv{W6ukT(+UqR%TXjyp;w|6d%P9{CMvqRvZC0UgVQM5TAVd*G= z?4r}$&(C7>8CI@5D+L?G#fly^xn8ADO%ilDXYEl%xZI3U+KKX&Gy!HE0tra)U?M| zen+gEe59&BEwoZTBGXqmy#N2P_1%G3_3!`3D0xBvquQY-kaa`zHiT?=llDd|GGQQd7t;V_Vs$vb3b=7&%U?6CAh{` zK+C+8%UV(Vz`=Vu@Gf&PJKDo=wm?qZ1uEmXK+jQQ%dR{p<8?KLnC?PIa~4bZPMw4Z z+gZ?8OVa6xrd>jT^COgJ3@Hen798yOv{YI92AIvg&a3em%s3nv;3@Ph|K-H@X)^od zm3gtAHG4PL#ngt{f`MH-Xe$;BSFg68^zag2YGjQMELe(;SD1GICYTUVGCOE>6vbOk zd_2Cx_hJvPxU1kA6gkc;kB0C-uv-I+GfiMwV0YPQzn~ zPq|$edpnf(y|hVG<@c@0zmJU~QEE%Wov zy#g`4NN%bHv7Md&ac}n^5AP!jPQZ6h&h8FJHpLHFpsd(!uzHo$XgqQ$0#1av*SM&U z1#AA<56nget5w5|OF(#3EbCJr{3Dz~IBEo%wA4>2y4+mxK*C#E;5Z zocG?Lsyj99LTa?`F@yJtVkl*VU=~&t-6B0+cnbUQ!3S#t1WTyeK+Rk+yxd@R3ZJF5 z_F(Eq*>l^1BSNB+LA~U~tMndMkBazwa)4a5(faBrBzEDe904)40ICge3K{>)ACrlF z{4}IqX5OXE@U^0n_H{1J6y+OIEs|5~@Ibe|Jn0MmF>}Ei-{6Tmmp61Ur{t_Gk<*g8 znHCofR3pV9RCkHg;xH1>Z*{5nxq~wRp}E>$E|rwQ!GYOL=fRr?5%u5~5^|Y=fVkz4 z@-4W$Y{*N@1Y~U+b>72 z(9H=&8O4c&d*Gd&+zC1az&`8B{%6?4$O=p)XrGT|=cS?+%|`_^5aic^w8QJ$`U_uI zXt0?P?~(NrB6piL*BPmBOG3XktVx=(P|#$~?Ei#y>)sk5#rKYu&){W5*)`uZSTTRVXIbk#?(atjl*H;uxtd6-^5q@b%{!QEcocT?cEpcLz&j9@NJ zE0o8Ezc{lQ{Ua_-4mNN#oz8OUJCNj|%+2QNIgO)@genCP$05eA#CDjrqB@93S5MhodJBbULaKdiE(_%gXcF@3g=%S}b z0;ibC$4JXG?D}i7bYOZX%t1Kuxy7U-aDYa9_Ct~-7C96+*GnuM5f)Vgu*V_^@u#D{ z&cxk52nnT~&~|F};eVZXEQ_uBg9#jCk^d9FQOchHpac5!RFl4HGj!jLY6XMBmAhn@in4mJX(aw+w^8S|AD(jc+Cz=j)MD`~?egMkN`VPepn`5@P z0?@aFoymcAX5Ck|J+!(4iA#ByU$39cR5Qsw|cfvcc zShhVMA>vc;0F<4KwqHIsLj}iQ&*@#yWw^JP+A-ehT8P@X%^;PpQYsRc)DU;z)72`c z+Cmo&$&-RlFpHZ!JVPIEroL74HURI~Ib(L<|HhV5&H-+%FM7b!rBGdB^k8 zv7Yt$?4gH(cLc-B1G2kCdFAF?g}g$#9Tu}|-FvurT^*K}^EWIy=U82?BM|OmrS56^ z`QHq(-R7F^8Yu?&t+`knjpIsH2Ib-+PexRbLWK9kzt@T3R?rU#Ebtoxk+n4Adw{Pd zn@^{rdpach=SU27*Hp55LyOm%bU^O7erTcOqY4)H!HA&lg&?ktxttLew`HT{zH;^* z*t_K_^GkU^oTQ)ti^zWpSja*&!7d_xkJ)?5=fUujgWAH+_lL$Na(|p&c^+8k*DPp3 zt{TN~c44>FbJ47B!NFx_yEQ{jn?qc(X2J6wLmTHY{#SIC^oP-TL z9n6GfXpMP|lbPJJ&vnId-J4smI|9UOc)FVaDw2oVXf3oiWm~m-NC0B*4xoW`9H+)ei zUOr5y*P>m?HM7*si*FKj)4$(y%`i}M#=Nop07muxinv(C&AH#g*4HEk+p~@Z2aeIx zpFD8Iec%lPL3RrHFAl8r_#@U!?;^i&HS08ni%^{;GU&zk(nS z2TR=f3?rpBYTWU92S}2$C8oz`JC%ltqGLTrgZ2Y*hUcgzi)(Gpn*5KEs1YeCj!E@n zcMVxjF1rUk{B%v4TE}-!yNT4o?B~wAU^(l#5bda}WS-9jZMi~m`uQB6s)2Yb#UD_{ zHrD;))45XL7X?BrgU%-Xsigge`$Y1(7C-(EJ454NJA-YXj8Fp-bWP8#cuWQQ1W8S) zIk-wp2V(iDGN_N=Ynv>kQEV`-^>9#D8)@)8kf372s_b4gQ@(1SHJ@e`%hm7v-B$M) zaC2?UZG^#ziyq=gJ1@qg8@WEIzYw3e_#@bQW_`U^bG_W+$1WVhfkPP<6cNKmD=oAE zoK!Ei&4kTleqvq!JeS6s*;l(E;9@~e*kxg(fyXe0Fl2P7&9LpG?eG`tKH;V>Fb;DZ z{HXGN#oKMv+gGN;H{7e7B6RFj7Iu5^b~!GEOEjnJ2|xB6AC{(f(9e9XvX6z)by)eh zzVN*JNm(mHgBjL_)b$}R!FWk-1%qyfE@t(Gslp-E_#uPUsT_@Tm)qz7G*A>d0|CSK z(j6(&_2LHvD+Fb)UfvNH49riRT9p=1G=vjxnW=a?zklk{UQ7iUq1am`>g~`|zC0J( zq9S%B9J=$MdF|Jpq4JeV8_+{Qd+b}x#a3b-A(>j8zM<#N_SNi4I zsrfvJJfQq97V}DF-|dg46fEI4cA6fNH3~j{&kRHe6pkeoc)8-Z z<=nOIf0fO`v^v*261IQy%kyi_Twq$yAeV`JO#KXS>864Pu00>fYPn#;YK^Cea5(rL z;2WSGoJig^$1N2w1c5oHw+q42j~`EJS&AEXkuW?)yRXjCM@`#J7U!3HYvacm%gY@+ z#yVvcK%Z4HpLV0V;iG*^B;R9=ai8?&G)~oBek&e+S*tSHfl**i!dXxs|uxpip3*fC~3wXw#IX(T2kE!(gkrmZ#$Uht;PNED3%743iQn@Laem&>>2RL^7s*OgP{h3Ya@1)9ZdxiKb)#)GkO# zZmyxKJ8TdHZ^WIP%T3rgEID5#hSxW=aU9TFDy#3Qw6PeP0Ret8uoeirK@*)Njrn_D zEH@YQ7eEJ3q{>_xtlJ$XXqrexMkA37X7KYul(g%7v|Un;?e9$ryj($@N34G--9w(I1phf#Ci()%Vg>DXreOqSt ziF6TlMZIrx@X=0v#;i(-vFz_?;%`S=u^%&L?dtk&XE^qz9g*qc+?p7@9zX5lxuHmE zVTN!Az)oFm7xEGv?6*Fz8K=Z)NreoVO5OW4$8}5l@>;L4mX7^}RJEam{o%lz_q1wU zyVFS2D4J8F4cN83HWo)jZQtL9tiSrhMmC&=4J!yXihsOuA!LR_;9*uR*ghC7^1E(l z*lEL+*6U}w*H`0H-_T5~cr{X_uHs{(HyXIW;OO#t4hXNex^+e9G;H1sP6~g!PO3)6 zR+r_z&}7G|VhrSN7KXn@ncVdgpEukg>z~-D*dYdDN)h7HmDVecIvykpPYAoKDFz?4 zlwlvs-s8R$KHCL*m&~}HZT>S?%b;4DWXo-jsVTLTrW9d$8vx!Xv^>4@e+^kH~UcAj<+Z#)#VsMm$=6ne621{^V;k+lmzo#DnlxiUywEN?4(g z)|?X^DzONf;FP<;9bVh_u{N@c4|C#HxLkDe(rAIt9&O>!!iKMEbE0Xr)~S#KhVBi> zZlXhjUHYGux!8%JJkIXum~Op{Odfr=`ezb%r*lIu3>mdf27Zzr`=GgWyChpNpw>c~ zUtd*k0iQ^nSv1|FRrf(h=TcsukAwB&AnST7s8`oIcyIj^t&F}kRH&(OJ*+9pE`-Ji zNK~CqxDa_Bs~*pSS~Tn}WBpkTr4^Ex)ACx}ABzw;k$?O`@YERcA}|#6C0sZTXe2=N zx)`dCB3|KVd$?5c;X`T&1~lQv@Dd<0=!WQy6{^rBXd%MyDzE7b6FqnSP)9RHw9FWaOxNr{-kfa}! z*v=n9*x%^4x=Dg@31xuf#i%Nkc;O!2+Vtb-l#mW5Tox8U(*#y@R%RxQ^vc$kG3cOZ z$SKtQWsHJE#A_F@UL+$|?9aE=0ZicZyE{9LB#5Nd{O7X|0mZ|hNQp7zaR2iR2xA2kQg&B}TNXWA z_{}vBL~gtn`~JsQptK;MQ4LiJlN^n zA64=xqtqbrvN-zhJibB2X~kdvcxnTLM>T<)YUqz~Jtrt3!c-z0bSPjg9PX>!i_IjA zIcdf#Otn;o9Ki-_RYUnyF|5&kbLCD*6YtF`9QT1JqibXBx}McOFDTfBK*Y?mAG}aL z2LpkyQC4<#0t%xP|3m}9#m{{#{qzhAlTP9*loIiY*6TTux-Eufb@3Y`4OV=G-f#!M zB>HZdMwDD=?2C5wgb-~Zpo!P4caEL)@5_y_yWo}YVh_K<(hkp!pg#lZhXE%D%kRL= z5T+-*xi32R@L1k82e{31QDs9|9k63u7Fj33Y?I9q*rNM78z6s`Nw0-Dm|<50Q}@{X4Yb17Wt zWxw=@A%&hIMq{ z$OvEHh&~)*e#K4y^^i6T3Nhtz)u=I*!dhk(X_u)-mOmc6A}?+#STg5RN>0sZD;(Qp zry^Ep@1(OTl&V^7R(^+Uc$eXt)pA=u{y{kw-y3II*`S)5cYMucpklVOT5M%?nDqo9;5f8n+}-Qx=N6Kzv3y`rm`9)`Diz%Hv4! z?ACd5N`~ecB+lbi|2+_Ib^p&koZXUyPDzkreo;c|b{`jc^6)+PByZ-ag#|Tt=NrqmeRHR~YZK%_Mcr|6B z>RAF)5r6*HxWhqbNQ)O;f+IJ&HL+G2NJl-uV}pn%iFm1HK&J}fTROL<&*~CDtWt+S zqzE3s^v5?|0&HW07U<-~!Wg%vRRx;8QL!^twQEbG4Srp$0L?5x7)0kE&$#ffqI3X8 zu+3q-Jqd)3t|G8)ph)ST(of*!E`#{)j22W*`tZWKZWDiuKHC*hag zn)z*-Qbl`lOIuQbJI8&ZP%IUY9jo~MogJgo(5MO`oj5zIs7{S|(Oy9_jFxCihJ1(t zMUBdg{`>&$FgksDc5Wz`An1fYzm>|Yr^pi(4{Fi<`#r^Xhru*RnhM6KfQ@7GWmx3C=Ko`0j!sSkPS66k1NUI-fN(zsv*4RQ=)OPxJnR;d2p*+bRzHJ8el0aqeP$gjxKiHa~s-)MR+$F{YyXQYYr=+A5AbwPgcFR`dw|n^T zAw&fYiwX$B59o9MnY%$1w6oBOLa9Qdt6PgVV+K8Xt^oIG14H`b=hq;X0zX{0v;=t- zqp)?>X0D+bBm^^?wr|ai0ki-PKow{*ux%0&0E92A$_3#8fk=X6`_J!`PaZ^`y2XK6 zJ^FW_@h7}?MQz-M<0w+v^!L`oFKk+A$i!;jn)#|k!uBC7e>>5MMQ^Qo0SOPV3nTEN zNP@iO*_!>AZCTcu2F4you&k}~6()qD_F#Hg1If+ShdLmd$-obO|BU{RMe#-=aU8X6 zt3_;WZ*3qEM%}=n1WPvy5mynfFej7<>jL;tma98tY_?X#tP5axh|>J6{?;hp`^=}& zU8S!!hW*zhx-|_NPO$&fSD?mhOKzE|2nP~&4ghZ|V}IV=3_@m4EEhmcVp)iRadRvp z2nP*q_Bldr(O$yO8Hd^dmLvJ@p-S`9@^?qbmL2O^uYv**h~tBg!+Qnk z4>X6wC!oNEoXd46uS8kU$V-p8Z8N|+f+97JLtszvp4}WEp&hje(OAKS+n#=Npdu5} zCrmp}89si$Drz4M1#O%V8;1Y|J#2ai9c+!H?AZ~ehI_lET)OvYo4UNuf$U6G)7IRv zZqD7Vc!@n4JaQaBJXL(%cXJYn6SMoD_g>+V)7S%!qtQn+n<6m9YKH$a+X|x$$IBUW zgTTQpH9?lyLa$x|^~w%bs!Fdp+8!c{xs5Q9;j%FLe)k#}l2}zdVeqH`YMo#kLSkXG zN_9hAa=Iz7K}_}KIWp?zADQKV4D!G^0bsaFNw~V@Y~%)7|Ec0jJ?= zCEg238>f$TWv==IH#{F&GB{egG}zqcw&ksRx?y*COkDV7&FZY(UaV1UQQGTScK61- z-f~kt>zm!F9zh48=f&HC)t5S?Ug9GX2r_F#RfD6Nv|C;mTGht*PiuT$F96pUdJ~}3 zekpxk0JprUL4+nbdpt!SF=An-HdHru&x9FaF`}pqIu&8Vf(!gsmAG8|WYOZw_rM_U zaI!cNxik#85;5~k5-$9ygcJSG^BVV!8B3p-xrV$2m(iyQQNWjNGYmz6?A)=ggkBM~ zPsBrL6z;YW0rXQd!x5afU&81Bbo+zDPz||^C-K1GEEyJiRlzYbpY$qv0yj`s8^hVp z^+bR|KuAS^nE@Bc1k9J!<#kKxTO=_dqtNo4s-Q_K{D-tV|JsuvsYF3G3T%mt%g8%x z0Vy~DRSr!A{|ZeWW^^7k&4X=_&4Do>b~fU5!vHCEbnFN;8~^zq>Vs5-10(#07`#+3 z(cID!+JQ%SA(#czmc)1i2QIjUPG1}PAu#_fEf%JhNt6z-&4pREB#A8a4pfeS(wYGP z3%&2@K<)Nt4rh`q}~Jji%^-W7e_ z?-J;zdA|emc4~_&qdX zzqH}`(4z6a`Vr%%pT=HO%H8oJA-D8i;57{#mKyA%-COt!L!4K?=_<1JJ19m+HAwZl z-o7!w7u98c%xkq%)j(M_itgNQPh+bEzWIwwI=*XPNBo9_XfzA#SKrq3wU!!pH`j|U z7c6)TpIVygn-WX5Wp;=z55UKNw=th^sQ-nZDwvQKJTdIB*cZe(ZT#TGxLC55=yIMf zK7803cr#4urBGoVAEV)px<%q=0s6z|c=zsGT{(3Z<0jUAEzRChl@?`?3-@U?Cl7h& zJ>pv{Uu<=pB%Sh2sRPe$wf$^4nwtG(?cK5;A9qQ(O22Gm8<5QtR62;@!A;vpIGAo& zYuI{Y?uC8tD@nuU8VfFUYPaLN%~d!`E)9J=kaXMj-O4eR8O?>;M|x5x(P5MBS|7Lf zx0qOstE}l>AFdBfdVueCnCKNal)|a{vZAH*SH-lN&i$djWvVII55w%RsjD2K?QVJW zdJDYzMA=t5I`ZmHHMqD^a;8 z$%453Bp)okzuS2_n4-Av=782W%5rsU%r{0i^L5uuk!D9j1!K7x+dO@5@`Ig$@FAF9qF!G%*J`-RpV}kkdOXlb z@9Y%QJMST}7f5LM<@2_;RN!xh_+AVd=nK0N@bzVYcZt=E9Ru-D9lIL|i$sIc5zpzbWzX%T8|MY!EtM(y`O^@b zY$lhP*>otwHT+IMUBva?h8F$jDY+h&w#+L(D!3Z=+cBNmE+}g0+^ zWq&A%(z|e|`^!@Q&7f^BtyF~l)ulEpbR*6(g`S|TFA-(En*8*D0mH2>xj@g#Qz;dXW)dE98ydr)!VS8Wx@yF9Na{*=o%o4T%=C1z&xT)eGQn zKGWmkR%OF3F+XQBQDZDo$+Ob)iJ$krXhbdFy#D=#zq}StO(<`S_ITMidYDNJfgsc; zg#e!XiWamk7hMNV$(c<9JiOvSw1fb!%*TGQ?(loch91)1p*o(j)YEk&T%g4h*6-x=xw15xlg0blPuaODR_07< zNu=if6}Z38ni?OgClM{(<9Ye-C&30@Q#q;NyJsX=uy*9w$HJcA*t9l;gV{3GPtppI z8mFloR^Zk6_&l<9GK24mP1|w1Z4s52wm^Uo>CFMSiN5prf!^fdqUIMl8)ZN4*Txnk z@dTqUMDH&+9`q;0bSSw(-T1?3*2}SLAWSk_#*~hvJ7H3;3fGyF=P*E&BhOIdD=yGuas}k$&NekQI@U!*e^GWWX zE;%s`T#@_IN+GqDHT%>1seN=36XQwEK9Pcf1=bgtWA81PeOts?d(&hyp8y*QK6RgA z#L{0GAW&%_bp2fSKRpa%osL)!cC!)x!)*eqjZJ({bi4zB2=Kr|9pbS-pyH zb9uSv)!Vl2jVq-`%W`ig3)zob-xxo+OM0chK(^OA&>$3hYX}`K-tE)?FLBU!PAGaw znKUMsm?ML39m?Y>&2k!cfeunxlOPG8ffcLbfzez6>R+_8;R^qX4iWSyB6l16*GP!j z0ny?ZLh&%+HVabFX>SRIJcNHUyo3|TvE_g6y0Air(fk|L!EFbGEL{1&R-z19BNXtM zY(4(8B`etvkN^Bz346Ka7YiSRVJai6F5>n7{hJQ`|5k=anDFoKkZ3^HuB6#z--hp% zAy8-KF6|}-Gv;CCtXVMiY?jJv;h8{-q-)<5w{cEB^+Hd3?w+~I(1DeQCj#uFQ`cI| z8_KBopnO>;2EJf;Q@dIieR7=}7`~jH!sO8Ph*cu|VYH{XNIvtYspi?w!J6+UBb#<_)jK*W?Y7Z5!|fwN63(?NDG`@k z-tS43BsI5NT2qA3x-7JuL8(W!H7l9X`OW!_@!{%rIoF<<=hQh@A3qPOZ?JNr(5x-z zV;1F8WCqlB4+6bum@33?P^6av57lxsTho8#VluF9h86{7%Tn)6+WE9A^BO8O;GdA1 zPYWCnQ9#89M(>C(!-dm)jjmDjVYw6VK|VCUVc z4_omZ|JA6zpgYQEE7aD1VC8Lr{-N-s3d37)(q+)<#5IiYooCfs@{Nu5f`1OyLlJxP zb#MGqbrPQ{*X<(V<%0gSC<%c}b0V~Ca{I@0r?Qgv;SVwt$}ZWz=LpcgUOIM0!kIjs z!)?^tKyhHo?PqKGF|XaF)`?em_R-=$=>Xt&u?J#ThY0J*6B@vvi%Z;GaVrJ#z;V}} zEcJ-!zj{{cXL3o7f1|v>L8^od{B!pt(oz>X-Cau58f*X-Ojem%!so~3(d?UjpS!R61iVhLP>79=bwP|Aosm9`cx$4`8 zj;qXe_PeUiX89WFU4L~VAh@q+YVrI0)aT%1UUpwSrKjuaXNsDw3}g4;N7mWz4F~W^ zJgwop>mFR|@D6{X#0tY^5KWpZ*a%n{V7#HTfMeKmIw{p|W$qj>gpgj1E-*Fyn?dEU z)WV}rGbuA4WlD!OJQh^c&lm)akt+114^q%zU#E?tkN6# z6AUeE8r4@ybPk1Q*WAkQvo~k>?IZJD4sZ1HK`4|R+SpIIWE|3I;aCn=cOP|0rWWP0 zd&DK@-hPGj4)wCz5MShmwbTpBE|u9rnpi;+Z1782*<+X)-p54Ky!h@)Q*!?oC_e9YS%ePQiVht!<~r6X=Fd>e%ssl9UH$`V5^;{t9|(w-}Ff2YgK zBu~41ouiIGSl52lw?*5gz(Ow|Ij0(Qi`MbN^}g8%O~a`5A8RcEZ9Vvn zMemH-Lbtc?bQrY?9xmpUrg7bMA294V;XLK!S06dhqHawV@vBvxDn{jdF4lD_>0@W9 zL*RriL~uR;RZyoCpsb|8|uE)=upkDMgs!KZb(&1`6(QLVZnHIDc-x5!FLiV6;;G!G9ljHzMACPSoN5{o7P2$slS} ziuwk0?wzOJDIQJvP<=dUI&U&JRyBH|=<@zM35hI$l%*=>egnC*nU(RgE>k*OhMJ~9 z9ggZB(@Tw&_Z7c$_PoFT*Y~1z`zlUxXZgwN;(ada*2=!d$RuEcm|Im6gpnciQ7=IY z**|`oO;dK8I?wE7Ke_kLYmA~^MZbsGS})U^`S3T$#gemY?PI5}ESv%Q=OLksAFPgN zM4u2V9M_BKZKq?_K+##%R7C(*k9(y7hqwAG%&`UY!CEXM0|y zPUG|>HfotY=WfQ82X-aCBl%W-#rkB@^$0+o*}dr@6Hqpbl{Z5TRPS_tnRE0eadj^) z+qdkG3>9l}ygJkK`fE=WVdwO4x-;5gzvxs_56z8@58TysI`>(+QL(m@?cS?+VSwuh1;|J_J%FPEcr98wS@qE zS}eP%MbPU-2o+!u?s=!9Hd@%CXE67Xnj1t77yh~?pRRas(Q6*7*H7v|4fdyS*=4*t zl3mbW@va3wwfNwZdwI@CLJP}$Ne-?{4lj)ykK3J;R>2gX%#M@d25Pc(g-T5_`f z_$6 zsNbu>lOqzer<2)mQDb@eqamw zp!bc}{KTh_1$CA^J~6d6%r=&6-{;kpSWY?%f?3s;V{|2s(m~z27o?SC*VF^f2FZzh z+p0P0)_2WK8ezY3+>n$0F zN-yM0ECpERF0Q|4a@Y1d7g=sk0Y(9mRwCi}*HGbR22pY(G>h;09;|gAmwb6f@-lVy zN^!y`jVuA~5;d^dI*UR&@Z;;#w{FsOnE32zRhI8)ag=ooV$PtG2O!b)jGNPBQ3s{eK&Jt!&<_M2&-#v+&dNHM%Qk@cB+R270NCT5VBxlD#T+d zqw<))Hg(kFaP~!&eP!O5>iw6uohCFJvQ?wPZy3ry=4Qy{XxFTd+Q0H{U{dR`jK-AP z@j^G{^_gvf7yZ`0p7>@JcscC1z3+_{Im5~d+Elgsl*TVYhNTNuq*cY=2}OAo z1Xv{-$xrg=G{h(A@bgC5C3|f^wX+?RV7vO$3W76H^--QLIymOS&cFZWFfwazukjzI zqoX?u^P?9wA|)^r!Wqx2^X}T4tM~oc==L9qq6`UYhJwDVs@X7QdB0nJ)cyzDIxC_bJeU*G13}yMntMr%J?T_tS@GyQS$d-K z#QXtu5%a*pBgX6UtHiC2j^i@g_5(1Qg;QAS*Z0e+@Cm@a3>_A~4_Qg18 ze_~>SiU*9pWi#+dFknBY0TL~sVeY^Nw=xM$xDLBdlBx9)W{yX&8NhQudpUks#h32_ zw6K1RcGoAxDYO7K2ckc^XH3W}@Ea1n6r*r*dTg*If_CFN+$1U}C@7vMSb~&N*guIH z6}x;7sMcD4zfKX#0YPVkf=zG}XrWTTNQyZAVqui7vYe`2S4QPtuH~hu{Z{zhKi0`p zcJRrnpmi9HEi9fga4c^M3eP%MCM(2_z(b)Hsm40<3BCS~Vp*c&fuq-2+kJjGyPZ!7 ztBNE{R?JgaGdoAe0YWy6n|$+Wue_!Ld0YI*_!ZgSjF@v{{_i~;2Wlcuq4G@Z!yaNL z14r+B<4Xjs;Y%r}S&Cr_a|31A&exQ02uqT#Zj5zNWkx!Aa;{1rJsNud4E^M@E5(UB zGH>c%xaL`U=(G0s<3YZWytr~H^?$Rr{ zV=8BhhF0#`(z0{0%YB55=qM?6d2HT6TE8{ zX3hl0MX!zBYGCBeW2YnH77Zz0sF5oAQFw3_W@~)-^ic zVW+B_ZCkCNsQ@twW$3L#i$TZf`oyoM1yW14MN5N@c*$EP3wG;Y7dtLLC7#y$Ah_%oE3W}Z7VnmofI_W0ooS6|O(zeRJFcNIc z63_s>pKrt0NX%4?&eNz$Ab0B{v5pygTWGH9M5(X*br0hh^9j+ zvB{luI{O`1T@9eyh3YF(y_lrK#$?Ip5fPAI6e^Iz-o`t<&#+`C!_~yxc`q&O48Ut9 zpVJ^lgmXS5qo7EA@LoM3-H7}A+UYtp2e&yv9D0Ql0dPVT z2<0{Y83?sZguw(?j3}G(zRzhMCZ?GI1FzV{4Fz`6WkaRzQ~&Hv0EUII#&KY3+MZ4s zpzlyEvIA>}ZfwVW?A%vp2@)n|{93_Pa{M&)BgoY5V;Z?eP3sq*OZoHxpz@fqpYLxa z!-7&%*tvaUxd4_glDB9PhCZf1MhHyReP=tnkLL=4Q7$po!T&zAd&jpUcI>|MxrDSQ zuKZbGie8pgFwCdTX3g*~9e{f?`g|9$xSsCcm7o$4^7fGh3-Cv^Jlf5A0kw~kKA`3X zQ~7LZeJ}wC4Cf3nJNhJV>icGYBxPzvkC@OdO$=!&SGRjeKA14%C1S zA#mtV0UaHG(qZs$nfzM-%=B+NVvBdBIh2mjx9N7HM_c-RcoPehez64T8>|cV76Jd+ z72!Yk%DHfZEE>;Y&yN;sE)~J}i+&51q@T!}_U4Q#lQqHW#y3aCtTaLg1WrK6k#BoY z7~#g!C3!fSW03bNA`v;@%Yf{Dz8Pu8fs> zvjeEp9O&txv`!t35O_oXGeS05NhEOZ8eg62E4^#%7g6w?cXyzCl0JFmOGAubv`ebw zCg9MJjX_TVeBVC%l3z@ zOk;U4Hr4_TLnBJi$Z&FH2k74{KLU&lDtC>Q-`-?IaB}HJ+V`t4+|%K~?TCHC^wrEv zt+ce{Y#mgZh!g5jVOJ-sX9bkXyZCu)*s^y3#6Rs(fYXHCOgUx#9Zm>9fohXr;!n2= ziUVc_oh10480N^3Y@A7B18%5jSE~$dEEiwSkGlw40s)S_2Yw55KU(aVF+gJO`e}oz z^r~ylz+V&>xXz>#@d2@4CYy#?N11LE0=Fh^2w+GyZ>jqmVTd7B4`^vYk*SEtIedEB z3_xR9P=;RQ2_(%B3!d3zn16ALbva&chsvA*5xIfBUI@W( zNiUU;&PT77N1MPt{P%AA{VK*l;b7dH^TuhX0ZsG+WZ?nuf!P8`ADI=ONv%~uIf%sp zfwU4WxS=%)aZx${0A zZAZ|74rExD1?a%UAUPv$(DWm%paY*`F%g?v29Vt05W~mvXVRH!UuXCAiZS?LbQKMEW6kqmX1RATckMKggbr#;2$Zd5`;VrNKQYMT>tC!ujREraLby(YKh0)LGS|? z4(Q!D?d%RcQkK0X4|ZUAk;LO=bOq3k7mosHytIa5%ykf2JFqYbRjDNt+Mj>d%R;G^ zjQ)Q%X0c0y_;#8zuz|bYK?LcwD>p+KXqs!WTqJJ+Pz$ZLOeH+?E8u!ILlN@>gtuJ$ zQw!N2ET{VYgc@wJyF##-3&Kz8fz0jmFTz(Fy@x;f=~LSddb1H~MABDa!G4*Hb;>w5 zs))s@-kuI>8{>>}VWWcOhy%i#ONPZB=uLB-88pp?q9IP0Bb)yS)8ezwb@4S) zE2EhJ=RV31CJqO2xG}-p#zZNUR|WJ*d)l4RyHvbq5vs2E#_CMtncb4lv6z(2qjeAd z96_rfDEnRk3$m?qobZ6^0D#cqsCE9G&vcw9)MJ) zdyMB-$~Kr4v(TJEuZ?wKul3dUFFsd z)iY#F4I&g_+nnBqqt4{z_wg74V1vsFF!dZi-elDF2tX$#i!<6okSNtC1Bph=+2OaE zfoJOuA=d_1|3cE;+1c|)@g2w~Vuy6_a1Mm(GzexZYT0m%2Z3e&Ij!IvKdEWG8f^>4 z|J|%QJUIT+eLTV9+!A27+ks9Y z#|k!6f>wuh@U+}o{B>#obd6jshrz0IGPfSr&it#N5fsc_6F3Y?n3^K$+A`c&ZpoNz zg;@gN9#vAz0LnK1Fmu3w?6b&68!zB=C;;oA^Uk2o_&FW4=@BeU`LamKO{k?B2GG