Target is in source's history #478

lucpeterson · 2024-05-16T17:18:16Z

Adds a GitHub action to check that the target branch is in the history of the source branch. This implies that the source is downstream of the target and it's valid to compare them. This should help catch future errors involve CHANGELOG comparisons

* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks * Fix filenames for OpenFoam tutorial (#475) * bugfix/deep-merge-existing-keys (#476) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * remove DeepMergeException and add conflict_handler to dict_deep_merge * add conflict handler to dict_deep_merge * fix broken tests for detailed-status * use caplog fixture rather than IO stream * add ability to define module-specific fixtures * add tests for read/write status files and conlict handling * add caplog explanation to docstrings * update CHANGELOG * run fix-style * add pytest-mock as dependency for test suite * clean up input check in dict_deep_merge * Improved Info (#477) * Add merlin version to banner * Add python package info to and clean up 'merlin info' * Add some unit tests * Force GitHub runner checkout to grab the whole history, fixing CHANGELOG test bug * Update CHANGELOG to show bugfix to CHANGELOG test * Target is in source's history (#478) * New github action test to make sure target has been merged into source * Fix link to merlin banner image (#479) * bugfix/status_nested_workspace (#480) * remove a merge conflict statement that was missed * have status ignore nested workspaces and modify merge rules * update CHANGELOG * fixed issue with escape sequences in ascii art * apply Luc's suggestion * add setuptools as a requirement since python 3.12 doesn't have it natively * modify unit tests for status to use pytest 10000 rather than unittest * update CHANGELOG * add fixtures for status testing and add nested workflow test * update CHANGELOG * bugfix/celery-chord-error (#481) * remove a merge conflict statement that was missed * add celery results backend patch to stop ChordErrors * add MERLIN_RAISE_ERROR return code * add tests to ensure chord error isn't raised * add RAISE_ERROR to docs * update CHANGELOG * fix lint issues * up the sleep time on the chord error test * add new steps to the chord err test spec * add tree statement to the new test for debugging * upping sleep time to see if that fixes github action for python 3.7 * change sleep time for new test based on python version * run fix style * remove specific sleep time for diff python versions * release/1.12.2b1 (#482) * remove a merge conflict statement that was missed * bump version to 1.12.2b1 * bugfix/flux-nodes (#484) * remove a merge conflict statement that was missed * fix flux node allocation issue * allow for vars to be used with nodes settings of workers/batch * add tests for var usage with nodes * update CHANGELOG * run fix-style * bugfix/flux-nodes-prior-versions (#487) * add a version check for flux when getting node count * update CHANGELOG * add major version check for flux * Change Task ID to directory path (#486) * Modifying task id to include directory --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov>

* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * remove github text that was causing errors * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks * Fix filenames for OpenFoam tutorial (#475) * bugfix/deep-merge-existing-keys (#476) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * remove DeepMergeException and add conflict_handler to dict_deep_merge * add conflict handler to dict_deep_merge * fix broken tests for detailed-status * use caplog fixture rather than IO stream * add ability to define module-specific fixtures * add tests for read/write status files and conlict handling * add caplog explanation to docstrings * update CHANGELOG * run fix-style * add pytest-mock as dependency for test suite * clean up input check in dict_deep_merge * Improved Info (#477) * Add merlin version to banner * Add python package info to and clean up 'merlin info' * Add some unit tests * Force GitHub runner checkout to grab the whole history, fixing CHANGELOG test bug * Update CHANGELOG to show bugfix to CHANGELOG test * Target is in source's history (#478) * New github action test to make sure target has been merged into source * Fix link to merlin banner image (#479) * bugfix/status_nested_workspace (#480) * remove a merge conflict statement that was missed * have status ignore nested workspaces and modify merge rules * update CHANGELOG * fixed issue with escape sequences in ascii art * apply Luc's suggestion * add setuptools as a requi 9E12 rement since python 3.12 doesn't have it natively * modify unit tests for status to use pytest rather than unittest * update CHANGELOG * add fixtures for status testing and add nested workflow test * update CHANGELOG * bugfix/celery-chord-error (#481) * remove a merge conflict statement that was missed * add celery results backend patch to stop ChordErrors * add MERLIN_RAISE_ERROR return code * add tests to ensure chord error isn't raised * add RAISE_ERROR to docs * update CHANGELOG * fix lint issues * up the sleep time on the chord error test * add new steps to the chord err test spec * add tree statement to the new test for debugging * upping sleep time to see if that fixes github action for python 3.7 * change sleep time for new test based on python version * run fix style * remove specific sleep time for diff python versions * release/1.12.2b1 (#482) * remove a merge conflict statement that was missed * bump version to 1.12.2b1 * bugfix/flux-nodes (#484) * remove a merge conflict statement that was missed * fix flux node allocation issue * allow for vars to be used with nodes settings of workers/batch * add tests for var usage with nodes * update CHANGELOG * run fix-style * bugfix/flux-nodes-prior-versions (#487) * add a version check for flux when getting node count * update CHANGELOG * add major version check for flux * Change Task ID to directory path (#486) * Modifying task id to include directory * Adding Several New Unit Tests (#490) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * split up create_server_config and write tests for it * add tests for config_merlin_server function * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * update CHANGELOG * add Ryan and Joe's suggestions * update tests to use newly named functions * fix linter issue * release/1.12.2 (#491) * update version to 1.12.2 * fix style issue --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov>

* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * remove github text that was causing errors * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks * Fix filenames for OpenFoam tutorial (#475) * bugfix/deep-merge-existing-keys (#476) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * remove DeepMergeException and add conflict_handler to dict_deep_merge * add conflict handler to dict_deep_merge * fix broken tests for detailed-status * use caplog fixture rather than IO stream * add ability to define module-specific fixtures * add tests for read/write status files and conlict handling * add caplog explanation to docstrings * update CHANGELOG * run fix-style * add pytest-mock as dependency for test suite * clean up input check in dict_deep_merge * Improved Info (#477) * Add merlin version to banner * Add python package info to and clean up 'merlin info' * Add some unit tests * Force GitHub runner checkout to grab the whole history, fixing CHANGELOG test bug * Update CHANGELOG to show bugfix to CHANGELOG test * Target is in source's history (#478) * New github action test to make sure target has been merged into source * Fix link to merlin banner image (#479) * bugfix/status_nested_workspace (#480) * remove a merge conflict statement that was missed * have status ignore nested workspaces and modify merge rules * update CHANGELOG * fixed issue with escape sequences in ascii art * apply Luc's suggestion * add setuptools as a requirement since python 3.12 doesn't have it natively * modify unit tests for status to use pytest rather than unittest * update CHANGELOG * add fixtures for status testing and add nested workflow test * update CHANGELOG * bugfix/celery-chord-error (#481) * remove a merge conflict statement that was missed * add celery results backend patch to stop ChordErrors * add MERLIN_RAISE_ERROR return code * add tests to ensure chord error isn't raised * add RAISE_ERROR to docs * update CHANGELOG * fix lint issues * up the sleep time on the chord error test * add new steps to the chord err test spec * add tree statement to the new test for debugging * upping sleep time to see if that fixes github action for python 3.7 * change sleep time for new test based on python version * run fix style * remove specific sleep time for diff python versions * release/1.12.2b1 (#482) * remove a merge conflict statement that was missed * bump version to 1.12.2b1 * bugfix/flux-nodes (#484) * remove a merge conflict statement that was missed * fix flux node al 4662 location issue * allow for vars to be used with nodes settings of workers/batch * add tests for var usage with nodes * update CHANGELOG * run fix-style * bugfix/flux-nodes-prior-versions (#487) * add a version check for flux when getting node count * update CHANGELOG * add major version check for flux * Change Task ID to directory path (#486) * Modifying task id to include directory * Adding Several New Unit Tests (#490) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * split up create_server_config and write tests for it * add tests for config_merlin_server function * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * update CHANGELOG * add Ryan and Joe's suggestions * update tests to use newly named functions * fix linter issue * release/1.12.2 (#491) * update version to 1.12.2 * fix style issue * Refactor/distributed-tests (#493) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * Include celerymanager and update celeryadapter to check the status of celery workers. * Fixed issue where the update status was outside of if statement for checking workers * Include worker status stop and add template for merlin restart * Added comment to the CeleryManager init * Increment db_num instead of being fixed * Added other subprocess parameters and created a linking system for redis to store env dict * Implemented stopping of celery workers and restarting workers properly * Update stopped to stalled for when the worker doesn't respond to restart * Working merlin manager run but start and stop not working properly * Made fix for subprocess to start new shell and fixed manager start and stop * Added comments and update changelog * Include style fixes * Fix style for black * Revert launch_job script that was edited when doing automated lint * Move importing of CONFIG to be within redis_connection due to error of config not being created yet * Added space to fix style * Revert launch_jobs.py: * Update import of all merlin.config to be in the function * suggested changes plus beginning work on monitor/manager collab * move managers to their own folder and fix ssl problems * final PR touch ups * Fix lint style changes * Fixed issue with context manager * Reset file that was incorrect changed * Check for ssl cert before applying to Redis connection * Comment out Active tests for celerymanager * split up create_server_config and write tests for it * add tests for config_merlin_server function * Fix lint issue with unused import after commenting out Active celery tests * Fixed style for import * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * Fixed kwargs being modified when making a copy for saving to redis worker args. * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * Added password check and omit if a password doesn't exist * update CHANGELOG * change testing log level to debug * add debug statement for redis_connection * change debug log to info so github ci will display it * attempt to fix password missing from Namespace error * run checks for all necessary configurations * convert stop-workers tests to pytest format * update github wf and comment out stop-workers tests in definitions.py * add missing key to GH wf file * fix invalid syntax in definitions.py * comment out stop_workers tests * playing with new caches for workflow CI * fix yaml syntax error * fix typo for getting runner os * fix test and add python version to CI cache * add in common-setup step again with caches this time * run fix-style * update CHANGELOG * fix remaining style issues * run without caches to compare execution time of test suite * allow redis config to not use ssl * remove stop-workers and query-workers tests from definitions.py * create helper_funcs file with common testing functions * move query-workers to pytest and add base class w/ stop-workers tests * update CHANGELOG * final changes for the stop-workers & query-workers tests * run fix-style * move stop and query workers tests to the same file * run fix-style * go back to original cache setup * try new cache for singularity install * fix syntax issue in github workflow * attempt to fix singularity cache * remove ls statement that breaks workflow * revert back to no common setup * remove unnecessary dependency * update github actions versions to use latest * update action versions that didn't save * run fix-style * move distributed test suite actions back to v2 * add 'merlin run' tests and port existing ones to pytest * update CHANGELOG * add aliased fixture types for typehinting * add tests for the purge command * update CHANGELOG * update run command tests to use conditions when appropriate * start work on adding workflow tests * create function and class scoped config fixtures * add Tuple fixture type * get e2e test of feature_demo workflow running * add check for proper variable substitution in e2e test * generalize functionality to run workflows * add create_testing_dir fixture * port chord error workflow to pytest * create dataclasses to house common fixtures and reduce fixture import requirements * fix lint issues * remove hard requirement of Annotated type for python 3.7 and 3.8 * remove distributed test CI and add unit test CI * fix typo in fixture_types and fix lint issues * run fix-style * add check for python2 before adding that condition check * convert local run test to use StepFinishedFilesCount condition * update CHANGELOG.md * fix problem created by merge conflict when mergin develop * remove manager functionality from this PR * update README for test suite * change SIGTERM to SIGKILL * update Makefile to include new changes to test suite --------- Co-authored-by: Ryan Lee <lee1098@llnl.gov> Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> * Drop Python 3.7 and Add Python 3.12 & 3.13 (#495) * drop support for py 3.7, add support for py 3.12 and 3.13 * fix docs build issue * remove py 3.7 and add py 3.12/3.13 to integration tests * update Makefile to use target_version variable and update spellbook requirements in examples * Requirements fixes (#501) * add fix for broken deepdiff dependency in py38 and remove py37 specific requirements that were missed * remove try/except imports that are no longer necessary * update CHANGELOG * docs/api_docs (#496) * add/update docstrings for top-level files and enable API docs * remove Generator type hint * fix invalid escape sequences * add docstrings for the common/ directory * fix styling and add in cross-references to portions of Merlin codebase * give code blocks types for formatting * add api docs for half of the study directory * add API docs for study.py * ignore the data path for API docs * finish API docs for study directory * finish api docs for spec/ directory * update CHANGELOG * final cleanup of API docs for common, spec, and study folders * finish API docs for examples folder * began work on API docs for config folder * write API docs for server utils * finish api docs for the server directory * finish API docs for config directory * final cleanup * fix doc build issues * add section explaining API docs to the docs' README file * run fix-style * update readthedocs to build with latest python * fix too few arguments to generator issue * rename in MerlinStepRecord so that it's hidden again * add most of Charles' suggestions * remove unused openfilelist.py, opennpylib.py, and merlin_templates.py files, and add remainder of Charles suggestions * run fix-style * rename load_default_user_names to set_username_and_vhost * Update README.md to remove lgtm banner (#488) * Update README.md to remove lgtm banner * Update CHANGELOG.md * Delete .lgtm.yml * move changelog update to the unreleased section * remove lgtm.yml --------- Co-authored-by: Brian Gunnarson <brianfunnarson14@gmail.com> * Refactor/simplified-ci (#504) * split python and singularity setup into individual actions * add shell to new actions * try to fix shell issue * remove install-deps * reorder cache step and check pip version in the output * update CHANGELOG and make path to definitions.py relative * fix style issues * move cache check to setup-python action * fix an issue with docs build on python 3.9 (#503) * fix an issue with docs build on python 3.9 * adding ChatGPT suggestion to use get-pip.py script * add python version check to reinstall pip CI step * add reinstall pip step to all jobs that need it * save get-pip.py to tmp folder instead of directly to the repo * update CHANGELOG * fix .wci.yml links (#505) * Refactor/config (#498) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * Refactor/config-defaults (#497) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * remove debug step from github ci * add user permission check to local test suite * create a CI workflow for push and pull_request_target so secrets can be accessed * add link to article where I found this solution * add a merlin info test so we can see what server CI is connecting to * add unit tests for n F438 ew config commands * add MerlinConfigManager class to replace config-related functions * add unit tests for MerlinConfigManager and update integration tests * comment out the pr-target workflow since it's not working * run fix-style * rewrite tests for find_config_file * remove uses of CONFIGFILE_DIR and chdir from configfile unit tests * run fix-style * re-order the find_config_file function to be local, config_path, merlin home * remove pr-target workflow, add missing docstrings, and run fix-style * add create and use commands, update MerlinConfigManager as necessary * update test suite for config command * update all documentation where 'merlin config' is mentioned * change merlin config to merlin config create * update CHANGELOG * run fix-style * update GitHub workflow to use 'merlin config create' * try to fix f-string issue * add Charles' suggestions * fix broken tests and security issue with new log statement * remove app.yaml file that was accidentally pushed * fix config file validation as Charles suggested * Refactor/server-config (#506) * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * change print to log statement * bugfix/local-config (#507) * add local mode config initialization * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * add unit tests for new functions in configfile.py * run fix-style and update CHANGELOG * change defaults to not use getpass * run fix-style * add changes that Charles suggested * feature/monitor-auto-restart (#494) * add classes for backends to help abstract them * establish templates and basic functionality for database interactions * flush out creation and retrieval of studies and runs * finish study/run storage, retrieval, and deletion to db functionality * add StudyNotFoundError and RunNotFoundError * add ability to dump run metadata to merlin_info directory * add new Celery signature to mark a run as complete * create merlin database commands * integrate database changes with monitor command * run fix-style and fix lint issues * update CHANGELOG and run fix-style with python 12 * last run of style fixes before push * run fix style with python 3.13 * create foundation for monitor auto-restart test * move get_backend_password import inside RedisBackend class * fix lint issue and comment out monitor test for now * fix redis connection so it works for ssl and non-ssl connections * add test for workflow auto-restart functionality * run linter * clean up print for DatabaseStudy and add -k option to delete all-studies * add docs for new monitor and database command * run fix-style * update built-in celery settings and retry exceptions to better handle timeouts * add new Monitor classes to handle the monitor command * update CHANGELOG and fix monitor test * fix lint issues * add worker entries and interactions with the database * add ABC class for database entities * add multi-run and multi-study queries, queries by study id, and queries by run workspace * add ability to delete multiple runs, studies, and workers. Also add ability to delete study by id and run by workspace * update docstrings for db and backend files * run fix-style * add repr method to HasRegex to make parsing GitHub CI easier * add ability to delete everything from db and retrieve connection string from MerlinDatabase * rename DatabaseRun and DatabaseStudy to more appropriate RunEntity and StudyEntity * split WorkerModel into LogicalWorkerModel and PhysicalWorkerModel * split redis backend class up into store classes for different entities. Removes db worker interaction temporarily * add logical worker functionality to database * add CLI functionality for physical workers and clean up MerlinDatabase * add physical worker db entry creation on celery worker startup * comment out code that's breaking worker launch * fix bug with get_all_runs method not existing anymore * run fix-style * fix bug with load/delete of physical workers and add functionality to set worker status to stopped * fix issue where ids were passed to wait_for_workers instead of names * move stop-worker celery functionality to a signal handler * add Logical Worker with ID 7620359f-6deb-0056-8956-39495dba8e59 ------------------------------------------------ Name: worker2 Runs: ['2e12e7cd-e7b9-468b-92c7-1ba1dffba852'] Queues: ['[merlin]_sim_queue', '[merlin]_seq_queue'] Physical Workers: ['dc863387-6b54-4a93-8a27-507bf0fd789e', '41a28c9e-f58d-49da-87bf-cea1569398b9'] Additional Data: {} Physical Worker with ID 41a28c9e-f58d-49da-87bf-cea1569398b9 ------------------------------------------------ Name: celery@worker2.%rzadams1011 Logical Worker ID: 7620359f-6deb-0056-8956-39495dba8e59 Launch Command: None Args: {} Process ID: None Status: WorkerStatus.RUNNING Last Heartbeat: 2025-04-14 12:40:05.786023 Last Spinup: 2025-04-14 12:40:05.786028 Host: rzadams1011 Restart Count: 0.0 Additional Data: {} Physical Worker with ID dc863387-6b54-4a93-8a27-507bf0fd789e ------------------------------------------------ Name: celery@worker2.%rzadams1010 Logical Worker ID: 7620359f-6deb-0056-8956-39495dba8e59 Launch Command: None Args: {} Process ID: None Status: WorkerStatus.RUNNING Last Heartbeat: 2025-04-14 12:40:05.027347 Last Spinup: 2025-04-14 12:40:05.027353 Host: rzadams1010 Restart Count: 0.0 Additional Data: {} Run with ID 2e12e7cd-e7b9-468b-92c7-1ba1dffba852 ------------------------------------------------ Workspace: /usr/WS1/gunny/scalability_testing/studies/long_running_wf_20250414-124613 Study ID: 64def1c2-8ee8-40a4-9f74-36d5b4872b9e Queues: ['[merlin]_seq_queue', '[merlin]_sim_queue'] Workers: ['7620359f-6deb-0056-8956-39495dba8e59'] Parent: None Child: None Run Complete: False Additional Data: {} Study with ID 64def1c2-8ee8-40a4-9f74-36d5b4872b9e ------------------------------------------------ Name: long_running_wf Runs: ['2e12e7cd-e7b9-468b-92c7-1ba1dffba852'] Additional Data: {} command * factor out common code from entity classes into mixin classes * run fix-style * remove process kill from celery shutdown * write tests for high-level modules of backends folder * add tests for the RedisBackend class * write unit tests for redis_logical_worker_store.py * add unit tests for RedisPhysicalWorkerStore * add tests for RedisRunStore * move db entities to their own folder * update version to 1.13.0a1 * add init.py file to backends/redis folder * change the monitor test so that it has more samples and purges tasks rather than stopping workers * run fix-style * fix error from merging develop * fix import issue with enums * condense store logic into base class and mixin class * add tests for new combined logic, remove tests for separated logic * run fix-style * fix imports and string representations of entities * update logical worker queue entry to be a set instead of list * make worker queues a set and fix str representations * update docs for database command * run fix style * change some info logs to debug * fix the majority of the docs warnings * fix docs warnings and run fix style * finish the database command docs * fix log messages and bug with deleting physical worker entry * add log files per Charles' suggestion * refactor db entities to share additional common logic * fix bug when deleting runs by workspace after workspace has been removed * add unit tests and some integration tests for data_models and db_commands * add unit tests for entity and mixin classes * refactor MerlinDatabase to use newly added entity managers * move mixins folder into entities folder * run fix-style * move mixin tests and remove todos from physical worker test file * add tests for the merlin_db module * add tests for entity managers * run fix-style * update CHANGELOG * fix unit tests for python 3.8 * fix typo * log warning if workspace doesn't exist rather than error * remove a function that was added by accident in merge from develop * fix import issue * reorganize redis_utils as they're no longer needed * move serialize and deserialize to common utils * add base store and move save, retrieve, delete methods up to ResultsBackend * initial work on adding sqlite support for local runs * finish implementing sqlite stores * update docstrings for the backend package * move existing tests of redis/results backend after refactor * add unit tests for sqlite modules and move around fixtures * add tests for StoreBase * fix unit tests for configfile * remove duplicate fixture * add some docs about sqlite and move database docs out of monitoring section * first attempt at fixing port issue * fix lint issues * second attempt at fixing the tests and style issues * implement Charles comments * update CHANGELOG * Update copyright (#510) * add COPYRIGHT file, update LICENSE, and change copyright header in all files * update Makefile commands related to copyright/copyright headers * Update CHANGELOG * add a check for the copyright header in 'make check-style' * update CHANGELOG * release/1.13.0b1 (#511) * Potential fix for code scanning alert no. 23: Workflow does not contain permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * codeql-fixes (#513) * fix logging security vulnerabilities * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: Ryan Lee <lee1098@llnl.gov> Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> Co-authored-by: Wout De Nolf <wout.de_nolf@esrf.eu> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * remove github text that was causing errors * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks * Fix filenames for OpenFoam tutorial (#475) * bugfix/deep-merge-existing-keys (#476) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * remove DeepMergeException and add conflict_handler to dict_deep_merge * add conflict handler to dict_deep_merge * fix broken tests for detailed-status * use caplog fixture rather than IO stream * add ability to define module-specific fixtures * add tests for read/write status files and conlict handling * add caplog explanation to docstrings * update CHANGELOG * run fix-style * add pytest-mock as dependency for test suite * clean up input check in dict_deep_merge * Improved Info (#477) * Add merlin version to banner * Add python package info to and clean up 'merlin info' * Add some unit tests * Force GitHub runner checkout to grab the whole history, fixing CHANGELOG test bug * Update CHANGELOG to show bugfix to CHANGELOG test * Target is in source's history (#478) * New github action test to make sure target has been merged into source * Fix link to merlin banner image (#479) * bugfix/status_nested_workspace (#480) * remove a merge conflict statement that was missed * have status ignore nested workspaces and modify merge rules * update CHANGELOG * fixed issue with escape sequences in ascii art * apply Luc's suggestion * add setuptools as a requirement since python 3.12 doesn't have it natively * modify unit tests for status to use pytest rather than unittest * update CHANGELOG * add fixtures for status testing and add nested workflow test * update CHANGELOG * bugfix/celery-chord-error (#481) * remove a merge conflict statement that was missed * add celery results backend patch to stop ChordErrors * add MERLIN_RAISE_ERROR return code * add tests to ensure chord error isn't raised * add RAISE_ERROR to docs * update CHANGELOG * fix lint issues * up the sleep time on the chord error test * add new steps to the chord err test spec * add tree statement to the new test for debugging * upping sleep time to see if that fixes github action for python 3.7 * change sleep time for new test based on python version * run fix style * remove specific sleep time for diff python versions * release/1.12.2b1 (#482) * remove a merge conflict statement that was missed * bump version to 1.12.2b1 * bugfix/flux-nodes (#484) * remove a merge conflict statement that was missed * fix flux node allocation issue * allow for vars to be used with nodes settings of workers/batch * add tests for var usage with nodes * update CHANGELOG * run fix-style * bugfix/flux-nodes-prior-versions (#487) * add a version check for flux when getting node count * update CHANGELOG * add major version check for flux * Change Task ID to directory path (#486) * Modifying task id to include directory * Adding Several New Unit Tests (#490) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * split up create_server_config and write tests for it * add tests for config_merlin_server function * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * update CHANGELOG * add Ryan and Joe's suggestions * update tests to use newly named functions * fix linter issue * release/1.12.2 (#491) * update version to 1.12.2 * fix style issue * Refactor/distributed-tests (#493) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * Include celerymanager and update celeryadapter to check the status of celery workers. * Fixed issue where the update status was outside of if statement for checking workers * Include worker status stop and add template for merlin restart * Added comment to the CeleryManager init * Increment db_num instead of being fixed * Added other subprocess parameters and created a linking system for redis to store env dict * Implemented stopping of celery workers and restarting workers properly * Update stopped to stalled for when the worker doesn't respond to restart * Working merlin manager run but start and stop not working properly * Made fix for subprocess to start new shell and fixed manager start and stop * Added comments and update changelog * Include style fixes * Fix style for black * Revert launch_job script that was edited when doing automated lint * Move importing of CONFIG to be within redis_connection due to error of config not being created yet * Added space to fix style * Revert launch_jobs.py: * Update import of all merlin.config to be in the function * suggested changes plus beginning work on monitor/manager collab * move managers to their own folder and fix ssl problems * final PR touch ups * Fix lint style changes * Fixed issue with context manager * Reset file that was incorrect changed * Check for ssl cert before applying to Redis connection * Comment out Active tests for celerymanager * split up create_server_config and write tests for it * add tests for config_merlin_server function * Fix lint issue with unused import after commenting out Active celery tests * Fixed style for import * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * Fixed kwargs being modified when making a copy for saving to redis worker args. * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * Added password check and omit if a password doesn't exist * update CHANGELOG * change testing log level to debug * add debug statement for redis_connection * change debug log to info so github ci will display it * attempt to fix password missing from Namespace error * run checks for all necessary configurations * convert stop-workers tests to pytest format * update github wf and comment out stop-workers tests in definitions.py * add missing key to GH wf file * fix invalid syntax in definitions.py * comment out stop_workers tests * playing with new caches for workflow CI * fix yaml syntax error * fix typo for getting runner os * fix test and add python version to CI cache * add in common-setup step again with caches this time * run fix-style * update CHANGELOG * fix remaining style issues * run without caches to compare execution time of test suite * allow redis config to not use ssl * remove stop-workers and query-workers tests from definitions.py * create helper_funcs file with common testing functions * move query-workers to pytest and add base class w/ stop-workers tests * update CHANGELOG * final changes for the stop-workers & query-workers tests * run fix-style * move stop and query workers tests to the same file * run fix-style * go back to original cache setup * try new cache for singularity install * fix syntax issue in github workflow * attempt to fix singularity cache * remove ls statement that breaks workflow * revert back to no common setup * remove unnecessary dependency * update github actions versions to use latest * update action versions that didn't save * run fix-style * move distributed test suite actions back to v2 * add 'merlin run' tests and port existing ones to pytest * update CHANGELOG * add aliased fixture types for typehinting * add tests for the purge command * update CHANGELOG * update run command tests to use conditions when appropriate * start work on adding workflow tests * create function and class scoped config fixtures * add Tuple fixture type * get e2e test of feature_demo workflow running * add check for proper variable substitution in e2e test * generalize functionality to run workflows * add create_testing_dir fixture * port chord error workflow to pytest * create dataclasses to house common fixtures and reduce fixture import requirements * fix lint issues * remove hard requirement of Annotated type for python 3.7 and 3.8 * remove distributed test CI and add unit test CI * fix typo in fixture_types and fix lint issues * run fix-style * add check for python2 before adding that condition check * convert local run test to use StepFinishedFilesCount condition * update CHANGELOG.md * fix problem created by merge conflict when mergin develop * remove manager functionality from this PR * update README for test suite * change SIGTERM to SIGKILL * update Makefile to include new changes to test suite --------- Co-authored-by: Ryan Lee <lee1098@llnl.gov> Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> * Drop Python 3.7 and Add Python 3.12 & 3.13 (#495) * drop support for py 3.7, add support for py 3.12 and 3.13 * fix docs build issue * remove py 3.7 and add py 3.12/3.13 to integration tests * update Makefile to use target_version variable and update spellbook requirements in examples * Requirements fixes (#501) * add fix for broken deepdiff dependency in py38 and remove py37 specific requirements that were missed * remove try/except imports that are no longer necessary * update CHANGELOG * docs/api_docs (#496) * add/update docstrings for top-level files and enable API docs * remove Generator type hint * fix invalid escape sequences * add docstrings for the common/ directory * fix styling and add in cross-references to portions of Merlin codebase * give code blocks types for formatting * add api docs for half of the study directory * add API docs for study.py * ignore the data path for API docs * finish API docs for study directory * finish api docs for spec/ directory * update CHANGELOG * final cleanup of API docs for common, spec, and study folders * finish API docs for examples folder * began work on API docs for config folder * write API docs for server utils * finish api docs for the server directory * finish API docs for config directory * final cleanup * fix doc build issues * add section explaining API docs to the docs' README file * run fix-style * update readthedocs to build with latest python * fix too few arguments to generator issue * rename in MerlinStepRecord so that it's hidden again * add most of Charles' suggestions * remove unused openfilelist.py, opennpylib.py, and merlin_templates.py files, and add remainder of Charles suggestions * run fix-style * rename load_default_user_names to set_username_and_vhost * Update README.md to remove lgtm banner (#488) * Update README.md to remove lgtm banner * Update CHANGELOG.md * Delete .lgtm.yml * move changelog update to the unreleased section * remove lgtm.yml --------- Co-authored-by: Brian Gunnarson <brianfunnarson14@gmail.com> * Refactor/simplified-ci (#504) * split python and singularity setup into individual actions * add shell to new actions * try to fix shell issue * remove install-deps * reorder cache step and check pip version in the output * update CHANGELOG and make path to definitions.py relative * fix style issues * move cache check to setup-python action * fix an issue with docs build on python 3.9 (#503) * fix an issue with docs build on python 3.9 * adding ChatGPT suggestion to use get-pip.py script * add python version check to reinstall pip CI step * add reinstall pip step to all jobs that need it * save get-pip.py to tmp folder instead of directly to the repo * update CHANGELOG * fix .wci.yml links (#505) * Refactor/config (#498) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * Refactor/config-defaults (#497) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * remove debug step from github ci * add user permission check to local test suite * create a CI workflow for push and pull_request_target so secrets can be accessed * add link to article where I found this solution * add a merlin info test so we can see what server CI is connecting to * add unit tests for new config commands * add MerlinConfigManager class to replace config-related functions * add unit tests for MerlinConfigManager and update integration tests * comment out the pr-target workflow since it's not working * run fix-style * rewrite tests for find_config_file * remove uses of CONFIGFILE_DIR and chdir from configfile unit tests * run fix-style * re-order the find_config_file function to be local, config_path, merlin home * remove pr-target workflow, add missing docstrings, and run fix-style * add create and use commands, update MerlinConfigManager as necessary * update test suite for config command * update all documentation where 'merlin config' is mentioned * change merlin config to merlin config create * update CHANGELOG * run fix-style * update GitHub workflow to use 'merlin config create' * try to fix f-string issue * add Charles' suggestions * fix broken tests and security issue with new log statement * remove app.yaml file that was accidentally pushed * fix config file validation as Charles suggested * Refactor/server-config (#506) * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * change print to log statement * bugfix/local-config (#507) * add local mode config initialization * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * add unit tests for new functions in configfile.py * run fix-style and update CHANGELOG * change defaults to not use getpass * run fix-style * add changes that Charles suggested * feature/monitor-auto-restart (#494) * add classes for backends to help abstract them * establish templates and basic functionality for database interactions * flush out creation and retrieval of studies and runs * finish study/run storage, retrieval, and deletion to db functionality * add StudyNotFoundError and RunNotFoundError * add ability to dump run metadata to merlin_info directory * add new Celery signature to mark a run as complete * create merlin database commands * integrate database changes with monitor command * run fix-style and fix lint issues * update CHANGELOG and run fix-style with python 12 * last run of style fixes before push * run fix style with python 3.13 * create foundation for monitor auto-restart test * move get_backend_password import inside RedisBackend class * fix lint issue and comment out monitor test for now * fix redis connection so it works for ssl and non-ssl connections * add test for workflow auto-restart functionality * run linter * clean up print for DatabaseStudy and add -k option to delete all-studies * add docs for new monitor and database command * run fix-style * update built-in celery settings and retry exceptions to better handle timeouts * add new Monitor classes to handle the monitor command * update CHANGELOG and fix monitor test * fix lint issues * add worker entries and interactions with the database * add ABC class for database entities * add multi-run and multi-study queries, queries by study id, and queries by run workspace * add ability to delete multiple runs, studies, and workers. Also add ability to delete study by id and run by workspace * update docstrings for db and backend files * run fix-style * add repr method to HasRegex to make parsing GitHub CI easier * add ability to delete everything from db and retrieve connection string from MerlinDatabase * rename DatabaseRun and DatabaseStudy to more appropriate RunEntity and StudyEntity * split WorkerModel into LogicalWorkerModel and PhysicalWorkerModel * split redis backend class up into store classes for different entities. Removes db worker interaction temporarily * add logical worker functionality to database * add CLI functionality for physical workers and clean up MerlinDatabase * add physical worker db entry creation on celery worker startup * comment out code that's breaking worker launch * fix bug with get_all_runs method not existing anymore * run fix-style * fix bug with load/delete of physical workers and add functionality to set worker status to stopped * fix issue where ids were passed to wait_for_workers instead of names * move stop-worker celery functionality to a signal handler * add Logical Worker with ID 7620359f-6deb-0056-8956-39495dba8e59 ------------------------------------------------ Name: worker2 Runs: ['2e12e7cd-e7b9-468b-92c7-1ba1dffba852'] Queues: ['[merlin]_sim_queue', '[merlin]_seq_queue'] Physical Workers: ['dc863387-6b54-4a93-8a27-507bf0fd789e', '41a28c9e-f58d-49da-87bf-cea1569398b9'] Additional Data: {} Physical Worker with ID 41a28c9e-f58d-49da-87bf-cea1569398b9 ------------------------------------------------ Name: celery@worker2.%rzadams1011 Logical Worker ID: 7620359f-6deb-0056-8956-39495dba8e59 Launch Command: None Args: {} Process ID: None Status: WorkerStatus.RUNNING Last Heartbeat: 2025-04-14 12:40:05.786023 Last Spinup: 2025-04-14 12:40:05.786028 Host: rzadams1011 Restart Count: 0.0 Additional Data: {} Physical Worker with ID dc863387-6b54-4a93-8a27-507bf0fd789e ------------------------------------------------ Name: celery@worker2.%rzadams1010 Logical Worker ID: 7620359f-6deb-0056-8956-39495dba8e59 Launch Command: None Args: {} Process ID: None Status: WorkerStatus.RUNNING Last Heartbeat: 2025-04-14 12:40:05.027347 Last Spinup: 2025-04-14 12:40:05.027353 Host: rzadams1010 Restart Count: 0.0 Additional Data: {} Run with ID 2e12e7cd-e7b9-468b-92c7-1ba1dffba852 ------------------------------------------------ Workspace: /usr/WS1/gunny/scalability_testing/studies/long_running_wf_20250414-124613 Study ID: 64def1c2-8ee8-40a4-9f74-36d5b4872b9e Queues: ['[merlin]_seq_queue', '[merlin]_sim_queue'] Workers: ['7620359f-6deb-0056-8956-39495dba8e59'] Parent: None Child: None Run Complete: False Additional Data: {} Study with ID 64def1c2-8ee8-40a4-9f74-36d5b4872b9e ------------------------------------------------ Name: long_running_wf Runs: ['2e12e7cd-e7b9-468b-92c7-1ba1dffba852'] Additional Data: {} command * factor out common code from entity classes into mixin classes * run fix-style * remove process kill from celery shutdown * write tests for high-level modules of backends folder * add tests for the RedisBackend class * write unit tests for redis_logical_worker_store.py * add unit tests for RedisPhysicalWorkerStore * add tests for RedisRunStore * move db entities to their own folder * update version to 1.13.0a1 * add init.py file to backends/redis folder * change the monitor test so that it has more samples and purges tasks rather than stopping workers * run fix-style * fix error from merging develop * fix import issue with enums * condense store logic into base class and mixin class * add tests for new combined logic, remove tests for separated logic * run fix-style * fix imports and string representations of entities * update logical worker queue entry to be a set instead of list * make worker queues a set and fix str representations * update docs for database command * run fix style * change some info logs to debug * fix the majority of the docs warnings * fix docs warnings and run fix style * finish the database command docs * fix log messages and bug with deleting physical worker entry * add log files per Charles' suggestion * refactor db entities to share additional common logic * fix bug when deleting runs by workspace after workspace has been removed * add unit tests and some integration tests for data_models and db_commands * add unit tests for entity and mixin classes * refactor MerlinDatabase to use newly added entity managers * move mixins folder into entities folder * run fix-style * move mixin tests and remove todos from physical worker test file * add tests for the merlin_db module * add tests for entity managers * run fix-style * update CHANGELOG * fix unit tests for python 3.8 * fix typo * log warning if workspace doesn't exist rather than error * remove a function that was added by accident in merge from develop * fix import issue * reorganize redis_utils as they're no longer needed * move serialize and deserialize to common utils * add base store and move save, retrieve, delete methods up to ResultsBackend * initial work on adding sqlite support for local runs * finish implementing sqlite stores * update docstrings for the backend package * move existing tests of redis/results backend after refactor * add unit tests for sqlite modules and move around fixtures * add tests for StoreBase * fix unit tests for configfile * remove duplicate fixture * add some docs about sqlite and move database docs out of monitoring section * first attempt at fixing port issue * fix lint issues * second attempt at fixing the tests and style issues * implement Charles comments * update CHANGELOG * Update copyright (#510) * add COPYRIGHT file, update LICENSE, and change copyright header in all files * update Makefile commands related to copyright/copyright headers * Update CHANGELOG * add a check for the copyright header in 'make check-style' * update CHANGELOG * release/1.13.0b1 (#511) * Potential fix for code scanning alert no. 23: Workflow does not contain permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * codeql-fixes (#513) * fix logging security vulnerabilities * update CHANGELOG * feature/disable-auto-restart (#514) * add no-restart option * run fix-style * update docstrings * add documentation about the no-restart option * update CHANGELOG * add Luc's suggestions * refactor/main-module (#515) * move each command's entry point logic to their own files * add missing docstrings * add copyright to the top of all new files * fix lint issues * rename backend utils test file * begin work on adding unit tests to the new cli folder * add common create_parser fixture * add tests for example and info * fix docstrings and move conftest file up a directory * add unit tests for 3 more commands * add tests for queue-info, restart, and run-workers * add tests for the remaining commands * remove parser fixtures when only one test uses it * run fix-style * add the no_restart functionality that got wiped from the merge * update CHANGELOG * feature/monitor-tests (#516) * split while loop inside monitor into smaller pieces for testing * add tests for task_server_monitor and celery_monitor * fix typo in list comprehension that broke a test * add tests for the Monitor class and MonitorFactory * update CHANGELOG * run fix-style and add missing docstrings/copyright headers * rename test_monitor to test_monitor_entry_point * release/1.13.0b2 (#517) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: Ryan Lee <lee1098@llnl.gov> Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> Co-authored-by: Wout De Nolf <wout.de_nolf@esrf.eu> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

lucpeterson added 2 commits May 16, 2024 10:12

New github action test to make sure target has been merged into source

2e58a8d

Update changelog

7d5718d

lucpeterson requested a review from bgunnar5 May 16, 2024 17:19

bgunnar5 approved these changes May 16, 2024

View reviewed changes

lucpeterson merged commit 12f2398 into LLNL:develop May 16, 2024

lucpeterson deleted the testing/merged_target branch May 16, 2024 17:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Target is in source's history #478

Target is in source's history #478

Uh oh!

Uh oh!

Uh oh!

Target is in source's history #478

Target is in source's history #478

Uh oh!

Conversation

Uh oh!

Uh oh!