FAIR MAST is a data management system designed for fusion research, enabling efficient storage, retrieval, and management of experimental data.
-
If you are using Mac for development, use podman instead of docker. Follow the installation guide to set it up, then follow the below set up. Also install
podman-mac-helper, which provides a compatibility layer that allows you to use most Docker commands with Podman on macOS.If using Linux or Windows, you need to make sure you have docker and
docker-composeinstalled on your system.
- Clone the repository:
git clone git@github.com:ukaea/fair-mast.git
cd fair-mast-
Start the development environment:
podman compose \ --env-file dev/docker/.env.dev \ -f dev/docker/docker-compose.yml \ up \ --build
docker-compose \ --env-file dev/docker/.env.dev \ -f dev/docker/docker-compose.yml \ up \ --build
The following services will be started:
- FastAPI REST & GraphQL Server - will start running at
http://localhost:8081.- The REST API documentation is at
http://localhost:8081/redoc. - The GraphQL API documentation is at
http://localhost:8081/graphql.
- The REST API documentation is at
- Postgres Database Server - will start running at
http://localhost:5432 - Postgres Admin Server - will start running at
http://localhost:5050
-
Shutting Down:
podman compose -f dev/docker/docker-compose.yml down podman volume rm --all
docker-compose -f dev/docker/docker-compose.yml down
Retrieve and ingest the metadata files using s5cmd:
s5cmd --no-sign-request --endpoint-url https://s3.echo.stfc.ac.uk cp "s3://mast/dev/mock_data*" ./tests
Create the database and ingest data using the following command:
podman exec -it mast-api python -m src.api.create /test_data/index/ docker exec -it mast-api python -m src.api.create /test_data/index/ Verify everything is setup correctly by running the unit tests.
Follow the below instructions to set up the environment.
uv run pytestWhen deploying for the first time (I.e. with no ssl certificates yet generated) you will need to follow some additional steps:
Rename the file "nginx.conf" to "nginx-final.conf" (or anything else so long as you remember what it is) Then rename: "nginx-initial.conf" to "nginx.conf"
(This is so that nginx can run without ssl certiifcates while they are being generated)
Proceed with the full deployment proceedure (As detailed below)
Now switch the nginx config files back to their original names and run the command:
docker exec reverse-proxy nginx -s reload(This reloads nginx to use the full configuration now inculding https with the generated certificates)
When deploying the full production stack (Either for testing or production) please check the .env.dev file in ./dev/docker and ensure the enviromental variables NGINX_CONFIG_PATH and CERTBOT_COMMAND are set to their correct varients for your use case (More information on this in the ./dev/docker README).
To run the production container which starts the postgres database, fastapi, nginx reverse proxy and certbot ssl certificate generator, run the following command:
docker-compose --env-file dev/docker/.env.dev -f dev/docker/docker-compose.yml -f dev/docker/docker-compose-prod.yml up --build --force-recreate --remove-orphans -dYou'll need to download and ingest the production data like so:
mkdir -p data/mast/meta
rsync -vaP <CSD3-USERNAME>@login.hpc.cam.ac.uk:/rds/project/rds-sPGbyCAPsJI/archive/metadata data/docker exec -it mast-api python -m src.api.create /test_data/indexTo shut down the production deployment, run the following command:
docker-compose --env-file dev/docker/.env.dev -f dev/docker/docker-compose.yml -f dev/docker/docker-compose-prod.yml downTo also destory the volumes (including the metadatabase) you may add the volumes parameter:
docker-compose --env-file dev/docker/.env.dev -f dev/docker/docker-compose.yml -f dev/docker/docker-compose-prod.yml down --volumesSee the documentation guide for more details.
If you would like to reference this work, please cite the followig publication.
Samuel Jackson, Saiful Khan, Nathan Cummings, James Hodson, Shaun de Witt, Stanislas Pamela, Rob Akers, Jeyan Thiyagalingam, FAIR-MAST: A fusion device data management system, SoftwareX, Volume 27, 2024,101869,ISSN 2352-7110, https://doi.org/10.1016/j.softx.2024.101869.
In BibTex format:
@article{jackson_fair-mast_2024,
title = {{FAIR}-{MAST}: {A} fusion device data management system},
volume = {27},
issn = {23527110},
shorttitle = {{FAIR}-{MAST}},
url = {https://linkinghub.elsevier.com/retrieve/pii/S2352711024002395},
doi = {10.1016/j.softx.2024.101869},
language = {en},
urldate = {2025-01-17},
journal = {SoftwareX},
author = {Jackson, Samuel and Khan, Saiful and Cummings, Nathan and Hodson, James and De Witt, Shaun and Pamela, Stanislas and Akers, Rob and Thiyagalingam, Jeyan},
month = sep,
year = {2024},
pages = {101869},
}