This project was developed as a part of PhD study. This project allows running geospatial workflows on serverless services provided by a cloud provider. It supports sequential, parallel, and loop executions. In addition, a new workflow and task definition models were introduced to design workflows in a simple, versionable, and readable way.
At the moment, it only works on AWS and local docker deployment. Other platforms are welcomed as pull requests.
The project uses Pulumi for infrastructure as code (IaC) to deploy resources on AWS. Follow these steps to deploy the application:
-
Install Pulumi:
- For Windows (PowerShell):
winget install Pulumi
- For macOS (Homebrew):
brew install pulumi
- For Linux:
curl -fsSL https://get.pulumi.com | sh
- For Windows (PowerShell):
-
Configure AWS credentials:
- Install and configure AWS CLI
- Run
aws configureand enter your AWS credentials - Ensure you have appropriate permissions for creating resources
-
Deploy the infrastructure: Navigate to the deployment directory and run the deployment script:
cd ./deployments/GEOrchestrator.Aws.Deployment ./Deploy.ps1
This will create the following AWS resources:
- Lambda functions for API endpoints
- DynamoDB tables (on-demand)
- Fargate cluster with required network components
- API Gateway
- S3 buckets for workflow storage
The deployment script will output the API endpoint URL once completed.
Under the root folder, run the following command to start the application:
docker compose up -d --buildThe application will use default configuration values. If you need to customize the configuration, you can modify the environment variables in the docker-compose.yml file.
The API will be started on: http://localhost:8000
The raster calculations workflow is provided as an example to demonstrate how the system works. It shows how to define, build, and execute a workflow using the GEOrchestrator system.
To run the example raster calculations workflow:
- Make sure you are in the root folder of the project and run the build script:
For PowerShell:
./definitions/raster-calculations/build_and_run.ps1For bash:
chmod +x ./definitions/raster-calculations/build_and_run.sh
./definitions/raster-calculations/build_and_run.sh- Once the workflow is created, you can execute it using cURL (for bash) or PowerShell. Here's an example of executing a reprojection workflow:
For bash:
curl --location 'http://localhost:8000/processes/ReprojectDemsParallel/execution' \
--header 'Content-Type: application/json' \
--data '{
"inputs": {
"dems": "[\"https://geo-public-assets.s3.eu-west-1.amazonaws.com/asciis/sample_1.asc\",\"https://geo-public-assets.s3.eu-west-1.amazonaws.com/asciis/sample_2.asc\"]",
"source_projection": "EPSG:4326",
"target_projection": "EPSG:3759"
}
}'For PowerShell:
$body = @{
inputs = @{
dems = '["https://geo-public-assets.s3.eu-west-1.amazonaws.com/asciis/sample_1.asc","https://geo-public-assets.s3.eu-west-1.amazonaws.com/asciis/sample_2.asc"]'
source_projection = 'EPSG:4326'
target_projection = 'EPSG:3759'
}
} | ConvertTo-Json
Invoke-RestMethod -Uri 'http://localhost:8000/processes/ReprojectDemsParallel/execution' `
-Method Post `
-ContentType 'application/json' `
-Body $bodyThis example demonstrates how to execute a parallel DEM reprojection workflow. Adjust the inputs according to your specific workflow requirements.
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.
Pakdil, M.E.; Celik, R.N. Serverless Geospatial Data Processing Workflow System Design. ISPRS Int. J. Geo-Inf. 2022, 11, 20. https://doi.org/10.3390/ijgi11010020
