Argo Workflows¶
Argo Workflows is a Kubernetes-native workflow orchestration tool that allows you to automate and scale complex data processing tasks. It provides a flexible way to define, schedule, and monitor multi-step workflows, making it easier to manage large-scale data pipelines and computational tasks across distributed resources. Leveraging Kubernetes’ scalability, Argo Workflows ensures efficient resource utilization and resilience, with advanced features such as parallel processing, retries, and conditional execution. This allows you to build, deploy, and maintain reproducible workflows, streamlining tasks like data ingestion, transformation, and analysis with minimal intervention and maximum scalability.
Endpoint: https://services.eodc.eu/workflows/
→ See How to use Argo Workflows for a step-by-step guide with code examples.
Who should use it?¶
Containerisation — if you already have a container, or want to run an existing container at scale across hundreds or thousands of satellite images.
Non-Python workflows — if your codebase is in another language and porting to Python is not practical, building a runtime and tailoring a workflow with Argo is a straightforward alternative to Dask.
Scheduled processing — if you need to run processing on a regular schedule (daily, weekly, etc.), Argo’s CronWorkflow support lets you automate this. Results can be uploaded to the eodc Object Store and registered in a STAC collection automatically.
Access¶
Access to Argo Workflows requires a request to support@eodc.eu. You will receive a token needed for authentication. Mention in your request if you intend to use the eodc-sdk.
Using the dashboard¶
Once you have access, workflows can be submitted and monitored through the Argo Workflows dashboard at services.eodc.eu/workflows.
Log in using the single sign-on option.
Navigate to the workflows section (top icon in the sidebar).
Click Submit New Workflow — a workflow can be loaded from a file or submitted directly as YAML.
After submission, monitor the workflow for success or failure.
