Dask Gateway¶
Dask Gateway provides users with a robust platform for scalable parallel computing and efficient resource management for data-intensive applications. Dask Gateway enables seamless integration with various cluster managers, allowing users to dynamically scale resources based on workload demands. It also offers secure, multi-tenant environments, making it ideal for both collaborative projects and enterprise-level deployments. With Dask Gateway, users can manage clusters easily, optimize computational workflows, and reduce overhead in resource allocation, all while maintaining performance at scale.
Endpoint: http://dask.services.eodc.eu
→ See How to use Dask Gateway for a step-by-step guide with code examples.
Why Dask Gateway at eodc?¶
Efficient resource management — Dask enables parallel processing of large datasets across multiple nodes, making it ideal for computationally intensive tasks such as satellite imagery analysis or climate data processing.
Flexibility and scalability — Unlike single-machine setups, Dask lets you scale computations across a cluster and adapt resources dynamically to your workload.
Native Python integration — Dask is built for Python, so it integrates seamlessly with existing workflows using NumPy, pandas, and xarray.
Who should use it?¶
Researchers and scientists processing large-scale environmental or geospatial datasets.
Data scientists performing advanced analysis or machine learning on data too large for a single machine.
Engineers and developers who need scalable computing without managing their own infrastructure.
Access and authentication¶
Access to Dask Gateway is managed via Keycloak, eodc’s identity and access management system. On authentication, Keycloak issues a JSON Web Token (JWT) that is passed automatically to the Dask Gateway server to authorize cluster operations. You do not need to manage this token manually.
To request access, contact support@eodc.eu with your name, organisation, and intended use.
Cluster configuration at eodc¶
Default resources¶
Resource |
Default |
|---|---|
Scheduler CPU |
2 cores |
Scheduler memory |
4 GB |
Worker CPU |
4 cores |
Worker memory |
8 GB |
Idle timeout |
6 hours |
Clusters shut down automatically after 6 hours of inactivity.
Configurable options¶
When creating a cluster you can adjust the following within the listed limits:
Option |
Range |
|---|---|
|
2 – 8 CPU cores |
|
2 – 16 GB RAM |
|
Custom Docker image |
Adaptive scaling¶
Clusters can scale the number of workers automatically based on workload using cluster.adapt(minimum=N, maximum=M). This avoids over-provisioning when the cluster is idle and ensures capacity is available during heavy computation.
Dashboard¶
Each cluster exposes a Dask dashboard at cluster.dashboard_link. The dashboard shows real-time task progress, memory usage, and CPU usage — useful for monitoring and debugging large computations.