dask-logo

How to use Dask Gateway¶

Install¶

pip install eodc

Connect and create a cluster¶

from eodc.dask import eodcDaskGateway

gateway = eodcDaskGateway(username="your-eodc-account@org.com")

cluster = gateway.new_cluster()
client = cluster.get_client()
cluster.scale(3)

eodcDaskGateway(username=...) — authenticates against the eodc Dask Gateway. Use the email address associated with your eodc account.
gateway.new_cluster() — provisions a new Dask cluster on the eodc infrastructure with default resources (see cluster configuration).
cluster.get_client() — creates a Dask Client connected to the cluster. All subsequent Dask computations are submitted through this client.
cluster.scale(3) — sets the number of workers. Start small and scale up as needed.

Customise cluster resources¶

To request specific CPU, memory, or a custom Docker image, configure options before creating the cluster:

cluster_options = gateway.cluster_options()
cluster_options.worker_cores = 4
cluster_options.worker_memory = "8GB"

cluster = gateway.new_cluster(cluster_options)
client = cluster.get_client()
cluster.scale(4)

Available options and their limits are listed in the Dask Gateway service page.

Adaptive scaling¶

Instead of a fixed number of workers, let Dask scale automatically based on workload:

cluster.adapt(minimum=2, maximum=8)

The cluster adds workers when tasks are queued and releases them when idle. This avoids over-provisioning during lighter workloads.

Monitor with the Dask dashboard¶

Each cluster exposes a real-time dashboard showing task progress, memory usage, and CPU:

print(cluster.dashboard_link)

# Embed directly in a notebook
from IPython.display import IFrame
IFrame(cluster.dashboard_link, width=1300, height=900)

Note: the dashboard URL uses HTTP, not HTTPS.

Connect to an existing cluster¶

If your session restarts or you want to reconnect to a cluster you already started:

clusters = gateway.list_clusters()
cluster = gateway.connect(cluster_name=clusters[0].name)
client = cluster.get_client()

Shut down¶

Always shut down clusters when you are done to free up resources:

# Shut down a single cluster
cluster.close(shutdown=True)

# Shut down all your running clusters
for c in gateway.list_clusters():
    gateway.connect(cluster_name=c.name).close(shutdown=True)

Clusters also shut down automatically after 6 hours of inactivity.

Worked example: processing EO data with STAC and Dask¶

This example searches for Sentinel-1 data via STAC, loads it with odc-stac, and runs a distributed computation on a Dask cluster.

import pystac_client
import odc.stac
from eodc.dask import eodcDaskGateway

# Start a Dask cluster
gateway = eodcDaskGateway(username="your-eodc-account@org.com")
cluster = gateway.new_cluster()
client = cluster.get_client()
cluster.scale(4)

# Search for data via STAC
eodc_stac = pystac_client.Client.open("https://stac.eodc.eu/api/v1")
items = eodc_stac.search(
    collections=["SENTINEL1_GRD"],
    bbox=[13.0, 47.0, 15.0, 48.5],
    datetime="2023-06-01/2023-06-30",
).item_collection()

# Load into an xarray dataset (lazy — nothing downloaded yet)
ds = odc.stac.load(items, chunks={"x": 2048, "y": 2048})

# Run a distributed computation
result = ds["vv"].mean(dim="time").compute()

print(result)

# Clean up
cluster.close(shutdown=True)
gateway.close()

Tips¶

Run Dask from within the eodc JupyterHub for the simplest setup — no extra authentication is needed when connecting from inside the hub.
Keep chunks large enough to avoid scheduler overhead but small enough to fit in worker memory. For raster data, 2048×2048 pixels per chunk is a reasonable starting point.
Use the dashboard to spot straggler tasks and memory pressure before they cause failures.
See the GFM: Flood Extent with Dask notebook for a full worked example with flood mapping data.