![dask-logo](../_static/services/dask_logo.png)

# How to use Dask Gateway

## Install

```
pip install eodc
```

## Connect and create a cluster

```python
from eodc.dask import eodcDaskGateway

gateway = eodcDaskGateway(username="your-eodc-account@org.com")

cluster = gateway.new_cluster()
client = cluster.get_client()
cluster.scale(3)
```

1. `eodcDaskGateway(username=...)` — authenticates against the eodc Dask Gateway. Use the email address associated with your eodc account.
2. `gateway.new_cluster()` — provisions a new Dask cluster on the eodc infrastructure with default resources (see [cluster configuration](../services/dask)).
3. `cluster.get_client()` — creates a Dask `Client` connected to the cluster. All subsequent Dask computations are submitted through this client.
4. `cluster.scale(3)` — sets the number of workers. Start small and scale up as needed.

## Customise cluster resources

To request specific CPU, memory, or a custom Docker image, configure options before creating the cluster:

```python
cluster_options = gateway.cluster_options()
cluster_options.worker_cores = 4
cluster_options.worker_memory = "8GB"

cluster = gateway.new_cluster(cluster_options)
client = cluster.get_client()
cluster.scale(4)
```

Available options and their limits are listed in the [Dask Gateway service page](../services/dask).

## Adaptive scaling

Instead of a fixed number of workers, let Dask scale automatically based on workload:

```python
cluster.adapt(minimum=2, maximum=8)
```

The cluster adds workers when tasks are queued and releases them when idle. This avoids over-provisioning during lighter workloads.

## Monitor with the Dask dashboard

Each cluster exposes a real-time dashboard showing task progress, memory usage, and CPU:

```python
print(cluster.dashboard_link)

# Embed directly in a notebook
from IPython.display import IFrame
IFrame(cluster.dashboard_link, width=1300, height=900)
```

Note: the dashboard URL uses HTTP, not HTTPS.

## Connect to an existing cluster

If your session restarts or you want to reconnect to a cluster you already started:

```python
clusters = gateway.list_clusters()
cluster = gateway.connect(cluster_name=clusters[0].name)
client = cluster.get_client()
```

## Shut down

Always shut down clusters when you are done to free up resources:

```python
# Shut down a single cluster
cluster.close(shutdown=True)

# Shut down all your running clusters
for c in gateway.list_clusters():
    gateway.connect(cluster_name=c.name).close(shutdown=True)
```

Clusters also shut down automatically after 6 hours of inactivity.

## Worked example: processing EO data with STAC and Dask

This example searches for Sentinel-1 data via STAC, loads it with `odc-stac`, and runs a distributed computation on a Dask cluster.

```python
import pystac_client
import odc.stac
from eodc.dask import eodcDaskGateway

# Start a Dask cluster
gateway = eodcDaskGateway(username="your-eodc-account@org.com")
cluster = gateway.new_cluster()
client = cluster.get_client()
cluster.scale(4)

# Search for data via STAC
eodc_stac = pystac_client.Client.open("https://stac.eodc.eu/api/v1")
items = eodc_stac.search(
    collections=["SENTINEL1_GRD"],
    bbox=[13.0, 47.0, 15.0, 48.5],
    datetime="2023-06-01/2023-06-30",
).item_collection()

# Load into an xarray dataset (lazy — nothing downloaded yet)
ds = odc.stac.load(items, chunks={"x": 2048, "y": 2048})

# Run a distributed computation
result = ds["vv"].mean(dim="time").compute()

print(result)

# Clean up
cluster.close(shutdown=True)
gateway.close()
```

## Tips

- Run Dask from within the [eodc JupyterHub](jupyterhub) for the simplest setup — no extra authentication is needed when connecting from inside the hub.
- Keep chunks large enough to avoid scheduler overhead but small enough to fit in worker memory. For raster data, `2048×2048` pixels per chunk is a reasonable starting point.
- Use the dashboard to spot straggler tasks and memory pressure before they cause failures.
- See the [GFM: Flood Extent with Dask](../notebooks/gfm-flood-extent) notebook for a full worked example with flood mapping data.