{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# GFM data discovery and download" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "EODC catalogs several datasets using the [STAC](http://stacspec.org/)\n", "(SpatioTemporal Asset Catalog) specification. By providing a [STAC API](https://stac.eodc.eu/api/v1) endpoint,\n", "we enable users to search our datasets by space, time and more filter criterias\n", "depending on the individual dataset.\n", "\n", "In this notebook, we demonstrate how to query the [Global Flood Monitoring](https://extwiki.eodc.eu/en/GFM) (GFM) STAC collection using the Python library\n", "[pystac_client](https://pystac-client.readthedocs.io/en/latest/index.html) and\n", "download the data using built-in Python libraries as well as utilizing the command line tool [stac-asset](https://github.com/stac-utils/stac-asset).\n", "\n", "You can install the pystac_client via pip:\n", "\n", " pip install pystac_client\n", "\n", "In the STAC items the respective assets (=file) are linked. These links are\n", "used to download the file to a specified folder on your machine." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from datetime import datetime\n", "from pystac_client import Client\n", "\n", "# EODC STAC API URL\n", "api_url = \"https://stac.eodc.eu/api/v1\"\n", "\n", "eodc_catalog = Client.open(api_url)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Gridding of the GFM data sets\n", "\n", "The GFM service processes all observations from the Sentinel-1A/B (soon including\n", "Sentinel-1C) satellites that are acquired over land in Interferometric Wide-swath mode and\n", "Ground Range Detected at High resolution (Sentinel-1 IW GRDH).\n", "\n", "The GFM service uses the [Equi7Grid](https://www.sciencedirect.com/science/article/pii/S0098300414001629) that employs\n", "the equidistant azimuthal projection and divides the\n", "Earth surface into seven continental zones. The Equi7Grid with a 20m pixel\n", "spacing and a 300km gridding (T3 level) serves as efficient working grid\n", "representation for all steps in the data processing workflow. Consequently, all\n", "input datasets, including auxiliary datasets from external sources, must be\n", "re-projected to the Equi7Grid beforehand. \n", "\n", "The spatial extent of a Sentinel-1 scene is too large to be represented on only\n", "one Equi7Tile (=file). Therefore, a search query usually will result with\n", "multiple items, even for the same timestamp of a single Sentinel-1 observation." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Searching\n", "\n", "We can use the STAC API to find items that match specific criteria. This may\n", "include the date and time the item covers, its spatial extent, or any other\n", "property saved in the item's metadata.\n", "\n", "If a specific Sentinel-1 scene is of interest, it is also possible to directly\n", "use the sensing date in the search query." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Search with AOI and time range\n", "\n", "In this example we are searching for GFM data which cover our area of interest\n", "over South Pakistan in September 2022.\n", "\n", "The area of interest can be specified as `bbox` using the Python library\n", "`shapely` or, alternateively, as GeoJSON object.\n", "\n", "The time range can be specified as tuples of datetime object or simply using\n", "strings. " ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "On EODC we found 21 items for the given search query\n" ] } ], "source": [ "from shapely.geometry import box\n", "\n", "# STAC collection ID\n", "collection_id = \"GFM\"\n", "\n", "# Time range\n", "time_range = (datetime(2022, 9, 15, 0, 0, 0), datetime(2022, 9, 16, 23, 59, 59))\n", "time_range = '2022-09-15/2022-09-16'\n", "\n", "# Area of interest (South Pakistan)\n", "aoi = box(63.0, 24.0, 73, 27.0)\n", "\n", "aoi = {\n", " \"type\" : \"Polygon\",\n", " \"coordinates\": [\n", " [\n", " [73.0, 24.0],\n", " [73.0, 27.0],\n", " [63.0, 27.0],\n", " [63.0, 24.0],\n", " [73.0, 24.0],\n", " ]\n", " ],\n", "}\n", "\n", "search = eodc_catalog.search(\n", " max_items=1000,\n", " collections=collection_id,\n", " intersects=aoi,\n", " datetime=time_range\n", ")\n", "\n", "items_eodc = search.item_collection()\n", "print(f\"On EODC we found {len(items_eodc)} items for the given search query\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Search with Sentinel-1 scene identifier\n", "\n", "In this example we are using a single Sentinel-1 scene identifier to retrieve\n", "the respective STAC items. Either use the following simple method to derive the\n", "sensing date from the Sentinel-1 scene identifier or directly use the exact\n", "datetime in the query." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# Method to derive the sensing date from a Sentinel-1 scene identifier\n", "def get_sensing_date(scene:str) -> datetime:\n", " parts = scene.split(\"_\")\n", " return datetime.strptime(parts[4], \"%Y%m%dT%H%M%S\")" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "On EODC we found 2 items for the given search query\n" ] } ], "source": [ "# Define Sentinel-1 scene identifier and asset name to plot\n", "scene_id = \"S1A_IW_GRDH_1SDV_20220930T224602_20220930T224627_045240_056863\"\n", "\n", "api_url = \"https://stac.eodc.eu/api/v1\"\n", "eodc_catalog = Client.open(api_url)\n", "\n", "search = eodc_catalog.search(\n", " collections=[\"GFM\"],\n", " datetime=get_sensing_date(scene_id),\n", ")\n", "\n", "items_eodc = search.item_collection()\n", "print(f\"On EODC we found {len(items_eodc)} items for the given search query\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Some information about the found STAC items\n", "\n", "We can print some more information like the available assets and their\n", "description. " ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
                                                Assets in STAC Item                                                \n",
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃ Asset Key                       ┃ Description                                                                   ┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│ TileJSON with default rendering │                                                                               │\n",
       "│ Rendered preview                │ Rendered preview image of GFM Observed Flood Extent (Final output - Ensemble  │\n",
       "│                                 │ algorithm)                                                                    │\n",
       "│ advisory_flags                  │ GFM Advisory Flags                                                            │\n",
       "│ dlr_likelihood                  │ GFM Likelihood (Intermediate output - DLR algorithm)                          │\n",
       "│ exclusion_mask                  │ GFM Exclusion Mask (Final output - Ensemble algorithm)                        │\n",
       "│ tuw_likelihood                  │ GFM Uncertainties (Intermediate output - TUW algorithm)                       │\n",
       "│ list_likelihood                 │ GFM Likelihood (Intermediate output - LIST algorithm)                         │\n",
       "│ dlr_flood_extent                │ GFM Observed Flood Extent (Intermediate output - DLR algorithm)               │\n",
       "│ tuw_flood_extent                │ GFM Observed Flood Extent (Intermediate output - TUW algorithm)               │\n",
       "│ list_flood_extent               │ GFM Observed Flood Extent (Intermediate output - LIST algorithm)              │\n",
       "│ ensemble_likelihood             │ GFM Likelihood (Final output - Ensemble algorithm)                            │\n",
       "│ reference_water_mask            │ GFM Reference Water Mask                                                      │\n",
       "│ ensemble_flood_extent           │ GFM Observed Flood Extent (Final output - Ensemble algorithm)                 │\n",
       "│ ensemble_water_extent           │ GFM Observed Water Extent (Final output - Ensemble algorithm)                 │\n",
       "└─────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────┘\n",
       "
\n" ], "text/plain": [ "\u001b[3m Assets in STAC Item \u001b[0m\n", "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n", "┃\u001b[1m \u001b[0m\u001b[1mAsset Key \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mDescription \u001b[0m\u001b[1m \u001b[0m┃\n", "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n", "│\u001b[36m \u001b[0m\u001b[36mTileJSON with default rendering\u001b[0m\u001b[36m \u001b[0m│ │\n", "│\u001b[36m \u001b[0m\u001b[36mRendered preview \u001b[0m\u001b[36m \u001b[0m│ Rendered preview image of GFM Observed Flood Extent (Final output - Ensemble │\n", "│\u001b[36m \u001b[0m│ algorithm) │\n", "│\u001b[36m \u001b[0m\u001b[36madvisory_flags \u001b[0m\u001b[36m \u001b[0m│ GFM Advisory Flags │\n", "│\u001b[36m \u001b[0m\u001b[36mdlr_likelihood \u001b[0m\u001b[36m \u001b[0m│ GFM Likelihood (Intermediate output - DLR algorithm) │\n", "│\u001b[36m \u001b[0m\u001b[36mexclusion_mask \u001b[0m\u001b[36m \u001b[0m│ GFM Exclusion Mask (Final output - Ensemble algorithm) │\n", "│\u001b[36m \u001b[0m\u001b[36mtuw_likelihood \u001b[0m\u001b[36m \u001b[0m│ GFM Uncertainties (Intermediate output - TUW algorithm) │\n", "│\u001b[36m \u001b[0m\u001b[36mlist_likelihood \u001b[0m\u001b[36m \u001b[0m│ GFM Likelihood (Intermediate output - LIST algorithm) │\n", "│\u001b[36m \u001b[0m\u001b[36mdlr_flood_extent \u001b[0m\u001b[36m \u001b[0m│ GFM Observed Flood Extent (Intermediate output - DLR algorithm) │\n", "│\u001b[36m \u001b[0m\u001b[36mtuw_flood_extent \u001b[0m\u001b[36m \u001b[0m│ GFM Observed Flood Extent (Intermediate output - TUW algorithm) │\n", "│\u001b[36m \u001b[0m\u001b[36mlist_flood_extent \u001b[0m\u001b[36m \u001b[0m│ GFM Observed Flood Extent (Intermediate output - LIST algorithm) │\n", "│\u001b[36m \u001b[0m\u001b[36mensemble_likelihood \u001b[0m\u001b[36m \u001b[0m│ GFM Likelihood (Final output - Ensemble algorithm) │\n", "│\u001b[36m \u001b[0m\u001b[36mreference_water_mask \u001b[0m\u001b[36m \u001b[0m│ GFM Reference Water Mask │\n", "│\u001b[36m \u001b[0m\u001b[36mensemble_flood_extent \u001b[0m\u001b[36m \u001b[0m│ GFM Observed Flood Extent (Final output - Ensemble algorithm) │\n", "│\u001b[36m \u001b[0m\u001b[36mensemble_water_extent \u001b[0m\u001b[36m \u001b[0m│ GFM Observed Water Extent (Final output - Ensemble algorithm) │\n", "└─────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────┘\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import rich.table\n", "from rich.console import Console\n", "\n", "console = Console()\n", "\n", "first_item = items_eodc[0]\n", "\n", "table = rich.table.Table(title=\"Assets in STAC Item\")\n", "table.add_column(\"Asset Key\", style=\"cyan\", no_wrap=True)\n", "table.add_column(\"Description\")\n", "for asset_key, asset in first_item.assets.items():\n", " table.add_row(\n", " asset.title, \n", " asset.description)\n", "\n", "console.print(table)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Download data with Python\n", "\n", "You can download the desired assets by specifying their respective asset keys in\n", "a list object. Then, iterate over all found items and specified asset keys to\n", "download the data to a local directory. The HTTP link saved in the asset\n", "references the actual file, which is downloaded using the Python library `urllib`. " ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Downlading ./downloaded_data/GFM/ENSEMBLE_FLOOD_20220930T224602_VV_AS020M_E057N006T3/ENSEMBLE_FLOOD_20220930T224602_VV_AS020M_E057N006T3.tif\n", "Downlading ./downloaded_data/GFM/ENSEMBLE_FLOOD_20220930T224602_VV_AS020M_E057N006T3/TUW_FLOOD_20220930T224602_VV_AS020M_E057N006T3.tif\n", "Downlading ./downloaded_data/GFM/ENSEMBLE_FLOOD_20220930T224602_VV_AS020M_E054N006T3/ENSEMBLE_FLOOD_20220930T224602_VV_AS020M_E054N006T3.tif\n", "Downlading ./downloaded_data/GFM/ENSEMBLE_FLOOD_20220930T224602_VV_AS020M_E054N006T3/TUW_FLOOD_20220930T224602_VV_AS020M_E054N006T3.tif\n", "Download done!\n" ] } ], "source": [ "import os\n", "import urllib\n", "\n", "# specify output directory\n", "download_root_path = \"./downloaded_data/\"\n", "\n", "# specify asset names to download\n", "asset_names = [\"ensemble_flood_extent\", \"tuw_flood_extent\"]\n", "\n", "for item in items_eodc[:2]:\n", " download_path = os.path.join(download_root_path, item.collection_id, item.id)\n", " \n", " os.makedirs(download_path, exist_ok=True)\n", " \n", " for asset_name in asset_names:\n", " asset = item.assets[asset_name]\n", " if \"data\" in asset.roles:\n", " fpath = os.path.join(download_path, os.path.basename(asset.href))\n", " print(f\"Downlading {fpath}\")\n", " urllib.request.urlretrieve(asset.href, fpath)\n", "\n", "print(\"Download done!\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Download data with stac-asset CLI\n", "\n", "The command line tool [stac-asset](https://github.com/stac-utils/stac-asset)\n", "provides another way to query STAC APIs and download found assets.\n", "\n", "You can install `stac-asset` via pip:\n", "\n", " pip install 'stac-asset[cli]'\n", "\n", "`stac-assets` expects the same input parameters as described above:\n", "- STAC API URL \n", " - https://stac.eodc.eu/api/v1\n", "- Collection ID\n", " - GFM\n", "- Bounding box \n", " - 63.0, 24.0, 73.0, 27.0 (minX, minY, maxX, maxY)\n", "- Time range \n", " - 2022-09-15/2022-09-16" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### List the number of matched STAC items" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "21 items matched\n" ] } ], "source": [ "!stac-client search https://stac.eodc.eu/api/v1 -c GFM --bbox 63 24 73 27 --datetime 2022-09-15/2022-09-16 --matched" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Save matched STAC items into a JSON file (items.json)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "!stac-client search https://stac.eodc.eu/api/v1 -c GFM --bbox 63 24 73 27 --datetime 2022-09-15/2022-09-16 --save items.json" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Download a specified asset of found STAC items into a given directory" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/bin/bash: line 1: stac-asset: command not found\n" ] } ], "source": [ "!mkdir -p ./stac_asset_download\n", "!stac-asset download -i ensemble_flood_extent items.json ./stac_asset_download -q" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Pipe query results directly into the stac-asset download command" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/bin/bash: line 1: stac-asset: command not found\n", "[Errno 32] Broken pipe\n", "Traceback (most recent call last):\n", " File \"/home/katharina/.pyenv/versions/3.11.5/lib/python3.11/site-packages/pystac_client/cli.py\", line 428, in cli\n", " return search(client, **args)\n", " ^^^^^^^^^^^^^^^^^^^^^^\n", " File \"/home/katharina/.pyenv/versions/3.11.5/lib/python3.11/site-packages/pystac_client/cli.py\", line 87, in search\n", " print(json.dumps(feature_collection))\n", "BrokenPipeError: [Errno 32] Broken pipe\n" ] } ], "source": [ "!mkdir -p ./stac_asset_download\n", "!cd stac_asset_download; stac-client search https://stac.eodc.eu/api/v1 -c GFM --bbox 63 24 73 27 --datetime 2022-09-15/2022-09-16 | stac-asset download -i ensemble_flood_extent -q" ] } ], "metadata": { "kernelspec": { "display_name": "Python (env_zarr)", "language": "python", "name": "env_zarr" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 2 }