Hands-on session — EPT Training 1¶
From metadata records and NetCDF files to a validated STAC catalog on EDITO — against real EMODnet data.
EPT Tutorials Overview · Install Guide
What you'll build¶
Your data becomes a STAC catalog — plain JSON files on S3:
Catalog ← root, ties it together
└── Collection ← shared extents, license
├── Item ← one dataset, one place, one time
│ ├── Asset Zarr ← the actual data
│ └── Asset ISO19139 ← the metadata record
└── Item
Items group into Collections (shared extent and
license); Collections tie together under a Catalog, the root
everything hangs off. It's all JSON with links, hosted as static
files on S3. Valid STAC can be published to the EDITO STAC API and
it is viewable in the STAC viewer.
EPT does the mapping: one call per Item, one per Collection, one for
the Catalog.
Alongside, the data itself is converted to ARCO — analysis-ready, cloud-optimized — formats, so clients read chunks over HTTP instead of downloading whole files:
- Zarr (from NetCDF) — the array is split into chunks, each chunk
its own object on S3 with small JSON metadata describing shape, dtype
and compression. Reading one time slice fetches only that slice's
chunks, not the whole NetCDF;
xarray.open_zarrreassembles it into the familiar labelled dataset, CF metadata intact. For gridded/raster data and time series. - GeoParquet (from GDB/shapefile) — Parquet with a standard geometry column: a columnar table grouped into compressed row groups. A query on two columns reads only those columns; a spatial or attribute filter skips whole row groups without reading them. For vector tables — points, lines, polygons.
EPT does all of this through one API — UI, curl, or python. Full background: the tutorials intro.
Setup¶
From your personal space:
- Run EPT — today on EDITO, zero install: the edito-publishing-toolkit-service playground · 🎬 video. The other option — running the backend yourself (docker or clone) — is the same API at the same endpoints; try it later, see the Install Guide.
- Your service URL — the address bar while your service is open,
or datalab → My services → click your service. It looks like
https://user-<your-username>-…-0.lab.dive.edito.eu/api/— that whole link,/api/included, is yourEPT_URL. - Side services for checking your work (optional, launch now, use later): STAC browser — paste a catalog/Collection/Item URL and browse the tree; Parquet explorer — inspect the GeoParquet the conversions produce.
- Jupyter notebook (launch now, use from step 3): Jupyter notebook service — create a new Python notebook. Running EPT locally? A local notebook/script instead.
Today you work in a shared project bucket (project-ept-training1).
Pick a unique handle — your EDITO username is easiest; avoid
project1/test. Every id and folder derives from it
(<handle>-output/…, <handle>-din, <handle>-catalog), so parallel
sessions in the shared bucket never collide.
Your deliverable: a validated catalog referencing cloud-optimized data in your output folder.
How we'll work¶
The walkthroughs we'll be going through:
| Step | Page | Done when |
|---|---|---|
| 1 — Transform data | walkthrough 1 | Zarr + GeoParquet under <handle>-output/data/, checked in the viewers |
| 2 — A data item | walkthrough 2 | <handle>-din.json in <handle>-output/items/din/ |
| 3 — Items from metadata | walkthrough 3 | <handle>-mangrove, <handle>-heritage-shipwrecks items |
| 4 — ARCO items | walkthrough 4 | every Item carries cloud-optimized assets |
| 5 — Collections | walkthrough 5 | two validated Collection folders |
| 6 — Catalog | walkthrough 6 | schema-valid <handle>-output/stac-final/ — paste it into the STAC browser |
In the frontend. Open your service URL and sign in — every EPT capability is a tool form, Run Tool submits it. There is a Logs tab so you can follow what calls are being made and what it's doing.
Also in the jupyter notebook you launched in Setup. First cell, the session values:
import os
os.environ["EPT_URL"] = "<your service url>" # the /api/ link of your service — see Setup above
os.environ["EPT_BUCKET"] = "project-ept-training1" # today's shared bucket
os.environ["EPT_PROJECTID"] = "<your-handle>" # your unique handle
Second cell, the session setup block from
Session setup (the python one): it
authenticates, mints your S3 credentials and defines the post/get
helpers — fill in your EDITO username and password when it prompts
(prefer the env-file way? that page shows it too).
From there, each walkthrough's python block is a cell — paste and run in order. While they run, datalab → My services → your service → Logs tab shows every call, whichever client made it.
Convert your own data. The calls only care about a source_uri.
Have your own NetCDF or ISO19139 record? Use its URL — public
https://…, or a file you've put in the bucket (s3://<bucket>/…) —
in place of the EMODnet one in the step-2 or step-3 cell, with your own
item_id. Add it to a Collection's item_links in the step-5 cell and
re-run steps 5–6 — it lands in your final catalog like any other Item.
One pass instead? The same six steps, one runnable script: Full walkthrough — python.
Publishing — by the session leader¶
Note: publishing replaces the target project catalog entirely — a STAC
published into projects/ept-training1 is the ept-training1 catalog.
So during this training, don't publish into the shared project catalog
yourself. At the end of the session the session leader will assemble a
single collaborative STAC from everyone's valid catalogs and publish
that to projects/ept-training1.
If you later publish data on EDITO under a project you own or join, you use the same tools yourself, following Walkthrough 7 — Publish.