Today we are excited to announce the release of stac-fastapi-indexed!
stac-fastapi-indexed is an implementation of the STAC API specification. Like several similar projects, it draws on the indispensable stac-fastapi package. This project implements a new approach to data management to reduce hosting overheads.
STAC and STAC API
The STAC and STAC API specifications have proven both useful and successful since their release. STAC continues to spread throughout the geospatial domain, and Sparkgeo’s client engagements regularly involve STAC and STAC API in some capacity.
The SpatioTemporal Asset Catalog (STAC) specification defines a consistent structure for geospatial asset metadata. It is commonly used to describe earth observation products, such as pixel data generated by satellite platforms, but its application is not limited to this format. In addition to helping better organise the world of geospatial data, STAC enables a growing ecosystem of interoperable tools through a common metadata format.
The following image shows a simplified STAC catalog in a Catalog / Collection / Item / Asset structure that is common but not mandatory.
Figure 1: Simplified STAC catalog with a common structure.
The STAC API specification describes a collection of core HTTP REST API endpoints to access and search STAC metadata. A number of extensions can optionally add standardised capabilities to a STAC API.
STAC API Implementations
A number of good open source solutions exist to support maintaining, storing, and serving STAC metadata.
One of the more common solutions is stac-fastapi-pgstac. This project combines the stac-fastapi endpoint layer “frontend” with a PostgreSQL / PostGIS-based data storage “backend”. The result is a STAC API implementation that manages a single STAC catalog in PostgreSQL.
stac-fastapi-pgstac has proven very capable since its release, and it supports a number of high profile STAC APIs with large and active user groups. For scenarios where the underlying PostgreSQL storage technology cannot meet a given use-case, comparable projects exist such as stac-fastapi-elasticsearch-opensearch.
These projects can offer great solutions for most use-cases that warrant a STAC API. However, they also carry maintenance overheads.
Overheads
Maintaining PostgreSQL or Elasticsearch instances carries costs. A wide-range of hosting strategies exist, and many users opt for the simplicity of managed services such as AWS RDS for PostgreSQL. Such services incur recurring costs as the instance(s) are typically persistent, available, and occupy reserved compute resources irrespective of client demand. “Serverless” solutions such as AWS Aurora Serverless may have the potential to reduce costs, but will not be appropriate in all scenarios.
Hosting is also required for API instances. API containers are often deployed as persistent resources, for example using AWS ECS or EC2 hosts. It is possible to deploy the API with transient compute resources, such as AWS API Gateway with AWS Lambda, but this can add complexity and performance overheads when managing database connections and connection pools.
Other hosting solutions – such as the Kubernetes model employed by the excellent EO API – might reduce cloud provider costs in exchange for larger maintenance overheads.
stac-fastapi-indexed Goals
The primary goal in developing stac-fastapi-indexed was to reduce hosting overheads. The API should run with transient compute resources and require no persistent data management platform beyond a basic object store, such as AWS S3. The API is intended for use alongside STAC generation processes that generate JSON documents and do not ingest those JSON documents in a persistent and searchable data management platform like a DBMS.
A secondary goal was identified during development. Due to stac-fastapi-indexed’s design it became possible to take an existing static STAC catalog and access its collections and items via the STAC API interface. In principle any static STAC catalog, whether public or accessible via credentials, can be made searchable in this way. This potentially opens up a wealth of existing STAC data to new use-cases.
Along the way we identified ways to address a number of limitations we sometimes hit in other STAC API implementations. For example:
- Understanding sub-catalogs within a root catalog.
- Recognising when a data change might impact the validity of paging through multi-page results.
- Addressing some common STAC validation issues with older or poorer-quality input data.
Architecture
Initial development efforts explored the use of SpatiaLite as a STAC storage format. This approach encountered several challenges and projected development effort was significant. SpatiaLite was not pursued as a result.
At the time we were considering our requirements, Sparkgeo had recently completed some experimental work with DuckDB and Parquet. We wondered if this efficient and capable pairing could play a role in stac-fastapi-indexed. A little research surfaced some interesting ideas combining DuckDB and STAC, but nothing that quite matched our goals.
Ultimately we settled on a novel design. An indexer component traverses a STAC catalog and generates an index. The index is a collection of Parquet files recording important properties of STAC resources and references to the original resources. The indexer is expected to run once, on an interval, or on-demand in response to mutation events – whatever best suits the use-case. A separate component acts as a stac-fastapi “backend” and manages STAC API data retrieval. DuckDB executes SQL queries using the Parquet index files to identify which STAC resources satisfy a caller’s request and the contents of those STAC resources are fetched from source.
Figure 2: Overview of the relationship between indexer and API
This approach will not suit every use-case. A number of trade-offs are considered acceptable in pursuit of reduced hosting overheads. We put together some documentation to help determine if this might be the right solution for you.
Get Involved!
stac-fastapi-indexed is now open source. This has been the plan for some time, and we wanted to get this project to a good place before pushing it out into the world. We see this as a work-in-progress and we expect to continue working through outstanding issues as capacity permits.
One of the more immediately rewarding uses of this project is indexing a public static STAC catalog. While stac-fastapi-indexed is currently capable of addressing some common data quality issues in STAC resources it cannot (yet?) address all problems. Your mileage may vary depending on the targeted catalogs, so we provide one known-good example below.
scripts/run-with-remote-source.sh https://esa.pages.eox.at/cubes-and-clouds-catalog/MOOC_Cubes_and_clouds/catalog.jsonFigure 3: Demo showing a public static STAC catalog being indexed and searched
Test it out, fork the repo, and please tell us where we’ve gone wrong. We look forward to any and all contributions on this latest addition to the STAC ecosystem.