Paginating OGC API Features Collections in Python

TL;DR: An OGC API - Features /collections/{id}/items endpoint never returns an entire collection at once. Instead of computing offsets yourself, follow the hypermedia links array: fetch a page, process its GeoJSON features, look for the link whose rel is next, resolve it (it may be relative) against the response URL, and repeat until no next link remains. Wrapping that loop in a Python generator lets you stream millions of features without ever holding them all in memory.

The Core Challenge

OGC API - Features — the RESTful, JSON-native successor to WFS covered in OGC API Features and the REST Transition — deliberately hides its paging mechanics behind hypermedia. When you GET a collection’s items endpoint, the server returns a GeoJSON FeatureCollection with two extra top-level members that WFS never had:

• A links array, where the entry with "rel": "next" carries the URL of the following page. When that link is absent, you have reached the end.
• Optional counters: numberReturned (features in this response) and numberMatched (the total across the whole result set, which the server is permitted to omit).

The trap is that different servers page differently underneath. One implementation appends ?offset=1000&limit=1000; another issues an opaque continuation token like ?token=eyJrIjo...; a third uses a keyset cursor on the primary key. If you try to reconstruct the paging scheme by incrementing your own offset, you couple your client to one server’s internals and break against the rest. The rel="next" URL is the contract: it is opaque, self-contained, and already encodes page size, cursor position, and any active filters. Your only job is to follow it faithfully — including the awkward case where the server returns a relative href that must be resolved against the request URL.

Production-Ready Code

The module below exposes a lazy generator, iter_features, plus a small helper that reads the count members. It depends only on requests. The first request carries our limit; every page after that is fetched from the server-supplied next URL, which is already complete — so we deliberately stop sending our own query parameters after the first call.

"""
ogc_api_features_paginator.py — page through an OGC API - Features collection.

Dependencies (pip install):
    requests>=2.31
"""
from __future__ import annotations

from typing import Any, Iterator
from urllib.parse import urljoin

import requests


def _next_link(links: list[dict[str, Any]]) -> str | None:
    """Return the href of the link whose rel == 'next', or None if absent."""
    for link in links:
        # rel MAY be a space-separated list; check membership defensively.
        rels = str(link.get("rel", "")).split()
        if "next" in rels:
            return link.get("href")
    return None


def iter_features(
    collection_url: str,
    *,
    limit: int = 1000,
    session: requests.Session | None = None,
    timeout: int = 30,
    extra_params: dict[str, str] | None = None,
) -> Iterator[dict[str, Any]]:
    """
    Yield GeoJSON features from an OGC API - Features items endpoint, lazily
    following the hypermedia rel="next" link across pages.

    collection_url : e.g. https://demo.example.com/collections/lakes/items
    limit          : requested page size; the server may cap it lower.
    session        : pass a shared requests.Session to reuse connections.
    extra_params   : filters such as {"bbox": "-1,50,1,52"} applied to page 1;
                     the server folds them into every subsequent next link.
    """
    owned_session = session is None
    session = session or requests.Session()
    session.headers.setdefault("Accept", "application/geo+json")

    # Only the FIRST request carries our query parameters. Each following page
    # is fetched verbatim from the rel="next" URL, which already encodes the
    # server's cursor, page size, and any active filter state.
    params: dict[str, str] | None = {"limit": str(limit)}
    if extra_params:
        params.update(extra_params)

    url: str | None = collection_url
    try:
        while url is not None:
            response = session.get(url, params=params, timeout=timeout)
            response.raise_for_status()
            payload = response.json()

            yield from payload.get("features", [])

            next_href = _next_link(payload.get("links", []))
            if next_href is None:
                url = None
            else:
                # The next href MAY be relative. Resolve it against the URL
                # that actually produced this response so redirects, proxies,
                # and path-only hrefs all normalise correctly.
                url = urljoin(response.url, next_href)

            # Query params are consumed once; the next URL is self-contained.
            params = None
    finally:
        if owned_session:
            session.close()


def collection_counts(
    collection_url: str,
    *,
    session: requests.Session | None = None,
    timeout: int = 30,
) -> dict[str, int | None]:
    """Fetch a single feature to read numberMatched / numberReturned."""
    session = session or requests.Session()
    response = session.get(
        collection_url,
        params={"limit": "1"},
        headers={"Accept": "application/geo+json"},
        timeout=timeout,
    )
    response.raise_for_status()
    payload = response.json()
    return {
        # Total across ALL pages. Optional — may be None on large collections.
        "number_matched": payload.get("numberMatched"),
        # Features returned in THIS response (here, at most 1).
        "number_returned": payload.get("numberReturned"),
    }


if __name__ == "__main__":
    items_url = "https://demo.pygeoapi.io/master/collections/lakes/items"

    counts = collection_counts(items_url)
    print(f"numberMatched={counts['number_matched']} "
          f"numberReturned={counts['number_returned']}")

    total = 0
    with requests.Session() as shared:
        for feature in iter_features(items_url, limit=50, session=shared):
            total += 1
            # Process each feature here without buffering the whole collection.
    print(f"accumulated {total} features across all pages")

Step-by-Step Walkthrough

Session reuse. iter_features accepts an optional requests.Session. Reusing one session keeps the underlying TCP connection warm across dozens or hundreds of page requests, which is the single largest latency win when a collection spans many pages. When the caller supplies a session the generator leaves it open; when it creates its own, the finally block closes it even if the consumer abandons the generator early. Setting Accept: application/geo+json with setdefault asks for the GeoJSON representation without clobbering a header the caller may have set deliberately.

The one-shot params. The limit and any extra_params are attached only to the first request. After the first page we set params = None, because the rel="next" URL already contains the server’s own limit and cursor. Re-appending ?limit=... to a URL that already has query parameters risks producing a malformed or contradictory request. This is the crux of hypermedia paging: after the entry point, the client stops constructing URLs and merely follows them.

Finding next. _next_link scans the links array for an entry advertising the next relation. The relation is checked with split() membership rather than string equality because the specification permits a space-separated list of relation types on a single link. If no next link is present, the helper returns None, the while condition fails, and iteration ends naturally.

Resolving relative hrefs. Some servers return an absolute next URL; others return only a path such as /collections/lakes/items?offset=50&limit=50. urljoin(response.url, next_href) handles both: an absolute href passes through unchanged, while a relative one is resolved against response.url — the final URL after any redirects, which is more reliable than the URL you originally requested.

Lazy yielding. yield from payload.get("features", []) streams each feature to the consumer immediately. Only one page ever lives in memory at a time, so the generator scales to collections far larger than available RAM. The counters read by collection_counts — numberReturned for the current page and the optional numberMatched for the grand total — are useful for progress bars but must never gate the loop, since a conformant server may omit numberMatched entirely.

This link-following model is the fundamental departure from the offset-driven approach used by the older protocol. If you are migrating an existing service, Converting a WFS Endpoint to OGC API Features walks through the endpoint mapping, while the WFS Transactional Operations Deep Dive documents the startIndex/count paging semantics that this generator replaces.

Verification

Run the module against the public pygeoapi demo server. Because numberMatched is advertised there, you can cross-check the streamed total against the reported count.

python ogc_api_features_paginator.py

Expected output shape (the exact totals depend on the live dataset):

numberMatched=25 numberReturned=1
accumulated 25 features across all pages

With a limit of 50 against a 25-feature collection the loop terminates after a single page, because the server returns no rel="next" link once every matching feature has been delivered. Lower the limit to 10 and re-run: the accumulated total stays 25, but the generator now transparently follows two additional next links to get there — proof that paging is driven entirely by the hypermedia links and not by your limit value.

Gotchas & Edge Cases

Back to OGC API Features and the REST Transition

Related:

• OGC API Features and the REST Transition — the JSON-native, hypermedia-driven successor to WFS and its conformance model
• Converting a WFS Endpoint to OGC API Features — mapping GetFeature and its paging parameters onto the new items endpoint
• WFS Transactional Operations Deep Dive — the startIndex/count offset paging model this generator replaces