Streaming Large GeoJSON Feature Collections in Python

Request with stream=True, hand response.raw to ijson.items(raw, "features.item"), and yield one feature at a time. Peak memory then tracks the largest single feature rather than the payload, and the first feature is available before the download finishes. The hard part is not the parser — it is finding the list comprehension downstream that collects everything anyway.

The Core Challenge: json.loads Costs Five Times the Payload

A GeoJSON FeatureCollection is a single JSON object. json.loads must therefore read every byte before it can return anything, and the Python object graph it produces — dicts, lists, floats, one object per coordinate pair — costs roughly five to six times the serialised size. A 400 MB export becomes well over two gigabytes resident, and nothing at all happens until the last byte lands.

This matters more for OGC API - Features than it did for WFS, because a JSON payload has no equivalent of the SAX-style incremental parsing that XML tooling has offered for decades. ijson fills that gap: it is an incremental JSON parser that emits events as it reads, and its items helper turns a JSON path prefix into a generator of complete objects.

Production-Ready Code

from __future__ import annotations

from dataclasses import dataclass, field
from typing import Any, Iterator

import ijson
import requests
from shapely.geometry import shape

@dataclass
class Feature:
    fid: str | None
    geometry: Any
    properties: dict[str, Any] = field(default_factory=dict)

def stream_features(url: str, params: dict[str, str] | None = None,
                    timeout: int = 300) -> Iterator[Feature]:
    """Yield features from a GeoJSON FeatureCollection at constant memory.

    stream=True stops requests from reading the body into memory; ijson then
    pulls from the socket as it parses. The 'features.item' prefix selects
    each element of the top-level features array, so each yielded dict is a
    complete feature and nothing larger is ever materialised.
    """
    with requests.get(url, params=params, stream=True, timeout=timeout) as resp:
        resp.raise_for_status()
        resp.raw.decode_content = True          # transparently gunzip
        for item in ijson.items(resp.raw, "features.item", use_float=True):
            geometry = item.get("geometry")
            yield Feature(
                fid=str(item["id"]) if item.get("id") is not None else None,
                geometry=shape(geometry) if geometry else None,
                properties=dict(item.get("properties") or {}),
            )

def stream_ndjson(url: str, timeout: int = 300) -> Iterator[Feature]:
    """Newline-delimited GeoJSON: one Feature object per line.

    Cheaper than ijson when the producer offers it, because each line is an
    independent document that json.loads handles in isolation.
    """
    import json

    with requests.get(url, stream=True, timeout=timeout) as resp:
        resp.raise_for_status()
        for line in resp.iter_lines(decode_unicode=True):
            if not line:
                continue
            item = json.loads(line)
            geometry = item.get("geometry")
            yield Feature(
                fid=str(item["id"]) if item.get("id") is not None else None,
                geometry=shape(geometry) if geometry else None,
                properties=dict(item.get("properties") or {}),
            )

def in_batches(features: Iterator[Feature], size: int = 500):
    """Group a feature stream into bounded lists for batched writes.

    This is the only place a list is allowed to exist, and its length is
    fixed — which is what keeps the whole pipeline's memory flat.
    """
    batch: list[Feature] = []
    for feature in features:
        batch.append(feature)
        if len(batch) >= size:
            yield batch
            batch = []
    if batch:
        yield batch

if __name__ == "__main__":
    url = "https://example.org/ogcapi/collections/parcels/items"
    written = 0
    for batch in in_batches(stream_features(url, {"limit": "10000", "f": "json"})):
        # write_batch(batch) — one round trip per 500 features
        written += len(batch)
        print(f"{written} features written", end="\r")

Step-by-Step Walkthrough

stream=True is load-bearing. Without it, requests reads the entire response body into memory before returning, and every parser downstream is incremental over an object that is already fully resident. With it, resp.raw is a file-like view onto the socket and ijson pulls bytes as it needs them.

decode_content = True handles gzip. Servers routinely gzip a large GeoJSON response. Setting this on the raw stream makes urllib3 decompress transparently as it reads, so the parser sees JSON rather than compressed bytes. Without it, ijson fails immediately with a parse error on the gzip magic number, which is a confusing symptom for an otherwise correct pipeline.

use_float=True avoids Decimal. ijson defaults to parsing JSON numbers as Decimal for exactness, which is correct for financial data and wrong here: shapely expects floats, and the conversion cost across millions of coordinates is significant. Setting the flag once at the boundary avoids a conversion in every geometry constructor.

in_batches is where a list is permitted. Streaming is not about avoiding lists entirely — a database writer that commits one row per transaction is slower than one that commits five hundred. The rule is that every list in the pipeline must have a bounded length known in advance. That is what separates batching from collecting.

Verification

Prove the memory property rather than assuming it, using the standard library’s allocation tracer:

import tracemalloc

tracemalloc.start()
count = 0
for feature in stream_features(url, {"limit": "50000", "f": "json"}):
    count += 1
    if count % 10000 == 0:
        current, peak = tracemalloc.get_traced_memory()
        print(f"{count:>7} features  current={current/1e6:.1f} MB  peak={peak/1e6:.1f} MB")

A correct implementation prints a peak that stops rising:

  10000 features  current=3.1 MB  peak=4.4 MB
  20000 features  current=3.1 MB  peak=4.4 MB
  30000 features  current=3.2 MB  peak=4.6 MB

A number that climbs linearly with the feature count means something downstream is holding references, which is exactly what the diagnostic below is for.

Gotchas & Edge Cases

One list comprehension undoes all of it. features = [f for f in stream_features(url)] is a generator consumed into a list, which has precisely the memory profile the generator was written to avoid. This is the single most common way streaming pipelines regress, because the change looks harmless in review.

Deduplication sets grow without bound. A pipeline that skips features it has already seen holds every identifier for the run. On ten million features that set alone is hundreds of megabytes. Bound it — a bloom filter, a windowed set, or a unique constraint in the destination — or accept that memory scales with input.

ijson has three backends with very different speeds. The pure-Python backend is portable and slow; yajl2_c is several times faster and is what production ingest should use. Installing it is a packaging concern rather than a code change, and confirming which backend is active is one line: ijson.backend.

Paging is still better than one huge response. Even a perfectly streaming client loses the whole run if the connection drops at ninety per cent. Requesting bounded pages and following the rel="next" link, as described in Paginating OGC API - Features Collections in Python, makes a failure cost one page rather than everything.

Frequently Asked Questions

---

Back to GML and GeoJSON Payload Handling

Related

• GML and GeoJSON Payload Handling — the shared ingest path both encodings feed
• Paginating OGC API - Features Collections in Python — bounding the response instead of streaming one huge one
• Validating GeoJSON Against a JSON Schema in Python — checking each feature as it streams past