Harvesting DCAT-AP Catalogs with Python and rdflib

TL;DR: Content-negotiate an RDF serialisation with an Accept header, load it with rdflib’s Graph().parse(), bind the dcat, dcterms and foaf namespaces, then run a SPARQL SELECT to pull each dcat:Dataset and its nested dcat:distribution into Python dicts. Follow hydra:next links to walk paged catalogues, and traverse from the dataset so blank-node distributions resolve without ever needing a stable identifier.

The Core Challenge

A DCAT-AP catalogue is not a JSON API you can page through with a predictable key path. It is an RDF graph — a set of subject/predicate/object triples — serialised as Turtle, RDF/XML or JSON-LD, where the same logical catalogue can arrive in any of those syntaxes depending on what the server decides to send you. DCAT-AP is the European application profile of the W3C Data Catalog Vocabulary; the wider modelling rules are covered in DCAT-AP for Spatial Data Portals. Harvesting it means solving three problems that a naive HTTP client does not:

  • Serialisation is negotiated, not fixed. You ask for a format with an Accept header, but the server may hand back a different one — or ignore you entirely and return HTML. The parser you pick has to match what actually arrived, not what you requested.
  • Distributions are usually blank nodes. A dcat:Distribution — the downloadable representation of a dataset, carrying dcat:accessURL, dcat:downloadURL and dcterms:format — is typically an anonymous node with no URI. You reach it only by walking the edge from its parent dcat:Dataset.
  • Large catalogues are paged. Portals split their dcat:Catalog across many pages using the Hydra vocabulary. Each page points to the next with a hydra:next triple, and you must follow that chain to see every record.
DCAT-AP Harvesting Data Flow with rdflib Left to right flow: the harvester sends an HTTP GET with an Accept header to the catalogue endpoint; the RDF serialisation (Turtle, RDF/XML or JSON-LD) is parsed by rdflib into a graph of triples containing blank-node distributions; a SPARQL SELECT extracts each dataset and its distributions into Python dictionaries; a hydra:next link loops back to fetch the following page. Catalogue endpoint dcat:Catalog GET Accept: RDF payload Turtle / RDF-XML / JSON-LD .parse() rdflib Graph triples + blank nodes SPARQL Python dicts dataset + distributions hydra:next → next page Blank-node distribution (no URI) reached only by walking dcat:distribution from the dataset

The wider harvesting patterns — scheduling, deduplication, and reconciling records across protocols such as the OGC Catalogue Service for the Web described in CSW Catalog Service Integration — build on the single-catalogue read shown here.

Production-Ready Code

The script below content-negotiates the RDF, parses each page into an rdflib.Graph, follows hydra:next links, and runs one SPARQL SELECT per page. The only third-party dependencies are rdflib and requests; everything else is standard library.

"""
harvest_dcat_ap.py — harvest a DCAT-AP catalogue with rdflib

Dependencies (pip install):
    rdflib>=7.0, requests>=2.31
"""
from __future__ import annotations

import json
from typing import Any, Iterator

import requests
from rdflib import Graph, Namespace
from rdflib.namespace import DCAT, DCTERMS, FOAF, RDF  # noqa: F401  (bound below)

# Serialisations we can parse, in preference order, mapped to rdflib format ids.
NEGOTIATION: dict[str, str] = {
    "text/turtle": "turtle",
    "application/rdf+xml": "xml",
    "application/ld+json": "json-ld",
}

# Hydra is not shipped as a bundled rdflib namespace; declare it explicitly.
HYDRA = Namespace("http://www.w3.org/ns/hydra/core#")

# One SPARQL query pulls each dcat:Dataset plus its nested, usually
# blank-node, dcat:distribution. Every OPTIONAL means missing predicates
# yield NULL rather than dropping the dataset from the results.
DATASET_SPARQL = """
PREFIX dcat:    <http://www.w3.org/ns/dcat#>
PREFIX dcterms: <http://purl.org/dc/terms/>
PREFIX foaf:    <http://xmlns.com/foaf/0.1/>

SELECT ?dataset ?title ?identifier ?theme ?publisher
       ?accessURL ?downloadURL ?format
WHERE {
    ?dataset a dcat:Dataset .
    OPTIONAL { ?dataset dcterms:title      ?title }
    OPTIONAL { ?dataset dcterms:identifier ?identifier }
    OPTIONAL { ?dataset dcat:theme         ?theme }
    OPTIONAL { ?dataset dcterms:publisher [ foaf:name ?publisher ] }
    OPTIONAL {
        ?dataset dcat:distribution ?dist .
        OPTIONAL { ?dist dcat:accessURL   ?accessURL }
        OPTIONAL { ?dist dcat:downloadURL ?downloadURL }
        OPTIONAL { ?dist dcterms:format   ?format }
    }
}
ORDER BY ?dataset
"""


def fetch_graph(url: str, timeout: int = 30) -> Graph:
    """Content-negotiate an RDF serialisation and parse it into a Graph."""
    accept = ", ".join(NEGOTIATION)  # e.g. "text/turtle, application/rdf+xml, ..."
    resp = requests.get(url, headers={"Accept": accept}, timeout=timeout)
    resp.raise_for_status()

    # Trust the response Content-Type, not the format we asked for.
    content_type = resp.headers.get("Content-Type", "").split(";")[0].strip()
    fmt = NEGOTIATION.get(content_type)

    graph = Graph()
    graph.bind("dcat", DCAT)
    graph.bind("dcterms", DCTERMS)
    graph.bind("foaf", FOAF)
    graph.bind("hydra", HYDRA)

    if fmt:
        graph.parse(data=resp.text, format=fmt)
    else:
        # Server ignored our Accept header; let rdflib sniff the body.
        graph.parse(data=resp.text)
    return graph


def harvest_pages(start_url: str, max_pages: int = 500) -> Iterator[Graph]:
    """Yield one Graph per page, following hydra:next until it runs out."""
    url: str | None = start_url
    seen: set[str] = set()
    while url and url not in seen and len(seen) < max_pages:
        seen.add(url)
        graph = fetch_graph(url)
        yield graph
        # hydra:next binds to the URL of the following page, or nothing.
        nxt = graph.value(predicate=HYDRA.next)
        url = str(nxt) if nxt is not None else None


def query_datasets(graph: Graph) -> dict[str, dict[str, Any]]:
    """Run the SPARQL query and fold the row product back into one dict/dataset."""
    records: dict[str, dict[str, Any]] = {}
    for row in graph.query(DATASET_SPARQL):
        key = str(row.dataset)
        rec = records.setdefault(key, {
            "uri": key,
            "title": str(row.title) if row.title else None,
            "identifier": str(row.identifier) if row.identifier else None,
            "publisher": str(row.publisher) if row.publisher else None,
            "themes": set(),
            "distributions": {},  # keyed to deduplicate the OPTIONAL product
        })
        if row.theme:
            rec["themes"].add(str(row.theme))
        if row.accessURL or row.downloadURL or row.format:
            dist_key = (
                str(row.accessURL or ""),
                str(row.downloadURL or ""),
                str(row.format or ""),
            )
            rec["distributions"].setdefault(dist_key, {
                "access_url": str(row.accessURL) if row.accessURL else None,
                "download_url": str(row.downloadURL) if row.downloadURL else None,
                "format": str(row.format) if row.format else None,
            })
    return records


def harvest(start_url: str) -> list[dict[str, Any]]:
    """Harvest every page and return a flat list of JSON-friendly dataset dicts."""
    merged: dict[str, dict[str, Any]] = {}
    for graph in harvest_pages(start_url):
        for key, rec in query_datasets(graph).items():
            if key in merged:
                merged[key]["themes"].update(rec["themes"])
                merged[key]["distributions"].update(rec["distributions"])
            else:
                merged[key] = rec

    return [
        {
            "uri": rec["uri"],
            "title": rec["title"],
            "identifier": rec["identifier"],
            "publisher": rec["publisher"],
            "themes": sorted(rec["themes"]),
            "distributions": list(rec["distributions"].values()),
        }
        for rec in merged.values()
    ]


if __name__ == "__main__":
    datasets = harvest("https://data.europa.eu/api/hub/search/datasets.ttl")
    print(f"harvested {len(datasets)} datasets")
    print(json.dumps(datasets[:2], indent=2, ensure_ascii=False))

Step-by-Step Walkthrough

Content negotiation (fetch_graph). The Accept header lists all three serialisations rdflib handles, comma-joined in preference order, so a compliant server returns the first it supports. Crucially, the function reads the response Content-Type — stripped of any ; charset=... suffix — and maps that to the rdflib format id, rather than assuming the reply matches the request. When the header is missing or unrecognised (fmt is None), it falls back to graph.parse(data=...) with no format, letting rdflib sniff the body. This is the single most important guard against BadSyntax errors on portals that quietly serve JSON-LD when you asked for Turtle.

Namespace binding. DCAT, DCTERMS and FOAF are bundled rdflib namespace objects, so DCAT.distribution and DCTERMS.title expand to full URIs without any string typing. Binding them onto the graph with graph.bind(...) keeps prefixes tidy in any re-serialisation and documents intent. Hydra is not bundled, so HYDRA is declared with Namespace(...) against http://www.w3.org/ns/hydra/core#.

Paging (harvest_pages). After each page is parsed, graph.value(predicate=HYDRA.next) returns the object of the single hydra:next triple — the URL of the following page — or None when the chain ends. The generator tracks a seen set and a max_pages ceiling so a server that accidentally links a page back to itself cannot spin forever. Yielding one Graph per page keeps memory flat: you never hold the whole catalogue in a single graph.

The SPARQL query (DATASET_SPARQL). Every property is wrapped in OPTIONAL so a dataset missing a dcterms:identifier or a dcat:theme is still returned rather than filtered out. The nested block binds ?dist to each dcat:distribution and reads dcat:accessURL, dcat:downloadURL and dcterms:format from it. Because ?dist is an intermediate variable, SPARQL resolves the blank node for you — you never have to name it. The dcterms:publisher [ foaf:name ?publisher ] pattern uses a blank-node property path to reach the publisher’s foaf:name in one hop, which is why the foaf prefix is bound.

Folding the row product (query_datasets). A SELECT returns the Cartesian product of all OPTIONAL matches, so a dataset with three themes and two distributions produces six rows. The function keys each record by the dataset URI via setdefault, adds themes to a set, and deduplicates distributions in a dict keyed on the (accessURL, downloadURL, format) tuple. harvest then merges records across pages and normalises the sets and dicts into sorted lists for clean JSON output.

Verification

Point the harvester at any DCAT-AP endpoint that serves a .ttl (or negotiable RDF) representation and inspect the first record:

python harvest_dcat_ap.py

Expected output shape (truncated):

harvested 128 datasets
[
  {
    "uri": "https://data.example.eu/dataset/corine-land-cover-2018",
    "title": "CORINE Land Cover 2018",
    "identifier": "corine-land-cover-2018",
    "publisher": "European Environment Agency",
    "themes": [
      "http://publications.europa.eu/resource/authority/data-theme/ENVI"
    ],
    "distributions": [
      {
        "access_url": "https://data.example.eu/dataset/corine-land-cover-2018",
        "download_url": "https://data.example.eu/files/clc2018.gpkg.zip",
        "format": "http://publications.europa.eu/resource/authority/file-type/GPKG"
      }
    ]
  }
]

Confirm that themes holds resolvable authority URIs, that each distribution carries at least one of access_url or download_url, and that the dataset count matches the total the portal advertises across all pages (a mismatch usually means the hydra:next chain terminated early).

Gotchas & Edge Cases

Why does rdflib raise a parser error even though the catalogue returned HTTP 200?

The 200 only tells you the transfer succeeded — not that the body is the format you passed to Graph.parse(). Portals frequently ignore the Accept header and return HTML content-negotiated for a browser, or a different RDF serialisation than you requested. The fix in fetch_graph is to read the response Content-Type, map it to an rdflib format id, and only fall back to format sniffing (parse with no format) when the header is missing or generic. If you still get a BadSyntax on the first bytes, print resp.text[:200] — an opening <!DOCTYPE html> means you negotiated your way into a web page.

How do I read a distribution's accessURL when it is a blank node?

A blank node has no stable identifier, so you cannot look it up directly or persist a reference to it across parses. Reach it by traversing the edge from its parent: either follow graph.objects(dataset, DCAT.distribution) and read DCAT.accessURL, DCAT.downloadURL and DCTERMS.format off each returned node, or — as in DATASET_SPARQL — let the query bind the intermediate ?dist variable. Both resolve the blank node automatically. Never write the blank-node id (for example N3f2a...) to your database; it is only meaningful inside the single graph that produced it.

My harvest stops after one page — where did the rest of the catalogue go?

The catalogue is paged and your first request only returned the first slice. DCAT-AP paged responses use the Hydra vocabulary: each page carries a hydra:next triple whose object is the URL of the following page (and often hydra:totalItems for the grand total). harvest_pages reads that object with graph.value(predicate=HYDRA.next) and loops until it is absent. If you see only one page, check that the server actually emits hydra:next — some expose paging through a non-standard dcat:Catalog link or a custom link header instead, in which case you adapt the “find next URL” step accordingly.

Why does one dataset appear as several rows in my SPARQL results?

A SPARQL SELECT returns the Cartesian product of every OPTIONAL match, so a dataset with three dcat:theme values and two distributions produces 3 × 2 = 6 rows — each a valid combination. This is expected, not a bug. Aggregate in Python: key each record by the ?dataset URI, collect themes into a set, and deduplicate distributions on a tuple of their URLs and format before serialising, exactly as query_datasets does. Expecting one row per dataset will silently drop themes and distributions.

Back to DCAT-AP for Spatial Data Portals

Related: