CSW Catalog Service Integration

The OGC Catalogue Service for the Web defines the discovery layer of a spatial data infrastructure: a single HTTP contract for querying, paging, and retrieving the metadata records that describe every dataset and service in a catalogue. For engineers building harvesters, federated search, or portal back-ends, CSW is where interoperability is won or lost — the same four operations behave differently across the 2.0.2 and 3.0 versions, output schemas range from a flat Dublin Core summary to a complete ISO 19139 document, and spatial filters fail silently when axis order or queryable names are wrong. This page covers the CSW operation set, the two request bindings, output schemas and ElementSetName, Filter Encoding constraints, a complete owslib-based harvesting client, error handling, record validation, and incremental-harvest performance patterns.

Prerequisites & Architecture Context

Before integrating a CSW endpoint, engineering teams should be comfortable with:

• HTTP request/response semantics for both bindings CSW uses — HTTP GET with key-value pairs (KVP) and HTTP POST with an XML request body — plus content negotiation and connection reuse
• XML namespace handling in Python: lxml.etree, XPath with explicit namespace maps, and schema validation against the OGC and ISO XSDs
• ISO 19115 / ISO 19139 metadata structure — CSW is a transport for these records, and the Implementing ISO 19115 Metadata Standards cluster covers the element model you will be reading back out of GetRecords responses
• Python packages: owslib>=0.30 (CSW client and Filter Encoding builder), requests>=2.31 (raw HTTP fallback), lxml>=5.0 (XML parsing and validation), and optionally python-dateutil for parsing dc:modified timestamps

CSW is one member of a wider family of catalogue and metadata protocols covered in Spatial Metadata & Catalog Integration. It is a pull-based discovery interface: a client asks a server “which records match this filter?” and pages through the answers. That makes it the backbone of scheduled crawls — the Automated Metadata Harvesting Workflows cluster builds orchestration, deduplication, and error-recovery around exactly the GetRecords loop shown below. Knowing where CSW fits prevents a common mistake: treating it as a data-download API when it is strictly a metadata discovery API.

Specification Deep-Dive: Core Operations

CSW defines a small operation set. Two operations are always available for discovery, and two return records. The GetCapabilities contract, filters, and output schemas are shared across all of them.

GetCapabilities — The Service Contract

GetCapabilities returns an XML document describing the service: the CSW version (2.0.2 or 3.0), the supported operations and their bindings, the output schemas the server can emit, the ElementSetName values it honours, and — critically — the list of queryable properties you are allowed to filter on. Never hard-code queryable names; read them from the ows:Constraint and ogc:Filter_Capabilities sections of this document, because a server that advertises apiso:Modified but not dc:modified will reject a filter on the wrong name with an ExceptionReport.

Mandatory request parameters:

DescribeRecord — The Schema Contract

DescribeRecord returns the XML Schema for the record types the server exposes, letting a generic client discover the structure of csw:Record and, where supported, gmd:MD_Metadata. In practice most harvesters skip it — they already know the Dublin Core and ISO 19139 schemas — but it is invaluable when integrating a catalogue that publishes a vendor or profile-specific record type you have not seen before.

GetRecords — Search and Paging

GetRecords is the workhorse: it accepts a Filter Encoding constraint plus paging parameters and returns a csw:SearchResults element whose attributes drive the harvest loop. The response reports numberOfRecordsMatched (the full result-set size), numberOfRecordsReturned (how many are in this page), and nextRecord (the 1-based startPosition for the next page, or 0 when the set is exhausted).

A resultType=hits call is the cheapest way to size a harvest before paging: it returns numberOfRecordsMatched with zero records, so you can pre-compute the number of pages and set a progress bar.

GetRecordById — Direct Retrieval

GetRecordById fetches one or more records by their dc:identifier, bypassing the filter machinery entirely. It is the natural second stage of a two-phase harvest: page cheaply through GetRecords with ElementSetName=summary to collect identifiers, then pull the complete gmd:MD_Metadata for each with GetRecordById and ElementSetName=full. It takes an id (or comma-separated id list), an optional outputSchema, and an optional ElementSetName.

Request Bindings, Output Schemas, and ElementSetName

CSW 2.0.2 supports two request bindings. HTTP GET / KVP encodes the request as query-string key-value pairs — trivial to construct and cache, but constrained for complex filters because a full Filter Encoding document does not encode cleanly into a URL. HTTP POST / XML sends a <csw:GetRecords> document as the request body; this is the binding you want for anything beyond a trivial PropertyIsEqualTo, and it is what owslib uses for getrecords2.

The outputSchema parameter chooses what a record looks like on the wire:

ElementSetName is an orthogonal control over verbosity within a chosen schema. brief returns just identifier, title, type, and bounding box; summary adds subject, format, and abstract; full returns the complete record. Pair a cheap summary page-scan with a full GetRecordById retrieval to keep the paging loop light while still capturing every field for records you decide to keep.

Filter Encoding and CQL Constraints

The constraint on a GetRecords request can be expressed two ways. OGC Filter Encoding is an XML predicate tree — <ogc:PropertyIsLike>, <ogc:BBOX>, <ogc:And> — and is the interoperable default that owslib builds for you. CQL (Common Query Language) is a compact text syntax, e.g. AnyText LIKE '%flood%' AND BBOX(ows:BoundingBox, 51.0, -0.5, 52.0, 0.5), offered by many servers as a convenience; support is less universal, so prefer Filter Encoding for portable code.

The two constraints you will use most:

• PropertyIsLike on csw:AnyText — a full-text match across all indexed fields of a record. csw:AnyText is the Dublin Core “search everything” queryable; the ISO equivalent is apiso:AnyText. Wildcards default to % (single-character _), and escaping is via a declared escapeChar.
• BBOX spatial filter — a bounding-box intersection against the record’s ows:BoundingBox. This is where axis order bites: Filter Encoding evaluates the box in the record’s declared CRS, and EPSG:4326 in Filter Encoding is latitude-first. Always pass an explicit CRS URN and confirm the ordering the server expects.

Combine predicates with And / Or operators. A harvest that filters on subject, extent, and modification date at once is a single And of three leaf predicates — pushing all three to the server is far cheaper than filtering client-side after a full crawl.

Python Implementation

The client below connects to a CSW endpoint with owslib.csw.CatalogueServiceWeb, builds a combined text + spatial + date filter, runs getrecords2 over the POST/XML binding, and pages through the full result set. It requests the ISO 19139 output schema so each record carries complete metadata.

"""
csw_harvest.py — page a CSW 2.0.2 catalogue with a combined filter.

Dependencies (pip install):
    owslib>=0.30, lxml>=5.0
"""
from __future__ import annotations

import datetime as dt
from dataclasses import dataclass, field

from owslib.csw import CatalogueServiceWeb
from owslib.fes import And, BBox, PropertyIsGreaterThanOrEqualTo, PropertyIsLike

ISO_OUTPUT_SCHEMA = "http://www.isotc211.org/2005/gmd"


@dataclass
class HarvestConfig:
    url: str
    text: str                                   # matched against csw:AnyText
    bbox: tuple[float, float, float, float]     # minx, miny, maxx, maxy (lon/lat)
    modified_since: dt.date | None = None       # incremental high-water mark
    page_size: int = 20
    timeout: int = 30


@dataclass
class HarvestResult:
    matched: int = 0
    identifiers: list[str] = field(default_factory=list)
    titles: dict[str, str] = field(default_factory=dict)


def build_constraints(cfg: HarvestConfig) -> list:
    """Compose a single And() of text, spatial, and (optional) date predicates.

    owslib expects the top-level constraint as a list; a list with one And()
    element serialises to a proper <ogc:And> Filter Encoding tree.
    """
    leaves: list = [
        # % wildcards for a substring match across all indexed fields
        PropertyIsLike("csw:AnyText", f"%{cfg.text}%"),
        # crs URN pins axis handling; EPSG:4326 is lat-first in Filter Encoding
        BBox(list(cfg.bbox), crs="urn:ogc:def:crs:EPSG::4326"),
    ]
    if cfg.modified_since is not None:
        leaves.append(
            PropertyIsGreaterThanOrEqualTo(
                "apiso:Modified", cfg.modified_since.isoformat()
            )
        )
    # A single leaf must not be wrapped in And(); >= 2 leaves → one And().
    return [And(leaves)] if len(leaves) > 1 else leaves


def harvest(cfg: HarvestConfig) -> HarvestResult:
    """Connect, then page GetRecords until the result set is exhausted."""
    csw = CatalogueServiceWeb(cfg.url, timeout=cfg.timeout)

    result = HarvestResult()
    constraints = build_constraints(cfg)
    start = 1  # CSW startPosition is 1-based

    while True:
        csw.getrecords2(
            constraints=constraints,
            typenames="gmd:MD_Metadata",
            outputschema=ISO_OUTPUT_SCHEMA,
            esn="summary",              # ElementSetName: brief | summary | full
            startposition=start,
            maxrecords=cfg.page_size,
        )

        sr = csw.results  # {'matches', 'returned', 'nextrecord'}
        result.matched = sr["matches"]
        for rec_id, record in csw.records.items():
            result.identifiers.append(rec_id)
            result.titles[rec_id] = getattr(record, "title", "") or ""

        next_record = sr["nextrecord"]
        # nextrecord == 0 → done; > matches → guard against servers that overrun
        if next_record == 0 or next_record > sr["matches"] or sr["returned"] == 0:
            break
        start = next_record

    return result


def fetch_full_record(url: str, identifier: str, timeout: int = 30) -> str:
    """Retrieve one complete ISO record by id and return its raw XML."""
    csw = CatalogueServiceWeb(url, timeout=timeout)
    csw.getrecordbyid(
        id=[identifier],
        outputschema=ISO_OUTPUT_SCHEMA,
        esn="full",
    )
    record = csw.records[identifier]
    return record.xml.decode("utf-8") if isinstance(record.xml, bytes) else record.xml


if __name__ == "__main__":
    config = HarvestConfig(
        url="https://example.org/geonetwork/srv/eng/csw",
        text="flood",
        bbox=(-0.5, 51.0, 0.5, 52.0),
        modified_since=dt.date(2026, 1, 1),
        page_size=20,
    )
    outcome = harvest(config)
    print(f"matched {outcome.matched} records; collected {len(outcome.identifiers)} ids")
    for ident in outcome.identifiers[:3]:
        xml = fetch_full_record(config.url, ident)
        print(f"--- {ident} ({len(xml)} bytes of ISO 19139) ---")

Annotated Walkthrough

Connection (CatalogueServiceWeb) — Constructing the client immediately issues a GetCapabilities call under the hood, so owslib knows the server version and available operations before you query. Pass an explicit timeout so a slow or hung catalogue fails fast rather than blocking a scheduled harvest indefinitely.

Constraint assembly (build_constraints) — owslib.fes mirrors the Filter Encoding element tree: PropertyIsLike becomes <ogc:PropertyIsLike>, BBox becomes <ogc:BBOX>, and And nests them. The critical rule is the wrapping: a single predicate is passed as a one-element list, but two or more must be combined inside one And(...) and that And placed in a single-element list. Passing a bare list of two leaves produces an invalid filter that many servers reject with an ExceptionReport.

The BBox CRS argument — Passing crs="urn:ogc:def:crs:EPSG::4326" pins the axis interpretation. This is the CSW analogue of the axis-order trap documented across OGC services; without an explicit CRS the server falls back to its own default and a box that looks correct returns zero matches. If a spatial filter mysteriously returns nothing, swap the coordinate order before assuming the data is missing.

The paging loop (harvest) — After each getrecords2 call, csw.results exposes matches, returned, and nextrecord, and csw.records is an ordered dict of parsed records keyed by identifier. The loop advances startposition to nextrecord and stops on three conditions: nextrecord == 0 (the spec’s end-of-set signal), returned == 0 (an empty page, guarding against a misbehaving server), and nextrecord > matches (an overrun some servers produce). Belt-and-braces termination prevents an accidental infinite loop against a non-conformant catalogue.

Two-phase retrieval (fetch_full_record) — The page-scan uses esn="summary" to keep each page light; fetch_full_record then pulls the complete record with getrecordbyid and esn="full". Each parsed record exposes its raw XML via record.xml, which is what you hand to a schema validator or persist verbatim for later re-processing.

When owslib is not enough — owslib covers the common Dublin Core and ISO paths, but reach for raw requests + lxml when you need a queryable the model does not surface, a CSW 3.0-only feature, a vendor extension, or byte-level control of the POST body. The pattern is a requests.Session().post(url, data=xml_body, headers={"Content-Type": "application/xml"}) with a hand-built <csw:GetRecords> document, then lxml.etree.fromstring(response.content) parsed with an explicit namespace map. The Querying a CSW Catalog with owslib in Python guide walks the owslib path end to end, including the raw-XML fallback.

Error Handling & Edge Cases

ExceptionReport responses — A CSW server signals faults with an ows:ExceptionReport XML body, not an HTTP 4xx status — the transport is often HTTP 200 with an exception payload inside. owslib raises on some of these but silently returns empty records on others, so after every call check whether csw.results["matches"] is plausible and inspect csw.response (the raw bytes) when a query returns nothing unexpectedly. Wrap getrecords2 in a try/except for owslib.util.ServiceException and log the exception code and locator.

Version mismatch 2.0.2 vs 3.0 — CSW 3.0 changed operation and parameter names (for example outputFormat handling and the Query/Constraint structure) and reorganised the capabilities document. Pin the version at connection time with version="2.0.2" if your code assumes 2.0.2 semantics; do not let the client negotiate to 3.0 and then send 2.0.2-shaped requests. When you must support both, branch on the csw.version reported after connection.

Namespace handling — Records arrive with heavily namespaced XML (gmd:, gco:, csw:, ows:). When you drop to lxml, always supply a complete namespace map to XPath; a query written without the gmd prefix bound returns an empty node list rather than an error, which is maddening to debug. Never rely on default-namespace shortcuts.

Schema-declared but unindexed queryables — A server may advertise a queryable in its capabilities yet not have it indexed, so a filter on it returns zero rather than an error. Cross-check unexpected empty results with a resultType=hits call carrying only the text predicate to confirm the catalogue holds matching records at all.

Inconsistent maxRecords caps — Requesting maxRecords=1000 does not guarantee 1000 records; the server silently returns its own cap. Always drive paging from the returned nextrecord, never from an assumption that your requested page size was honoured.

Testing & Compliance Verification

Validate that returned ISO records actually conform before trusting them downstream. The snippet validates a harvested gmd:MD_Metadata document against the ISO 19139 schema and asserts the paging contract with a resultType=hits probe.

"""
test_csw_harvest.py — validate returned records and the paging contract.
"""
import lxml.etree as ET
import pytest

from owslib.csw import CatalogueServiceWeb
from owslib.fes import PropertyIsLike

CSW_URL = "https://example.org/geonetwork/srv/eng/csw"
GMD = "http://www.isotc211.org/2005/gmd"


def test_hits_count_is_nonnegative():
    """resultType=hits returns a count and zero records."""
    csw = CatalogueServiceWeb(CSW_URL, timeout=30)
    csw.getrecords2(
        constraints=[PropertyIsLike("csw:AnyText", "%flood%")],
        resulttype="hits",
    )
    assert csw.results["matches"] >= 0
    assert csw.results["returned"] == 0


def test_returned_iso_record_is_well_formed():
    """A full ISO record parses and carries the gmd root element."""
    csw = CatalogueServiceWeb(CSW_URL, timeout=30)
    csw.getrecords2(
        constraints=[PropertyIsLike("csw:AnyText", "%flood%")],
        typenames="gmd:MD_Metadata",
        outputschema=GMD,
        esn="full",
        maxrecords=1,
    )
    assert csw.records, "expected at least one record"
    record = next(iter(csw.records.values()))
    root = ET.fromstring(
        record.xml if isinstance(record.xml, bytes) else record.xml.encode()
    )
    assert root.tag == f"{{{GMD}}}MD_Metadata"


@pytest.mark.parametrize("page_size", [5, 10, 20])
def test_next_record_never_exceeds_matches_by_a_page(page_size):
    """nextrecord stays within one page of the matched total."""
    csw = CatalogueServiceWeb(CSW_URL, timeout=30)
    csw.getrecords2(
        constraints=[PropertyIsLike("csw:AnyText", "%data%")],
        maxrecords=page_size,
    )
    sr = csw.results
    assert sr["nextrecord"] <= sr["matches"] + page_size

For full ISO conformance testing, validate each harvested record against the official ISO 19139 XSD with lxml.etree.XMLSchema — the Implementing ISO 19115 Metadata Standards cluster covers loading the schema set and interpreting validation errors. For the CSW interface itself, the OGC CITE (Compliance, Interoperability, Testing, Evaluation) test suite for CSW 2.0.2 exercises the mandatory operations, filter capabilities, and paging behaviour against your endpoint.

Performance & Scaling Notes

Right-size paging — Larger pages mean fewer round-trips but heavier responses and more server memory per request. For ISO full records a page of 10 to 25 is usually optimal; for a summary identifier-scan you can push toward the server cap. Always start with a resultType=hits probe so you know the total before committing to a page size.

Incremental harvest by dc:modified — A full re-crawl of a large catalogue is wasteful when only a handful of records changed. Persist the timestamp of each successful run and add a PropertyIsGreaterThanOrEqualTo on apiso:Modified (or dc:modified for Dublin Core) to the next run’s filter, turning a full crawl into a cheap delta. The Scheduling Recurring CSW Harvests with Python guide builds the high-water-mark bookkeeping and cron orchestration around this predicate.

Connection reuse — Constructing a fresh CatalogueServiceWeb per page re-issues GetCapabilities every time. Build the client once and reuse it across the paging loop, and when dropping to raw HTTP use a single requests.Session with a HTTPAdapter sized for your concurrency so TCP and TLS handshakes are amortised across hundreds of record fetches.

Two-phase over one-phase — Paging full ISO records is expensive; paging summary records to collect identifiers and then fetching only the records you actually want with GetRecordById moves the heavy transfer off the critical scan path and lets you parallelise the per-record retrieval.

Politeness and backoff — Public catalogues rate-limit. Add a short delay between pages, honour HTTP 429 with exponential backoff, and cap concurrent GetRecordById calls; a harvester that hammers a shared national catalogue will be throttled or blocked long before it finishes.

Gotchas / Frequently Asked Questions

What is the difference between csw:Record and gmd:MD_Metadata output?

csw:Record is the Dublin Core profile — a flat set of common queryable fields (title, subject, type, bounding box) that every CSW server is required to support. gmd:MD_Metadata is the full ISO 19139 encoding of ISO 19115, carrying lineage, distribution, contacts, and constraints. Choose the output schema per query with outputSchema: request Dublin Core for fast cross-catalogue discovery, and switch to the ISO schema when you need the complete record for capture or re-publishing.

How do I page through a large CSW result set?

Set maxRecords to your page size and increment startPosition (which is 1-based) by that size on each call. The server returns numberOfRecordsMatched and a nextRecord pointer inside SearchResults; stop when nextRecord is 0 or greater than the matched total. Never assume the whole set arrives in one response — most servers cap maxRecords between 10 and 100 and silently return their cap regardless of what you request.

Why does my BBOX filter return no records against a CSW server?

Two common causes: coordinate axis order and the queryable property name. CSW BBOX filters run against ows:BoundingBox in the record’s declared CRS, and for EPSG:4326 in Filter Encoding that is latitude-first. owslib’s BBox helper takes minx,miny,maxx,maxy, so pass crs="urn:ogc:def:crs:EPSG::4326" explicitly and confirm from the capabilities document whether the server expects lat,lon or lon,lat before assuming your data is missing.

Should I use owslib or raw requests plus lxml for CSW?

Use owslib.csw.CatalogueServiceWeb for the common case — it builds the Filter Encoding XML, manages paging state, and parses records into typed objects. Drop to requests plus lxml when you need a queryable the owslib model does not expose, a vendor extension, a CSW 3.0 feature owslib does not yet cover, or byte-level control over the POST body while debugging an ExceptionReport.

How do I harvest only records that changed since the last run?

Add a PropertyIsGreaterThanOrEqualTo constraint on dc:modified (or the ISO apiso:Modified queryable) set to the timestamp of your previous successful harvest, and combine it with your spatial and text predicates using an And operator. Persist the high-water mark after each run and you convert a full re-crawl into a cheap incremental delta — the foundation of any scheduled harvesting pipeline.

---

Back to Spatial Metadata & Catalog Integration

Related:

• Querying a CSW Catalog with owslib in Python — a focused, end-to-end owslib walkthrough with the raw requests + lxml fallback
• Scheduling Recurring CSW Harvests with Python — cron orchestration and dc:modified high-water-mark bookkeeping for incremental crawls
• Implementing ISO 19115 Metadata Standards — the element model behind the gmd:MD_Metadata records CSW returns, and how to validate them
• Automated Metadata Harvesting Workflows — deduplication, error recovery, and orchestration around the GetRecords loop
• DCAT-AP for Spatial Data Portals — the RDF catalogue vocabulary you map harvested CSW records into for open-data portals