CI/CD and Compliance Testing for Spatial Services

A GeoServer or MapServer endpoint can return HTTP 200 for its landing page and still be broken in ways that only surface when a real client connects: a layer silently dropped during a catalog reload, a coordinate reference system missing from the advertised list, or a GetCapabilities document that no longer validates against the OGC schema after a version upgrade. These defects are invisible to a smoke test that only checks the process is up. The remedy is to treat every OGC service as a compiled artifact that must pass automated compliance tests before it is allowed to deploy. This page shows how to build that gate: a pytest suite that asserts capabilities validity, layer presence, CRS advertisement, and axis-order correctness; an invocation of the official OGC CITE conformance engine from CI; a disposable GeoServer provisioned with docker-compose as a service container; and a GitHub Actions workflow that blocks any deploy whose compliance checks are not green.

Prerequisites & Architecture Context

Before wiring compliance tests into a pipeline, engineering teams should be comfortable with:

• HTTP request/response handling in Python via requests, including timeouts, retries, and reading raw response bytes for XML parsing
• XML schema validation with lxml.etree.XMLSchema and XPath queries using explicit namespace maps — OGC capabilities documents are namespaced and version-sensitive
• Container orchestration basics: docker-compose service definitions, health checks, and the difference between a port being open and an application being ready
• GitHub Actions concepts — jobs, steps, services: containers, actions/cache, and artifact upload — plus pytest fixtures and parametrisation

This work sits downstream of the automation covered across Python Automation for GeoServer & MapServer: once you can publish layers programmatically, the next discipline is proving that what you published is correct before it ships. The provisioning half of the pipeline reuses the same REST calls described in Automating GeoServer with the Python REST API, and the container-and-configuration approach builds directly on Environment Parity for Spatial Servers — a compliance test is only meaningful if the CI service container is byte-for-byte the environment you deploy. The assertions themselves target the GetCapabilities contract explained in Understanding OGC Web Map Service Specifications, so knowing which elements are mandatory in WMS 1.3.0 is a prerequisite for asserting they are present.

Compliance Contract: What a Green Build Must Prove

A deploy gate is only as good as the properties it asserts. For an OGC service, four categories of check catch the overwhelming majority of production regressions. The table below maps each property to the concrete assertion and the failure it prevents.

The first five rows are fast, self-authored assertions — the kind of thing a pytest module runs in seconds. The last row is the authoritative, exhaustive conformance check performed by the OGC CITE (Compliance + Interoperability Testing + Evaluation) TEAM Engine. The two are complementary: pytest gives you a tight feedback loop on the properties you care about most, while CITE guarantees you have not violated a corner of the specification you never thought to test. A robust pipeline runs the pytest gate on every push and the CITE gate on merges and releases.

The GetCapabilities parsing contract

Both WMS versions carry the same conceptual information but under different roots and namespaces: WMS 1.3.0 uses a WMS_Capabilities root in the http://www.opengis.net/wms namespace, whereas WMS 1.1.1 uses WMT_MS_Capabilities with no namespace at all. Compliance assertions must therefore either pin a version explicitly (recommended in CI, where you control the request) or branch on the detected root. Because CI always issues the request itself, the simplest and most deterministic choice is to force VERSION=1.3.0 and assert against the namespaced schema.

Python Implementation

The module below is a complete, runnable compliance harness. It contains a wait-for-ready poll loop, a schema validator that loads a locally cached XSD, targeted capabilities assertions, and a helper that drives the OGC CITE TEAM Engine through its REST API. It is written as a pytest module so it plugs directly into CI, but the functions are importable for ad-hoc use.

"""
test_ogc_compliance.py — compliance gate for an OGC WMS/WFS endpoint.

Dependencies (pip install):
    pytest>=8.0, requests>=2.31, lxml>=5.0

Environment variables:
    OGC_BASE_URL   base URL of the service under test, e.g. http://localhost:8080/geoserver
    CITE_BASE_URL  base URL of the TEAM Engine REST API, e.g. http://localhost:8081/teamengine
    OGC_XSD_PATH   local path to the cached capabilities_1_3_0.xsd
"""
from __future__ import annotations

import os
import time
from pathlib import Path

import pytest
import requests
from lxml import etree

BASE_URL = os.environ.get("OGC_BASE_URL", "http://localhost:8080/geoserver")
CITE_URL = os.environ.get("CITE_BASE_URL", "http://localhost:8081/teamengine")
XSD_PATH = Path(os.environ.get("OGC_XSD_PATH", "schemas/wms/1.3.0/capabilities_1_3_0.xsd"))

WMS_NS = {"wms": "http://www.opengis.net/wms"}
EXPECTED_LAYERS = {"topp:states", "topp:tasmania_roads"}
REQUIRED_CRS = {"EPSG:4326", "EPSG:3857"}


# ---------------------------------------------------------------------------
# 1. Wait-for-ready — service containers open the port long before the app loads
# ---------------------------------------------------------------------------

def wait_for_capabilities(
    base_url: str,
    *,
    timeout: float = 180.0,
    interval: float = 3.0,
) -> bytes:
    """Poll GetCapabilities until it returns valid-looking XML or the timeout expires.

    Returns the raw capabilities bytes. Raises TimeoutError if the service never
    becomes ready — a hard failure that must fail the CI job, not hang it.
    """
    url = f"{base_url}/wms"
    params = {"SERVICE": "WMS", "VERSION": "1.3.0", "REQUEST": "GetCapabilities"}
    deadline = time.monotonic() + timeout
    last_error = "no attempt made"

    while time.monotonic() < deadline:
        try:
            resp = requests.get(url, params=params, timeout=10)
            if resp.status_code == 200 and resp.content.lstrip().startswith(b"<"):
                # Confirm it parses and has the expected root before declaring ready
                root = etree.fromstring(resp.content)
                if etree.QName(root).localname == "WMS_Capabilities":
                    return resp.content
                last_error = f"unexpected root {etree.QName(root).localname!r}"
            else:
                last_error = f"HTTP {resp.status_code}"
        except (requests.RequestException, etree.XMLSyntaxError) as exc:
            last_error = f"{type(exc).__name__}: {exc}"
        time.sleep(interval)

    raise TimeoutError(f"service at {url} not ready after {timeout}s (last: {last_error})")


# ---------------------------------------------------------------------------
# 2. Schema validation against a locally cached OGC XSD (no network in CI)
# ---------------------------------------------------------------------------

def validate_against_xsd(capabilities: bytes, xsd_path: Path) -> None:
    """Raise AssertionError with the first schema error if validation fails."""
    if not xsd_path.exists():
        raise FileNotFoundError(f"cached XSD not found at {xsd_path}; run the fetch step")
    schema = etree.XMLSchema(etree.parse(str(xsd_path)))
    doc = etree.fromstring(capabilities)
    if not schema.validate(doc):
        first = schema.error_log[0]
        raise AssertionError(f"capabilities failed XSD validation: {first.message} (line {first.line})")


# ---------------------------------------------------------------------------
# 3. Targeted capabilities assertions
# ---------------------------------------------------------------------------

def advertised_layer_names(capabilities: bytes) -> set[str]:
    doc = etree.fromstring(capabilities)
    return {el.text for el in doc.iterfind(".//wms:Layer/wms:Name", WMS_NS) if el.text}


def advertised_crs_for(capabilities: bytes, layer_name: str) -> set[str]:
    doc = etree.fromstring(capabilities)
    for layer in doc.iterfind(".//wms:Layer", WMS_NS):
        name = layer.findtext("wms:Name", namespaces=WMS_NS)
        if name == layer_name:
            return {el.text.strip() for el in layer.iterfind("wms:CRS", WMS_NS) if el.text}
    return set()


# ---------------------------------------------------------------------------
# 4. OGC CITE TEAM Engine driver (REST API)
# ---------------------------------------------------------------------------

def run_cite_wms(cite_url: str, service_capabilities_url: str, *, timeout: float = 600.0) -> dict:
    """Trigger a WMS 1.3.0 conformance run and return a parsed summary.

    TEAM Engine's REST API exposes suites at /rest/suites/<suite>/<version>/run and
    returns an EARL/XML report. We request JSON where available and fall back to
    counting <test> verdicts in the XML report otherwise.
    """
    run_url = f"{cite_url}/rest/suites/wms/1.3.0/run"
    resp = requests.get(
        run_url,
        params={"capabilities-url": service_capabilities_url},
        headers={"Accept": "application/xml"},
        timeout=timeout,
    )
    resp.raise_for_status()
    report = etree.fromstring(resp.content)

    # EARL verdicts: passed / failed / skipped counts across the report
    verdicts = report.findall(".//{http://www.w3.org/ns/earl#}outcome")
    counts = {"passed": 0, "failed": 0, "skipped": 0}
    for v in verdicts:
        outcome = v.get("{http://www.w3.org/1999/02/22-rdf-syntax-ns#}resource", "")
        if outcome.endswith("#passed"):
            counts["passed"] += 1
        elif outcome.endswith("#failed"):
            counts["failed"] += 1
        else:
            counts["skipped"] += 1
    return counts


# ---------------------------------------------------------------------------
# pytest wiring
# ---------------------------------------------------------------------------

@pytest.fixture(scope="session")
def capabilities() -> bytes:
    """Session-scoped: fetch capabilities once after the service is ready."""
    return wait_for_capabilities(BASE_URL)


def test_capabilities_is_schema_valid(capabilities: bytes) -> None:
    validate_against_xsd(capabilities, XSD_PATH)


def test_expected_layers_present(capabilities: bytes) -> None:
    advertised = advertised_layer_names(capabilities)
    missing = EXPECTED_LAYERS - advertised
    assert not missing, f"layers missing from capabilities: {sorted(missing)}"


@pytest.mark.parametrize("layer", sorted(EXPECTED_LAYERS))
def test_required_crs_advertised(capabilities: bytes, layer: str) -> None:
    crs = advertised_crs_for(capabilities, layer)
    missing = REQUIRED_CRS - crs
    assert not missing, f"{layer} does not advertise {sorted(missing)}"


def test_axis_order_probe() -> None:
    """A WMS 1.3.0 GetMap over central London must return a real PNG, not an error.

    EPSG:4326 is latitude-first in 1.3.0, so BBOX is minLat,minLon,maxLat,maxLon.
    A server that mishandles axis order returns a ServiceException or a blank tile.
    """
    resp = requests.get(
        f"{BASE_URL}/wms",
        params={
            "SERVICE": "WMS", "VERSION": "1.3.0", "REQUEST": "GetMap",
            "LAYERS": "topp:states", "STYLES": "", "CRS": "EPSG:4326",
            "BBOX": "24.0,-125.0,50.0,-66.0",  # lat,lon per 1.3.0 axis rule
            "WIDTH": "256", "HEIGHT": "256", "FORMAT": "image/png",
        },
        timeout=30,
    )
    assert resp.status_code == 200
    assert resp.headers["Content-Type"].startswith("image/png"), resp.text[:400]


@pytest.mark.cite
def test_cite_wms_conformance() -> None:
    caps_url = f"{BASE_URL}/wms?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetCapabilities"
    counts = run_cite_wms(CITE_URL, caps_url)
    assert counts["failed"] == 0, f"CITE reported {counts['failed']} failures: {counts}"

Annotated Walkthrough

Wait-for-ready (wait_for_capabilities) — This is the single most important function for CI reliability. A GitHub Actions service container reports healthy as soon as its TCP port accepts connections, but a Java servlet container running GeoServer needs many more seconds to load its catalog and register layers. The loop retries GetCapabilities on a fixed interval until it not only gets HTTP 200 but confirms the payload parses and carries the expected WMS_Capabilities root. It swallows RequestException and XMLSyntaxError during the window — both are expected while the service warms up — and raises TimeoutError only when the deadline passes, converting a flaky hang into a deterministic failure the CI runner reports.

Schema validation (validate_against_xsd) — etree.XMLSchema compiles the OGC XSD once and validates the parsed capabilities document against it. The XSD is loaded from a local xsd_path rather than fetched over the network, so the test is hermetic and does not fail when schemas.opengis.net is slow or unreachable. On failure it surfaces the first entry from schema.error_log, including the line number, which is far more actionable than a bare boolean.

Targeted assertions (advertised_layer_names, advertised_crs_for) — Both use iterfind with the explicit WMS_NS namespace map. Forgetting the namespace is the classic mistake: an XPath without the wms: prefix silently matches nothing against a 1.3.0 document, producing a green test that asserts absolutely nothing. advertised_crs_for walks to the named layer and collects its <CRS> children so the parametrised test_required_crs_advertised can fail per-layer with a precise message.

Axis-order probe (test_axis_order_probe) — Schema validity does not prove the server renders correctly. This test issues a real GetMap with a WMS 1.3.0 latitude-first BBOX and asserts the response is an actual image/png, not a ServiceException. If the server mishandles axis order it typically returns an exception XML body with a 200 status, which the Content-Type assertion catches — the axis-order mechanics behind this are covered in the Understanding OGC Web Map Service Specifications guide.

CITE driver (run_cite_wms) — The TEAM Engine exposes each conformance suite under /rest/suites/<suite>/<version>/run and accepts the service’s capabilities URL as a query parameter. The function requests the EARL/XML report and tallies earl:outcome verdicts into passed, failed, and skipped counts. The @pytest.mark.cite marker on the test lets CI run capabilities checks alone on every push (pytest -m "not cite") and the full conformance run only on release (pytest -m cite).

Provisioning a Disposable GeoServer in CI

The service under test must be created fresh inside the pipeline so that what you certify is exactly what you deploy. A docker-compose.ci.yml brings up a throwaway GeoServer alongside the TEAM Engine:

services:
  geoserver:
    image: docker.osgeo.org/geoserver:2.25.2
    environment:
      SKIP_DEMO_DATA: "false"        # ship the sample topp:* layers for the probe
      GEOSERVER_ADMIN_PASSWORD: ci-only-password
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/geoserver/web/"]
      interval: 10s
      timeout: 5s
      retries: 20

  teamengine:
    image: ogccite/teamengine-production:latest
    ports:
      - "8081:8080"

In a real deploy you would replace SKIP_DEMO_DATA with a volume mount or an init step that applies your production catalog via the REST API — the exact provisioning calls are covered in Automating GeoServer with the Python REST API, and keeping this container identical to production is the whole subject of Environment Parity for Spatial Servers.

The GitHub Actions workflow

The workflow below fetches and caches the OGC XSD bundle, provisions the services, waits for readiness, runs the fast pytest gate, then the CITE gate, and uploads reports as artifacts. It gates the downstream deploy job on the compliance job succeeding.

name: spatial-service-ci

on:
  push:
    branches: [main]
  pull_request:

jobs:
  compliance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install test dependencies
        run: pip install pytest requests lxml pytest-xdist

      - name: Cache OGC XSD bundle
        id: xsd-cache
        uses: actions/cache@v4
        with:
          path: schemas
          key: ogc-xsd-wms-130-v1

      - name: Fetch OGC XSDs (cache miss only)
        if: steps.xsd-cache.outputs.cache-hit != 'true'
        run: |
          mkdir -p schemas
          curl -sSL https://schemas.opengis.net/SCHEMAS_OPENGIS_NET.zip -o schemas.zip
          unzip -q schemas.zip -d schemas

      - name: Start disposable services
        run: docker compose -f docker-compose.ci.yml up -d

      - name: Run fast capabilities checks
        env:
          OGC_BASE_URL: http://localhost:8080/geoserver
          OGC_XSD_PATH: schemas/wms/1.3.0/capabilities_1_3_0.xsd
        run: pytest -m "not cite" -n auto --junitxml=reports/pytest.xml

      - name: Run OGC CITE conformance
        env:
          OGC_BASE_URL: http://localhost:8080/geoserver
          CITE_BASE_URL: http://localhost:8081/teamengine
        run: pytest -m cite --junitxml=reports/cite.xml

      - name: Upload test reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: compliance-reports
          path: reports/

      - name: Tear down
        if: always()
        run: docker compose -f docker-compose.ci.yml down -v

  deploy:
    needs: compliance          # gate: deploy only runs if compliance succeeded
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - run: echo "compliance green — promoting build"

The needs: compliance key is the gate. GitHub Actions will not schedule deploy unless every step in compliance exits zero, and any red pytest assertion or non-zero CITE failure count makes the job exit non-zero. The if: always() on the artifact and teardown steps ensures reports are captured and containers are removed even when tests fail, so a red build still yields a downloadable diagnosis.

Error Handling & Edge Cases

Flaky service startup — The dominant failure mode is testing before GeoServer has loaded its catalog. Never use a fixed sleep; a value long enough to be safe on a cold runner wastes minutes on a warm one, and a value short enough to be fast is a coin flip. The wait_for_capabilities poll loop with an overall timeout is the correct pattern — it returns the instant the service is genuinely ready and fails hard, with the last error recorded, if it never is.

Capabilities returns 200 with an error body — GeoServer and MapServer both return HTTP 200 for ServiceException responses. A readiness check that only inspects the status code will treat an error document as success. The poll loop guards against this by parsing the body and checking the root element name before declaring the service ready.

XSD import resolution — OGC schemas <import> and <include> each other using relative paths. If you cache only capabilities_1_3_0.xsd without the surrounding directory tree, etree.XMLSchema raises an obscure import-resolution error. Cache the whole SCHEMAS_OPENGIS_NET tree and keep its layout intact, which is why the workflow unzips the full bundle rather than fetching a single file.

Namespace-blind XPath — An XPath that omits the wms: prefix returns an empty node set against a namespaced 1.3.0 document without raising, so an assertion like assert names passes vacuously if names was populated by a broken query. Always pass the explicit namespace map, and write assertions against a known-present sentinel layer so a silently-empty result fails loudly.

CITE run timeouts — A full conformance suite can run for minutes. Set a generous timeout on the requests call driving the TEAM Engine and a matching timeout-minutes on the GitHub Actions job, otherwise the default runner timeout or the default requests behaviour can sever the connection mid-run and report a spurious failure.

Stateful residue between runs — A GetFeature transaction test that inserts features can pollute a shared instance. Because the GeoServer here is disposable and torn down with docker compose down -v, each pipeline run starts from a clean catalog — never point conformance tests that mutate state at a shared or production endpoint.

Testing & Compliance Verification

The harness is itself the test suite, but you should verify it fails correctly — a compliance gate that cannot go red is worthless. Point OGC_BASE_URL at a deliberately broken service (rename an expected layer, remove EPSG:3857 from a datastore) and confirm the relevant assertion fails with an actionable message. A quick local smoke check:

# Bring up the disposable services
docker compose -f docker-compose.ci.yml up -d

# Wait and run only the fast checks, verbose
OGC_BASE_URL=http://localhost:8080/geoserver \
OGC_XSD_PATH=schemas/wms/1.3.0/capabilities_1_3_0.xsd \
pytest test_ogc_compliance.py -m "not cite" -v

Expected output when the sample data is present and correct:

test_ogc_compliance.py::test_capabilities_is_schema_valid PASSED
test_ogc_compliance.py::test_expected_layers_present PASSED
test_ogc_compliance.py::test_required_crs_advertised[topp:states] PASSED
test_ogc_compliance.py::test_required_crs_advertised[topp:tasmania_roads] PASSED
test_ogc_compliance.py::test_axis_order_probe PASSED

For the authoritative conformance verdict, the OGC CITE TEAM Engine produces an EARL report whose earl:outcome verdicts the run_cite_wms helper tallies; a certification-grade run requires zero failed outcomes across every mandatory WMS 1.3.0 test. The same engine ships suites for WFS 2.0 and other OGC services, so the pattern extends by changing the suite path in the run URL. Schema-validating capabilities against the cached XSD before invoking CITE is a cheap pre-filter: a document that fails the XSD will never pass CITE, and catching it in milliseconds saves a multi-minute conformance run.

Performance & Scaling Notes

Parallel test shards — The fast capabilities checks are independent and read-only, so pytest -n auto with pytest-xdist distributes them across every available CI core. Because the session-scoped capabilities fixture caches the fetched document, sharding does not multiply the number of network round-trips per worker beyond the initial fetch, keeping the fast gate under a few seconds even as the assertion count grows.

Caching the TEAM Engine image — The ogccite/teamengine-production image is large. On a cold runner, pulling it can dominate the job time. Cache it with a keyed docker save/docker load step or a registry pull-through cache so repeated runs reuse the layers; the same applies to the GeoServer image. Combined with the actions/cache step for the XSD bundle, a warm pipeline avoids all three heavy downloads.

Split the gates by trigger — Running full CITE conformance on every push does not pay for itself: the result rarely changes between commits that do not touch service configuration. Run pytest -m "not cite" on every push for a sub-minute signal, and reserve pytest -m cite for pushes to main, nightly schedules, and release tags. This keeps developer feedback fast while still certifying anything that ships.

Reuse the disposable instance across suites — Provisioning GeoServer is the slowest setup step. Bring it up once per job and run both the WMS and WFS conformance suites against the same running instance rather than tearing down and recreating between suites. Only recreate when a test mutates state that a later test must not observe.

Schema-validate before certifying — XSD validation is orders of magnitude faster than a CITE run. Ordering the pipeline so the cheap schema check runs first means a malformed capabilities document fails the job in milliseconds instead of consuming a full conformance slot, tightening the feedback loop on the most common regression.

Gotchas / Frequently Asked Questions

Why do my compliance tests pass locally but fail intermittently in CI?

Almost always a race condition: the test job connects before GeoServer finishes deploying its catalog. GitHub Actions service containers report healthy when the TCP port is open, but a Java servlet container needs many more seconds to load workspaces and register layers. Replace fixed sleep calls with a poll loop that retries GetCapabilities until it returns HTTP 200 with a valid WMS_Capabilities root element, bounded by an overall timeout that fails the job deterministically if the service never becomes ready. The wait_for_capabilities function above is exactly this pattern.

Should I run the full OGC CITE suite on every commit?

No. A full CITE TEAM Engine run for WMS 1.3.0 plus WFS 2.0 can take several minutes and is largely deterministic between commits that do not change service configuration. Run the fast pytest -m "not cite" capabilities checks on every push, and reserve full conformance certification for merges to main, nightly schedules, or release tags via pytest -m cite. Cache the TEAM Engine Docker image so repeated runs do not re-pull it.

How do I schema-validate GetCapabilities without hitting the internet in CI?

Download the official OGC XSD bundle (SCHEMAS_OPENGIS_NET.zip) once, cache it with actions/cache, and load the local capabilities_1_3_0.xsd with lxml.etree.XMLSchema. OGC schemas <import> and <include> each other using relative paths, so you must keep the whole directory tree intact rather than caching a single file — unzip the full bundle and preserve its layout. The validation then runs offline and does not fail when schemas.opengis.net is slow or unreachable.

Can I run compliance tests against a service that is not yet public?

Yes, and you should. Provision a disposable GeoServer as a docker-compose service container inside the CI job, seed it with the exact configuration your deploy will apply, and test that private instance. This certifies the build artifact before it ever reaches a shared environment — which is the entire point of gating deploys on green compliance. Tearing the container down with docker compose down -v guarantees a clean catalog for the next run.

What is the difference between pytest capabilities checks and OGC CITE certification?

pytest checks are fast, targeted assertions you author yourself: layer topp:states exists, EPSG:3857 is advertised, the XML validates against the XSD, an axis-order probe renders. OGC CITE is the official conformance suite that exhaustively exercises every mandatory operation and error path to earn a certification badge. Use pytest as the fast inner gate on every push and CITE as the authoritative outer gate on release — they catch different classes of defect and are strongest together.

---

Back to Python Automation for GeoServer & MapServer

Related:

• Running OGC CITE Tests in GitHub Actions — driving the TEAM Engine Docker image and REST API from a workflow, step by step
• Writing Pytest Checks for WMS GetCapabilities Validity — namespace-safe assertions for layer presence, CRS advertisement, and schema validity
• Automating GeoServer with the Python REST API — the REST provisioning that seeds the disposable service under test
• Environment Parity for Spatial Servers — keeping the CI service container byte-for-byte identical to production
• Understanding OGC Web Map Service Specifications — the WMS GetCapabilities contract and axis-order rules the assertions target