Running OGC CITE Tests in GitHub Actions

TL;DR: Declare your service under test and the ogccite/teamengine-production container as GitHub Actions service containers, poll both until they answer HTTP 200, then run one Python step that calls TEAM Engine’s REST endpoint /teamengine/rest/suites/wms/{version}/run?capabilities-url=…, retrieves the EARL/RDF result, counts pass/fail outcomes with lxml, uploads the report as an artifact, and exits non-zero on any failure. The one trap that breaks most first attempts is networking: TEAM Engine must reach your service by its service label hostname, never localhost.

The Core Challenge

OGC CITE — Compliance, Interoperability, Testing and Evaluation — is the official conformance test programme run through TEAM Engine, a servlet application that executes suites of assertions against a live endpoint. Running it locally is a manual, click-through affair in a web UI. Wiring it into continuous integration means solving three problems at once.

First, TEAM Engine and the service it tests are two long-running network services that must both be alive before any assertion fires, and both take tens of seconds to become ready. Second, they must be able to talk to each other, which under GitHub Actions is not the same address space your workflow steps see. Third, TEAM Engine’s native output is an EARL (Evaluation and Reporting Language) RDF document, not a JUnit XML file that Actions understands natively — so you have to parse pass and fail counts yourself and translate them into a job exit code.

The endpoint under test here is a WMS, whose request contract and version quirks are covered in Understanding OGC Web Map Service Specifications; the same workflow shape drives the wfs suite by swapping the suite name and version. This page slots into the wider CI/CD and Compliance Testing for Spatial Services workflow, where cheaper, faster pytest checks for WMS GetCapabilities validity run on every push and the heavyweight CITE suite runs on a schedule or before a release.

OGC CITE run inside a GitHub Actions job Three swimlanes: the runner's Python step, the TEAM Engine service container, and the service under test. The step triggers a REST run; TEAM Engine fetches GetCapabilities from the service by its label hostname, executes assertions, and returns an EARL RDF report; the step parses outcomes, uploads an artifact, and sets the exit code. Python step (runner) TEAM Engine container Service under test poll capabilities URL until HTTP 200 GET /rest/suites/wms/1.3.0/run GET ?REQUEST=GetCapabilities WMS_Capabilities XML run assertions: GetMap, axis order… EARL / RDF report (application/rdf+xml) lxml: count outcomes upload artifact + exit code

Production-Ready Code

Two files carry the whole workflow: .github/workflows/cite.yml orchestrates the containers, and run_cite.py drives the run and parses the result. Start with the workflow. It declares two service containers, each with a Docker HEALTHCHECK, then a Python step that runs the driver script and a final step that always uploads the report.

# .github/workflows/cite.yml
name: OGC CITE Compliance

on:
  workflow_dispatch:
  schedule:
    - cron: "0 3 * * 1"   # weekly, Mondays 03:00 UTC

jobs:
  cite-wms:
    runs-on: ubuntu-latest
    timeout-minutes: 45

    services:
      # The service under test. Reachable from TEAM Engine as http://geoserver:8080/…
      geoserver:
        image: docker.osgeo.org/geoserver:2.25.2
        ports:
          - 8080:8080
        options: >-
          --health-cmd "curl -fsS http://localhost:8080/geoserver/web/ || exit 1"
          --health-interval 15s
          --health-timeout 10s
          --health-retries 30

      # TEAM Engine. Its 8080 is mapped to 8081 on the runner to avoid a clash.
      teamengine:
        image: ogccite/teamengine-production:latest
        ports:
          - 8081:8080
        options: >-
          --health-cmd "curl -fsS http://localhost:8080/teamengine/ || exit 1"
          --health-interval 15s
          --health-timeout 10s
          --health-retries 30

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install driver dependencies
        run: pip install "requests>=2.31" "lxml>=5.0"

      - name: Wait for both services to answer
        run: |
          poll() { timeout 240 bash -c "until curl -fsS \"$1\" >/dev/null; do sleep 5; done"; }
          poll "http://localhost:8080/geoserver/web/"
          poll "http://localhost:8081/teamengine/"

      - name: Run OGC CITE WMS 1.3.0 suite
        env:
          TEAMENGINE_BASE: http://localhost:8081/teamengine
          # NOTE: the hostname here is the *service label*, resolved on the
          # containers' shared network — NOT localhost, which TEAM Engine
          # would resolve to its own container.
          CAPABILITIES_URL: "http://geoserver:8080/geoserver/ows?service=WMS&version=1.3.0&request=GetCapabilities"
        run: python run_cite.py

      - name: Upload EARL report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: cite-earl-report
          path: cite-report.xml

The driver script issues the REST call, saves the raw EARL document for the artifact upload, and reduces it to pass/fail/skip counts. The parsing snippet is the summarise_earl function — it walks every earl:TestResult and buckets outcomes by the fragment of their rdf:resource URI.

# run_cite.py — trigger a TEAM Engine run and gate CI on the EARL result
import os
import sys
from collections import Counter

import requests
from lxml import etree

# TEAM Engine's default in-container credentials.
TE_USER, TE_PASS = "ogctest", "ogctest"
SUITE, VERSION = "wms", "1.3.0"
REPORT_PATH = "cite-report.xml"

EARL_NS = {
    "earl": "http://www.w3.org/ns/earl#",
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
}


def trigger_run(base_url: str, capabilities_url: str) -> bytes:
    """Call the TEAM Engine REST run endpoint and return the raw EARL bytes."""
    run_url = f"{base_url}/rest/suites/{SUITE}/{VERSION}/run"
    response = requests.get(
        run_url,
        params={"capabilities-url": capabilities_url},
        headers={"Accept": "application/rdf+xml"},
        auth=(TE_USER, TE_PASS),
        timeout=1800,  # a full suite can run for many minutes
    )
    response.raise_for_status()
    return response.content


def summarise_earl(earl_bytes: bytes) -> Counter:
    """
    Count assertion outcomes in a TEAM Engine EARL report.

    Each assertion carries an <earl:TestResult> whose <earl:outcome> points,
    via rdf:resource, at one of the EARL outcome URIs — e.g.
    http://www.w3.org/ns/earl#passed. We bucket by the URI fragment.
    """
    root = etree.fromstring(earl_bytes)
    outcome_uris = root.xpath(
        "//earl:TestResult/earl:outcome/@rdf:resource", namespaces=EARL_NS
    )
    return Counter(uri.rsplit("#", 1)[-1] for uri in outcome_uris)


def main() -> None:
    base_url = os.environ["TEAMENGINE_BASE"]
    capabilities_url = os.environ["CAPABILITIES_URL"]

    earl_bytes = trigger_run(base_url, capabilities_url)
    with open(REPORT_PATH, "wb") as handle:
        handle.write(earl_bytes)  # persisted for the upload-artifact step

    counts = summarise_earl(earl_bytes)
    passed = counts.get("passed", 0)
    failed = counts.get("failed", 0) + counts.get("cantTell", 0)
    skipped = counts.get("inapplicable", 0) + counts.get("untested", 0)

    print(f"CITE {SUITE} {VERSION}: passed={passed} failed={failed} skipped={skipped}")

    if failed:
        # GitHub Actions workflow-command annotation, then a non-zero exit.
        print(f"::error::{failed} failing CITE assertion(s) — see cite-report.xml")
        sys.exit(1)

    if passed == 0:
        print("::error::no assertions passed — the run probably never executed")
        sys.exit(1)


if __name__ == "__main__":
    main()

Step-by-Step Walkthrough

Service containers, not docker run steps. Declaring geoserver and teamengine under services lets GitHub Actions start them before the first step, attach them to a shared user-defined bridge network, and tear them down afterwards. The --health-cmd options are Docker HEALTHCHECK overrides; Actions will not run steps until each container reports healthy, but a healthy container is not yet a ready application, which is why an explicit readiness poll still follows.

Port mapping versus internal hostnames. Each service maps a container port onto the runner: GeoServer’s 8080 stays on 8080, and TEAM Engine’s 8080 is remapped to 8081 to avoid the clash. Your workflow steps run on the runner host, so they reach TEAM Engine at http://localhost:8081/teamengine. TEAM Engine itself, however, lives in a container where localhost is that container — so the capabilities-url it must dereference uses the service label geoserver as the hostname and the container port 8080. Getting this backwards is the single most common cause of a run that reports zero assertions.

The readiness gate. The poll shell function wraps curl -fsS in a timeout-bounded until loop. -f makes curl exit non-zero on HTTP 4xx/5xx, so the loop keeps polling until GeoServer has finished deploying and actually serves its web interface. GeoServer commonly needs 60 to 120 seconds; issuing the CITE run before then yields spurious GetCapabilities failures.

Triggering the run. In trigger_run, the REST path /teamengine/rest/suites/wms/1.3.0/run selects the WMS 1.3.0 suite; swapping to wfs and a WFS version reuses the entire workflow. The capabilities-url query parameter is the only mandatory input for these suites. The Accept: application/rdf+xml header is what makes TEAM Engine return a machine-parseable EARL document rather than an HTML session log, and the ogctest/ogctest basic-auth pair is the container image’s built-in test account. The 30-minute timeout reflects how long a full suite legitimately takes.

Parsing outcomes. summarise_earl is deliberately format-driven rather than string-matching. TEAM Engine emits one earl:Assertion per test, each wrapping an earl:TestResult whose earl:outcome element references an outcome URI through the rdf:resource attribute. The XPath //earl:TestResult/earl:outcome/@rdf:resource collects every outcome; uri.rsplit("#", 1)[-1] reduces http://www.w3.org/ns/earl#passed to passed; and Counter tallies them. Using lxml (rather than xml.etree) gives full XPath support including attribute-axis selection with a namespace map — the same lxml XSD-validation toolkit used across the metadata pipeline.

Gating the job. Only failed (and, conservatively, cantTell) are treated as conformance violations. inapplicable and untested are counted as skipped and never fail the build, because they correspond to optional operations the endpoint does not advertise. A guard on passed == 0 catches the silent-misconfiguration case where the run never really executed. The ::error:: prints are GitHub Actions workflow commands that surface as annotations in the run summary, and sys.exit(1) is what turns a red X on the job.

Always uploading the report. The final step uses if: always() so the EARL XML is uploaded even when the driver exits non-zero — otherwise a failing run would discard exactly the artifact you need to diagnose it.

Verification

Run the job from the Actions tab (the workflow_dispatch trigger) or push to a branch that includes the workflow. A clean pass prints a summary line to the step log:

CITE wms 1.3.0: passed=142 failed=0 skipped=17

A non-conformant endpoint fails the step and annotates the run:

CITE wms 1.3.0: passed=138 failed=4 skipped=17
::error::4 failing CITE assertion(s) — see cite-report.xml

Download the cite-earl-report artifact and confirm the failing assertions with a one-liner against the saved report:

python -c "from run_cite import summarise_earl; print(summarise_earl(open('cite-report.xml','rb').read()))"

Expected output is a Counter mirroring the step summary, for example Counter({'passed': 142, 'inapplicable': 17}) for a clean run.

Gotchas & Edge Cases

Why can't TEAM Engine reach my service at localhost inside GitHub Actions?

TEAM Engine runs inside its own service container, so localhost there means that container, not the service under test. GitHub Actions attaches every service container in a job to a shared bridge network on which each is addressable by its service label. Pass a capabilities-url of http://geoserver:8080/… (the label plus the container port), while your own workflow steps continue to reach TEAM Engine through the mapped http://localhost:8081/…. If the run returns zero assertions or an immediate capabilities error, this mismatch is almost always the cause.

How do I get an EARL report instead of an HTML test log?

Send Accept: application/rdf+xml on the REST run request, as trigger_run does. Without it, TEAM Engine may content-negotiate an HTML or plain XML session log whose structure is not stable to parse. The RDF/XML EARL representation exposes each assertion as an earl:TestResult with an earl:outcome resource of passed, failed, inapplicable, or untested, which is exactly what the lxml XPath in summarise_earl keys on.

The run times out in CI even though it passes locally. What changed?

Two separate clocks matter. Set the requests timeout high enough for a full suite — 1800 seconds is safe — and raise the job’s timeout-minutes. But most apparent timeouts are really readiness failures: GeoServer needs a minute or two to deploy before its capabilities document exists, and firing the run before then produces cascading GetCapabilities failures that look like a hang. The poll loop against the live capabilities URL, not merely the container health check, is what removes this flakiness.

Should a skipped or inapplicable assertion fail the build?

No. EARL separates failed from inapplicable and untested. Only outcomes ending in #failed — and #cantTell if you want to be strict — are conformance violations. inapplicable assertions map to optional operations your endpoint legitimately does not advertise; gating on them would fail every server that omits an optional capability. summarise_earl folds those into a skipped bucket that is reported but never affects the exit code.

Can I reuse this for WFS, and against a remote endpoint instead of a container?

Yes on both counts. Change SUITE/VERSION to wfs and the target WFS version, and adjust the capabilities-url. To test an already-deployed public endpoint you can drop the geoserver service entirely and point CAPABILITIES_URL at its full public URL — but keep TEAM Engine as a service container, and remember the URL must be reachable from inside that container, which a public HTTPS endpoint is and a private localhost service is not.

Back to CI/CD and Compliance Testing for Spatial Services

Related