Architecture

Purpose and scope

dataretrieval is a Python client library for discovering and retrieving hydrologic data from several independently operated USGS and partner services. It is a modular monolith: one installable distribution with public service facades, service- and protocol-specific adapters, and shared infrastructure. The architecture favors incremental evolution and stable user-facing functions over a framework that forces unlike upstream APIs into one shape.

This document records the current structure, the rules contributors should preserve, and known variances that later changes may remove. Rationale for architecturally significant choices is recorded in Architecture Decision Records.

Architecture characteristics

The characteristics below are prioritized in descending order when trade-offs are necessary:

1. Installability and artifact integrity. A built wheel must contain every supported package and work outside a source checkout.

2. Public API compatibility. Established imports, function signatures, return shapes, metadata, warnings, and exception types should remain stable; intentional changes follow the project’s deprecation policy.

3. Correctness and data integrity. Pagination and fan-out must not silently return truncated or duplicated data after a failure.

4. Resilience. Retry, rate-limit, interruption, and resume behavior must be explicit and bounded. Capabilities may differ where upstream protocols do.

5. Maintainability. Modules should have cohesive responsibilities and dependencies should point toward stable policies rather than service details.

6. Bounded performance and resource use. Concurrency, connection pools, retries, and page accumulation require explicit limits.

7. Observability. Logs, metadata, progress, and typed failures should make remote-service behavior diagnosable without changing results.

8. Portability. Supported Python and operating-system combinations must work; optional geospatial dependencies must remain isolated.

Context view

The package sits between Python callers and remote hydrologic services:

Python user / notebook / batch process
                |
                v
    dataretrieval public service modules
                |
      HTTP(S), JSON, CSV, RDB, GeoJSON
                |
                v
USGS Water Data APIs, NGWMN, NWDC Water Use,
Water Quality Portal, NLDI, StreamStats, legacy NWIS

The remote services own their schemas, paging mechanisms, rate limits, and availability. The library adapts those differences to documented Python contracts but does not hide meaningful service-specific behavior.

Composition and dependency view

Public service facades

dataretrieval.waterdata

Modern USGS Water Data API facade. waterdata.api is a logic-free compatibility facade over collection-family modules: time_series, metadata, measurements, reference, samples, and cql. Focused modules own ratings, nearest-value selection, statistics execution, shared service policy, and type vocabularies. Internal modules import protocol helpers from their canonical OGC modules rather than re-exporting them through Water Data utilities.

dataretrieval.ngwmn

NGWMN facade. Its only OGC dependency is the public OGC facade, which it configures with an NGWMN-specific base URL, output identifiers, state translation, and OgcDialect.

dataretrieval.wateruse

NWDC Water Use facade. Builds CSV requests, follows Link headers, and uses service-neutral transport for bounded fan-out, retry, pagination, response aggregation, and synchronous dispatch. It does not depend on OGC modules.

dataretrieval.wqp, dataretrieval.nldi, and dataretrieval.streamstats

Service-specific adapters over shared synchronous HTTP and bounded retry policy. Their return types intentionally reflect their upstream data models.

dataretrieval.nwis

Deprecated legacy NWIS facade, scheduled for removal on or after 2027-05-06. Modern code must not depend on it.

Shared components

dataretrieval.ogc

Protocol subsystem for Water Data and NGWMN. A small facade (__init__.py) exposes the service-adapter seam: OgcDialect, prepare_request_args, get_ogc_data, and fetch_ogc_request. Internally, policy defines the dialect type, control validation, and endpoint constants; context owns ambient base URL, dialect, and row cap state; requests owns argument normalization and HTTP request construction; schema executes queryables/schema requests; engine supplies OGC cursor and response strategies to transport pagination; planning determines chunk boundaries; chunking connects those plans to the shared fan-out executor and retains compatibility aliases; interruptions and retry re-export their moved compatibility surfaces; and shaping, dates, filters, and errors isolate their named protocol concerns. The full runtime OGC graph, including the facade, is acyclic – enforced by the package-wide fitness function in tests/architecture_test.py.

dataretrieval.transport

Internal service-neutral execution layer. Owns guarded client lifecycle and timeouts, host-scoped authentication, cursor pagination, bounded retry, response aggregation, fan-out execution, progress integration, and sync-over-async dispatch. fanout drives an injected plan and fetch callback, owning bounded concurrency, deterministic failure precedence, sparse completion state, resume, and the progress line. It is also the one entry point from synchronous getter code into the async internals: a query with nothing to divide runs as a one-item fan-out rather than crossing a separate bridge. Internally, liveness is a stdlib-only leaf recording when data last arrived, so the page loop that observes progress and the retry loop that acts on it depend on liveness rather than on each other. Transport imports no service adapter or OGC protocol module, and is not exposed as a public framework API.

dataretrieval.interruptions

Shared resumable fan-out failure contract. It owns FanOutInterrupted and its subclasses; ChunkInterrupted remains a permanent alias for compatibility. The module is outside transport because adapters catch these errors independently of how they execute requests.

dataretrieval.exceptions

Stable error-policy leaf. It has no runtime third-party dependency, and every service can import it without creating an infrastructure cycle.

dataretrieval._response_metadata

BaseMetadata, the second half of every getter’s (DataFrame, metadata) return contract. A dependency-free leaf: nearly every service module needs this class, and while it lived in utils beside the legacy query machinery, importing it pulled that module’s whole HTTP stack in transitively. The implementation module is private; the established public class path remains dataretrieval.utils.BaseMetadata.

dataretrieval._ambient

Dependency-free implementation of scoped context values used by concurrent OGC internals. dataretrieval.utils.Ambient remains the stable public import and resolves to this same class.

dataretrieval.utils

Data-shaping helpers, plus compatibility imports for names that historically lived here (including Ambient, BaseMetadata, query and to_str, so their original import paths keep working). OGC does not depend on this legacy module; by default, do not add new service-specific behavior there.

dataretrieval._querying

The one-shot HTTP query path the single-request adapters (nwis, wqp, nldi, streamstats, wateruse) use: compose the URL, send it, map the status, retry a transient. It left utils because the two halves shared only a filename – this one depends on exceptions and transport, the shaping half on codes and pandas, and no caller wanted both. The implementation module is private; the established public function paths remain dataretrieval.utils.query and dataretrieval.utils.to_str.

dataretrieval.codes and dataretrieval.rdb

State/time-zone code conversion and RDB parsing leaves.

The intended direction is:

public facade -> service/protocol adapter -> service-neutral transport
                                         -> stable policy/infrastructure
                                         -> third-party library / network

Dependencies must not point from shared infrastructure back to a public service adapter. .importlinter declares this as a layer stack and lint-imports checks it over the transitive import graph, so a violation routed through an intermediary fails as surely as a direct one. The stack is exhaustive: a new top-level module fails the contract until it is placed, so where a module belongs is decided when it is added rather than inferred later.

tests/architecture_test.py complements those contracts without repeating their direction rules. It covers which symbols cross a seam, declared __all__ surfaces, the AST shape of a facade, imports that must exist rather than be forbidden, and full-graph cycle detection (see ADR 0003).

Interface view

The primary API is a collection of synchronous functions grouped by data portal. Most tabular download functions return (DataFrame, metadata). NLDI and StreamStats retain service-specific geospatial or response-object contracts; consistency alone is not sufficient reason for a breaking change.

Failed requests derive from dataretrieval.DataRetrievalError. Callers can inspect status_code, retry_after, and retryable without knowing the concrete subtype. A fanned-out call – an over-large OGC request, or a Water Use query naming several locations – may raise FanOutInterrupted subclasses (formerly, and still aliased as, ChunkInterrupted) carrying a resumable call handle and completed partial state.

Package/module exports and documentation define the public surface. Underscore-prefixed symbols are implementation details even where existing internal adapters currently import them; those imports are known variances, not new extension points.

Service return contracts

The library preserves meaningful upstream differences rather than forcing every service into one return shape:

• Water Data, NGWMN, and Water Use tabular getters return (DataFrame, BaseMetadata). Geometry-bearing Water Data and NGWMN results may use a GeoDataFrame in the first position when geopandas is installed. BaseMetadata carries request URL, elapsed query time, response headers, and comments where the upstream format provides them.

• WQP getters return (DataFrame, WQP_Metadata); the service-specific metadata extends BaseMetadata with WQP query parameters and site lookup.

• waterdata.get_ratings returns a mapping of feature IDs to parsed rating DataFrame objects by default, or the raw STAC feature list when downloads are disabled.

• NLDI navigation functions return GeoDataFrame objects directly, or raw GeoJSON-like dictionaries when as_json=True; they do not add a metadata tuple.

• StreamStats functions return raw httpx.Response objects or the service-specific Watershed domain object, depending on the requested format.

• Deprecated NWIS functions retain their established DataFrame and legacy metadata contracts through the published deprecation window.

Changing one of these shapes is a public compatibility change and requires the project’s deprecation process; consistency alone is not sufficient reason.

Interaction view

A typical OGC-backed call follows this sequence:

synchronous public getter
    -> normalize and validate arguments
    -> select OGC dialect and build a chunk plan
    -> enter a short-lived anyio blocking portal
    -> execute chunks through a shared httpx.AsyncClient
    -> paginate each chunk
    -> retry bounded transient failures
    -> combine and deduplicate pages/chunks
    -> shape columns and types
    -> return DataFrame and BaseMetadata

A transient failure after some chunks complete raises a resumable interruption. The retained FanOut (available as ChunkedCall on the OGC compatibility path) reissues only missing work and applies the same finalization path when resumed. Cancellation and non-transient programming errors take precedence over retry/resume wrapping.

Non-OGC services use the same transport policy only where their protocols have matching semantics. Retry, cursor pagination, and fan-out remain explicit adapter choices. Chunk planning remains OGC-specific; resumable fan-out is also used by Water Use because the NWDC accepts only one location per request.

Resource and configuration view

API_USGS_PAT

Optional USGS API token. It is attached only to requests for api.waterdata.usgs.gov. Shared synchronous and asynchronous clients re-check every redirected request and strip the token before following a link to any other host, including external rating assets.

API_USGS_CONCURRENT

Fan-out concurrency cap; defaults to 32 for OGC and 4 for Water Use when unset. An explicit value applies to every service, 1 is sequential, and unbounded removes the explicit cap. A semaphore, not pool waiting, is the execution throttle.

API_USGS_RETRIES

Number of retries after the first attempt on supported active request paths; defaults to four. Backoff is exponential with full jitter and honors bounded Retry-After values. Only failures a later attempt could survive are re-sent: 429 and gateway 5xx, not a 500 rejecting the query itself, and not a transport failure that is settled before the request leaves (unresolvable host, unsupported scheme). Deprecated NWIS compatibility paths do not opt in.

API_USGS_STALL_TIMEOUT

Seconds a call may go without receiving any data before retrying stops and the failure surfaces; defaults to 60, and 0 disables the bound. It complements API_USGS_RETRIES, which caps attempts rather than elapsed time: without this bound, four retries of a request that times out after a minute add up to four silent minutes. Progress restarts the budget — a page received, or a queued chunk acquiring its concurrency slot. Neither a slow but productive download nor the tail of a wide fan-out is cut short, and an attempt already in flight is never interrupted. This bound never withholds the first retry, so one slow attempt cannot disable retry by itself; after that, the budget decides whether to continue. A dead connection therefore costs about two read timeouts rather than five attempts’ worth.

API_USGS_PROGRESS

Controls best-effort progress display. Reporting failures must never change retrieval results.

dataretrieval.transport centralizes HTTP timeout, redirect, and authentication policy. OGC chunk fan-out and Water Use location fan-out retain separate explicit concurrency caps because their upstream costs and request shapes differ.

Known architectural debt

This view records categories and representative locations of debt. .importlinter is authoritative for exact current dependency allowlists.

• ogc/engine.py retains a compatibility pagination wrapper alongside OGC orchestration. The sync-dispatch wrapper is gone: every retrieval path now enters through transport.fanout.FanOut.

• utils.py combines shaping with compatibility imports for metadata, ambient configuration, transport, and the query path.

• waterdata/utils.py combines endpoint constants, argument normalization, and the OGC engine wrappers.

These are documented so guardrails distinguish accepted current dependencies from new erosion. They should be removed through small, test-protected changes, not a rewrite.

Change process

Architecturally significant changes should:

1. add or supersede an ADR;

2. identify affected characteristics and trade-offs;

3. add or update an executable fitness function, and the matching contract in .importlinter when the change moves a dependency boundary;

4. preserve public contracts or provide a deprecation path; and

5. update this view when component responsibilities or dependency rules change.

• Architecture Decision Records