dataretrieval.exceptions

Exception taxonomy for dataretrieval.

Every service module (nwis, wqp, nldi, waterdata, streamstats) raises a subclass of DataRetrievalError when a request fails, so one except dataretrieval.DataRetrievalError catches them all. That includes connection-level failures (timeouts, DNS, refused connections), which remain inside this taxonomy rather than leaking httpx exceptions. A deterministic failure is NetworkError; a recoverable failure that exhausts retries during fan-out is a resumable ServiceInterrupted.

Most failures are an HTTPError carrying the response .status_code, of which TransientError (429 / 5xx) is the retryable subset. The rest aren’t a plain status: RequestTooLarge (with URLTooLong / Unchunkable), NetworkError (a failed connection, per above), NoSitesError, and ConfigurationError for an unusable setting. error_for_status() maps a status to its type. ConfigurationError is the one member that is not a request failure at all: it reports an unusable setting or config file, raised from wherever a setting is first resolved – which, because resolution is lazy, is inside whichever getter runs first. The warning side of the taxonomy lives here too: SkippedItemWarning (specialized by SkippedRatingWarning) for a per-item skip inside a batched retrieval, and DataCurrencyWarning for an upstream dataset that has stopped being updated.

This module has no third-party runtime dependencies – httpx is imported only for type checking. Any module can therefore import it without pulling in pandas or httpx, and without risking an import cycle.

exception dataretrieval.exceptions.ConfigurationError[source]

Bases: DataRetrievalError, ValueError

A dataretrieval setting holds a value that can’t be used, so no request was issued – an environment variable, a policy field, a malformed config.toml, or a profile the file does not define.

It is a DataRetrievalError so except around a retrieval catches it rather than letting a bare ValueError escape a request path. That matters because settings resolve lazily, on the request path: a broken config file surfaces from inside whichever getter runs first, and belongs in the same handler as any other failure of that call. It is also a ValueError, so code that already treats a bad setting as one keeps working whether the value came from the environment, a file, or a dataretrieval.configure() block.

exception dataretrieval.exceptions.DataCurrencyWarning[source]

Bases: UserWarning

An upstream dataset is frozen, retired, or no longer updated.

Distinct from DeprecationWarning, which promises that a name in this package is going away and gives the caller something to migrate to. Here the API is fine and there is nothing to migrate: the service’s own data has stopped moving, and only the caller can judge whether that matters.

It is a UserWarning for that reason. Emitting it as a DeprecationWarning meant a downstream project running -W error::DeprecationWarning – ordinary CI hygiene – could not call the affected getters with their default arguments at all.

exception dataretrieval.exceptions.DataRetrievalError[source]

Bases: Exception

Base class for every dataretrieval error.

Almost every member is a failed request, and the read-anywhere fields below describe one. The exception is ConfigurationError, which reports a configuration the library cannot use; it appears here because configuration is resolved lazily on the request path, so it surfaces from inside a getter and one except DataRetrievalError should cover it too. It carries no status and is not retryable, so the branching idiom below routes it to the final raise.

Catch it to handle any USGS or EPA service failure uniformly, and branch on the read-anywhere fields below without needing the concrete subclass:

try:
    df, md = dataretrieval.waterdata.get_daily(...)
except dataretrieval.DataRetrievalError as e:
    if e.retryable:  # 429 / 5xx / connection failure
        time.sleep(e.retry_after or backoff)
        ...  # re-issue the request
    elif e.status_code == 404:  # ``None`` unless an HTTP status error
        ...
    else:
        raise

Connection-level failures (timeouts, DNS) remain subclasses of this base: NetworkError when deterministic, or a resumable ServiceInterrupted when recoverable fan-out retries are exhausted.

retry_after: float | None = None: Seconds the server asked us to wait before retrying (its Retry-After header), or None when it gave no hint. Set by TransientError.

retryable: ClassVar[bool] = False: Whether re-issuing the same request might succeed – True for the transient HTTP statuses (429 / 5xx, TransientError) and for connection failures (NetworkError); False otherwise.

status_code: int | None = None: HTTP status that triggered the error, or None for errors without one (connection failure, too-long URL, no data). Set by HTTPError.

exception dataretrieval.exceptions.HTTPError(message: str, *, status_code: int)[source]

Bases: DataRetrievalError

The service returned an error HTTP status.

The numeric status is on status_code; branch on it, e.g. except HTTPError as e: ... if e.status_code == 404. TransientError (429 / 5xx) is the retryable subset, and is itself an HTTPError. The one exception to “a status is an HTTPError” is a request the service rejects as too long: it surfaces as URLTooLong (a RequestTooLarge), not an HTTPError. Catch DataRetrievalError to be certain of spanning every failure. See error_for_status() for the full mapping.

Parameters:

message (str) – Human-readable error message.
status_code (int) – The HTTP status the service returned.

exception dataretrieval.exceptions.NetworkError[source]

Bases: DataRetrievalError

The request never completed a round-trip to the service.

A DNS failure, refused connection, or timeout stopped it, so no HTTP response arrived to classify.

Wraps the underlying httpx transport exception, preserved on __cause__. Worth retrying (retryable is True), but carries no .status_code because no response came back.

retryable: ClassVar[bool] = True: Whether re-issuing the same request might succeed – True for the transient HTTP statuses (429 / 5xx, TransientError) and for connection failures (NetworkError); False otherwise.

exception dataretrieval.exceptions.NoSitesError(url: httpx.URL)[source]

Bases: DataRetrievalError

A request succeeded (HTTP 200) but matched no sites/data.

A no-data result is normally not an error: the modern getters (waterdata, wqp, nldi) return an empty DataFrame. Only the deprecated nwis (waterservices) path still raises this.

exception dataretrieval.exceptions.RateLimited(message: str, *, status_code: int | None = None, retry_after: float | None = None)[source]

Bases: TransientError

A request was rejected with HTTP 429 (too many requests).

_DEFAULT_STATUS: ClassVar[int] = 429: Canonical status a concrete transient stamps when built without an explicit status_code (RateLimited = 429, ServiceUnavailable = 503). TransientError itself is abstract and sets none, so constructing it bare requires status_code.

exception dataretrieval.exceptions.RequestTooLarge[source]

Bases: DataRetrievalError

The request is too large for the service to satisfy.

Base for the two ways that happens; catch it to handle either: URLTooLong (a single request rejected for length) and Unchunkable (a Water Data call the chunker could not split small enough to fit).

exception dataretrieval.exceptions.ServiceUnavailable(message: str, *, status_code: int | None = None, retry_after: float | None = None)[source]

Bases: TransientError

A request was rejected with a server error (HTTP 5xx).

Raised by both the legacy query path and the Water Data path, so a 5xx surfaces as one type whichever subsystem issued the request. .status_code holds the actual 5xx; it falls back to 503 only on a bare hand-construction.

_DEFAULT_STATUS: ClassVar[int] = 503: Canonical status a concrete transient stamps when built without an explicit status_code (RateLimited = 429, ServiceUnavailable = 503). TransientError itself is abstract and sets none, so constructing it bare requires status_code.

exception dataretrieval.exceptions.SkippedItemWarning[source]

Bases: UserWarning

One item of a batched retrieval was skipped; the rest were returned.

The policy for batch getters whose items are independent documents: an item that fails deterministically – so retrying would reproduce the failure – is dropped from the result under a warning naming it, because aborting would discard every other item’s data over one bad entry. Transient failures (429 / 5xx / timeouts / connection drops) are never skipped – they are retried and, if retries run out, raised as a resumable interruption. Rate limiting in particular is systematic, so skipping there would silently drop most of a batch; that silent loss is the failure mode this policy exists to prevent.

A warning rather than a log line so it is visible by default. To make any skip fatal (strict all-or-nothing behavior):

warnings.filterwarnings("error", category=SkippedItemWarning)

Getters emit a subclass naming their surface (e.g. SkippedRatingWarning), so a filter can also target one getter.

exception dataretrieval.exceptions.SkippedRatingWarning[source]

Bases: SkippedItemWarning

A rating feature was skipped by dataretrieval.waterdata.get_ratings().

Emitted when a single STAC feature fails deterministically – a stale catalog entry (404 on its data asset), a feature carrying no data asset, a malformed RDB file. The failed feature’s id is absent from the returned dict. See SkippedItemWarning for the policy and how to escalate a skip to an error.

exception dataretrieval.exceptions.TransientError(message: str, *, status_code: int | None = None, retry_after: float | None = None)[source]

Bases: HTTPError

A 429 or 5xx the server may serve on a later try.

RateLimited covers 429 and ServiceUnavailable covers 5xx.

This only classifies the condition; it does not itself retry. Whether to retry is up to the calling path: a single-shot request raises it for the caller to handle (e.g. wait retry_after seconds, then re-issue), while the Water Data chunker retries and resumes automatically.

Parameters:

message (str) – Human-readable error message.
status_code (int, optional) – The HTTP status the service returned. Defaults to the leaf’s canonical code (429 / 503) when omitted; error_for_status() always passes the real status.
retry_after (float, optional) – Seconds to wait before retrying, parsed from the Retry-After response header; None when the header is absent or unparseable.

_DEFAULT_STATUS: ClassVar[int]: Canonical status a concrete transient stamps when built without an explicit status_code (RateLimited = 429, ServiceUnavailable = 503). TransientError itself is abstract and sets none, so constructing it bare requires status_code.

retryable: ClassVar[bool] = True: Whether re-issuing the same request might succeed – True for the transient HTTP statuses (429 / 5xx, TransientError) and for connection failures (NetworkError); False otherwise.

exception dataretrieval.exceptions.URLTooLong[source]

Bases: RequestTooLarge

A single request URL was too long for the service.

Raised on the legacy query path (which sends one un-chunked request), whether the URL is rejected client-side before sending or by the server (see error_for_status()). Remediation: query fewer sites, or split the call manually.

exception dataretrieval.exceptions.Unchunkable[source]

Bases: RequestTooLarge

No chunking plan fits the URL byte limit.

Raised by the Water Data chunker when even the smallest reducible plan (every list axis at one atom per chunk, the filter at one clause per chunk) still exceeds the server’s byte limit. Unlike URLTooLong, then, automatic splitting has already been tried and exhausted. Shrink the input lists, simplify the filter, or split the call manually.

dataretrieval.exceptions.error_for_status(status: int, message: str, *, retry_after: float | None = None) → DataRetrievalError[source]

Return the typed DataRetrievalError for an HTTP error status.

The one status-to-type mapping every request path shares (the legacy query path, waterdata, streamstats), so a given status becomes the same type everywhere:

413, 414 -> URLTooLong (a RequestTooLarge) – the “too long” semantic is more actionable than a bare status, and it matches the client-side over-long-URL case
429 -> RateLimited
5xx -> ServiceUnavailable
anything else -> HTTPError

message is used verbatim; retry_after is attached only to the transient (TransientError) types. status must be an error status (>= 400) – classifying a success or redirect is a usage error and raises ValueError.

dataretrieval.exceptions.parse_retry_after(value: str | None) → float | None[source]

Parse a Retry-After header into seconds, or None for no usable hint.

Both header forms mean the same thing and are treated the same way: the seconds are returned as given, however large. A value past what a caller will wait out inline stops the retry and surfaces a transient carrying the hint on .retry_after, so a long wait becomes the caller’s decision (and, for a chunked call, a resumable interruption) instead of being ignored.

An over-long hint is honored rather than discarded. Dropping it would make the client retry harder against a service that just asked for a long pause, and would deny the caller the number it needs on .retry_after. Clock skew can inflate a date-form hint, but trusting one costs a recoverable escalation while ignoring it costs hammering a service that is already asking for room.

A date that has already passed yields no hint at all rather than 0.0. Read literally it says “retry now”, but the likelier reading is that our clock runs ahead of the server’s – and acting on it would re-send almost immediately against a service that just asked for a pause. Falling back to our own bounded backoff is right under either reading. (Delta-seconds is clock-independent, so a literal Retry-After: 0 is still honored as the instruction it is, floored by backoff()’s jitter.)

Resumable fan-out interruptions

These are raised when a fanned-out request is interrupted mid-stream; the completed work is preserved and exc.call.resume() continues it. They are defined in dataretrieval.interruptions (they carry pandas/httpx state) but are importable from the top level, e.g. from dataretrieval import FanOutInterrupted.

ChunkInterrupted is a permanent alias of FanOutInterrupted – the same class object under the name it was first published as – so except ChunkInterrupted and except FanOutInterrupted are the same handler. The base class is named for the fan-out rather than for chunking because a Water Use call fans out without dividing anything: the NWDC simply accepts one location per request.

class dataretrieval.FanOutInterrupted(*, completed_chunks: int, total_chunks: int, call: FanOut[Any] | None = None, retry_after: float | None = None, cause: BaseException | None = None)[source]

Bases: DataRetrievalError

Base class for mid-stream chunk failures whose completed work is preserved and resumable.

A FanOutInterrupted subclass means: a chunk failed, but FanOut still owns whatever completed successfully before the failure. Call self.call.resume() to pick up where the failure stopped you — only still-pending chunks are re-issued.

Subclasses describe why FanOut stopped so callers can pick a retry policy: QuotaExhausted for 429 (wait for the rate-limit window), ServiceInterrupted for 5xx (wait for the upstream to recover). The .call handle is the same object across every interruption of a single fanned-out call — frames accumulate across retries.

call

Resumable handle into the FanOut that raised this exception. None only on hand-constructed exceptions (test fixtures), where .call-derived accessors degrade to empty/None.

Type:: FanOut or None

retry_after

Seconds the server suggested waiting (Retry-After header). None when the server gave no hint.

Type:: float or None

completed_chunks

Number of chunks successfully completed before the failure.

Type:: int

total_chunks

Total chunks in the plan.

Type:: int

partial_frame

Combined frame of work completed by the moment this exception was raised. Snapshot at raise time — does NOT advance on a later call.resume() (use exc.call.partial_frame for the live view).

Type:: pandas.DataFrame

partial_response

Raw aggregate response covering the completed chunks at raise time; None if nothing had completed yet. Same snapshot semantics as partial_frame. (Raw, not finalized — use exc.call.resume() for the finalized (df, metadata) result.)

Type:: httpx.Response or None

Examples

Retry on any transient interruption, honoring the server’s Retry-After hint when present and falling back to a fixed wait otherwise. Each new interruption keeps the already-completed work intact — only the still-pending chunks are re-issued.

import time
from dataretrieval import ChunkInterrupted

# ``getter`` is any chunked OGC getter — e.g.
# ``waterdata.get_daily`` or ``ngwmn.get_water_level``.
try:
    df, md = getter(monitoring_location_id=long_list_of_sites)
except ChunkInterrupted as exc:
    while True:
        time.sleep(exc.retry_after or 5 * 60)
        try:
            df, md = exc.call.resume()
            break
        except ChunkInterrupted as next_exc:
            exc = next_exc

_format_message(completed_chunks: int, total_chunks: int, cause: BaseException | None) → str[source]: Build the exception message from the template, appending cause info.

_resolve_status_code(cause: BaseException | None) → int | None[source]: Resolve the HTTP status code from the class default or cause chain.

retryable: ClassVar[bool] = True: Whether re-issuing the same request might succeed – True for the transient HTTP statuses (429 / 5xx, TransientError) and for connection failures (NetworkError); False otherwise.

class dataretrieval.QuotaExhausted(*, completed_chunks: int, total_chunks: int, call: FanOut[Any] | None = None, retry_after: float | None = None, cause: BaseException | None = None)[source]

Bases: FanOutInterrupted

A chunk returned HTTP 429 — the per-key rate-limit window is exhausted. Subclass of FanOutInterrupted.

The completed chunks are preserved on .call; once the rate-limit window resets, .call.resume() re-issues only the still-pending work. partial_frame holds what completed before the 429.

class dataretrieval.ServiceInterrupted(*, completed_chunks: int, total_chunks: int, call: FanOut[Any] | None = None, retry_after: float | None = None, cause: BaseException | None = None)[source]

Bases: FanOutInterrupted

A chunk returned HTTP 5xx — the upstream service failed transiently. Subclass of FanOutInterrupted.

The completed chunks are preserved on .call; once the upstream recovers, .call.resume() resumes only the still-pending work.