dataretrieval.configuration

Layered configuration: a dataretrieval.configure(...) block holding one configuration per adapter, then the API_USGS_* environment variables, then the adapter’s table and the package-wide keys in ~/.dataretrieval/config.toml, then built-in defaults. A [<adapter>.<name>] table is a named profile, selected in code with <Adapter>Configuration.load("<name>"). See the configuration guide for the settings and worked examples.

Layered configuration resolution for dataretrieval.

Every tunable setting – the Water Data API key, the fan-out concurrency cap, the retry count, and the progress line – resolves through one ordered chain so a caller never has to mutate os.environ to configure a single call.

Sources, highest precedence first:

A configuration passed to configure() – delivered through a ContextVar, so a setting applies to the current thread or asyncio task and cannot leak into another one.
The environment variable for that setting (API_USGS_PAT, API_USGS_CONCURRENT, API_USGS_RETRIES, API_USGS_PROGRESS).
The configuration file (TOML): ~/.dataretrieval/config.toml, or the path in DATARETRIEVAL_CONFIG. Top-level keys are the package-wide defaults; a [<adapter>] table is that adapter’s default profile, always in effect; a [<adapter>.<name>] table is a named profile, inert until a caller selects it with <Adapter>Configuration.load("<name>").
The built-in default.

Those are the four sources, which is the decomposition this module is built around – one branch each in _resolve(). ADR 0011 states the same order as seven rungs by splitting three of them into the scopes inside: source 1 into a configuration instance and a selected profile, which cannot disagree because both name one adapter and two configurations for one adapter raise; source 3 into the [<adapter>] table above the top-level keys; and source 4 into an adapter’s own built-in preference above the package default. That last scope is invisible here because this module never supplies it – it arrives as the default a read site like concurrency() passes for its own service.

Precedence applies per setting, not per source: an environment that sets only API_USGS_PAT leaves a file-provided concurrency fully in effect. Putting the environment above the file follows common deployment conventions and keeps the original environment-variable interface authoritative (see ADR 0009) – with one exception ADR 0011 carves out: a profile named in code is a more deliberate act than a variable inherited from a shell, and a profile reaches the chain by being passed to configure(), which is above the environment.

A caller configures by passing configuration objects, at most one per adapter:

with dataretrieval.configure(
    Configuration(api_key=vault.read("usgs/pat")),
    WaterdataConfiguration.load("bulk"),
    NgwmnConfiguration(concurrency=4),
):
    ...

Settings are scoped per adapter (ADR 0010): a [ngwmn] table in the file, or an NgwmnConfiguration, applies to NGWMN calls and no others, so one block can be gentle with one service while leaving the rest alone. Precedence stays source-major: the chain still walks block, then environment, then file, and an adapter-scoped value outranks a package-wide one only within the same source. So a variable exported for one run still beats a stale adapter table. Within the block source that tie-break applies per block: an adapter configuration outranks a package-wide value set by the same configure call, while a value set by a block nested inside it wins over both, so the innermost block still decides.

Which settings an adapter accepts is its own vocabulary – concurrency means nothing to an adapter that issues one request – so each adapter declares them on its own BaseConfiguration subclass, defined in the module that reads them. The API key is not among them: it belongs to the gateway fronting a host, which Water Data and NGWMN share.

This module is a leaf: it imports only the standard library plus the Python 3.10 tomli backport, so any module can depend on it without an import cycle or pulling in httpx or pandas. That is also why it holds the adapter names but never imports an adapter – see ADAPTERS. It centralizes each setting’s parser while retaining legacy environment behavior and stricter validation for the new Python/TOML surfaces.

class dataretrieval.configuration.BaseConfiguration[source]

Bases: object

A named set of settings for one adapter – a configuration profile.

Subclasses declare the settings their adapter reads as fields, and set adapter to that adapter’s module name. Every field is optional, so an empty configuration is legal and one can be built up conditionally.

Frozen, because a configuration is a value: two with the same settings are interchangeable, and one already handed to configure() must not change under the block that entered it.

Values are checked when the configuration is constructed, so a typo raises where it was written rather than at a later with statement or, worse, inside a request.

_provenance() → str[source]

How show_configuration() reports a value this supplied.

The profile is named in the file’s own spelling – [waterdata.bulk] – so the report answers “which profile set this?” rather than only “a block did”, and the answer is greppable in the file that holds it. A configuration written in code has no profile, so it names its adapter alone; the package-wide one narrows to nothing and names neither.

_source(name: str) → str[source]: How one of this configuration’s settings is named in an error.

adapter: ClassVar[str | None] = None: The adapter this configuration targets, by the name of the module a caller imports. None on the package-wide Configuration, which every adapter reads. A ClassVar, not a field: the adapter is a property of the class, which is what stops the caller restating it at every call site and stops the roster being spelled twice.

classmethod load(profile: str) → _C[source]

Read a named profile for this adapter from the configuration file.

[<adapter>.<profile>]. Only the keys that table names are carried, so the profile still inherits the adapter’s default profile and the package-wide keys per setting from the tiers below.

Selecting a profile the file does not define raises: a name a caller just typed is a typo worth reporting, not a silent fall-through to settings they did not ask for.

Parameters:: profile (str) – The name after the adapter, so [waterdata.bulk] is "bulk".
Returns:: An instance of the class it was called on, remembering the profile it was read from so show_configuration() can name it.
Return type:: BaseConfiguration

profile: ClassVar[str | None] = None

The named profile these settings were read from, or None for a configuration written in code. Provenance rather than a setting: it records where the values came from, which is what lets show_configuration() name the profile that supplied each value instead of reporting every block alike.

A ClassVar shadowed per instance by load(), so it is neither a field nor part of equality – two configurations carrying the same settings stay interchangeable however each was spelled, which is what “a configuration is a value” means.

classmethod settings() → frozenset[str][source]: The setting names this configuration accepts.

validate() → None[source]

Check rules that span more than one setting.

Does nothing by default. Per-setting grammar lives in this module’s parsers and is shared with the file and the environment, so a value means the same thing whichever source wrote it; override this only for a rule no single setting can express.

values() → dict[str, Any][source]

The settings actually supplied, omitting those left unset.

An omitted setting inherits from an outer block or a lower source; an explicit None suppresses them. Distinguishing the two is the whole job of the _UNSET default, so it is done here rather than by every reader.

Bases: BaseConfiguration

Settings that apply to every adapter.

The package-wide profile: adapter stays None, so nothing narrows and every adapter reads what this sets unless its own configuration, or a block nested inside, overrides that setting.

Parameters:

api_key (str, optional) – Water Data API key, sent as X-Api-Key and only ever to api.waterdata.usgs.gov. Prefer reading it from a secret store, the environment, or the configuration file over writing a literal into a script. Pass None to make a call without an ambient key.
concurrency (int or str, optional) – Cap on simultaneous sub-requests: a positive integer, or "unbounded" to disable the cap.
retries (int, optional) – Retries attempted after a transient failure; 0 disables retrying.
progress (bool or str, optional) – Whether to draw the progress line. None leaves the automatic behavior (on for a TTY or Jupyter kernel, off otherwise).
parallel_chunks (int, optional) – Default optional fan-out for multi-value queries. It limits extra refinement, but URL-byte safety may already require more sub-requests. Sets the baseline that dataretrieval.parallel_chunks() overrides per call. Each sub-request spends rate-limit quota, so raise it only for pulls you know are large.
stall_timeout (float, optional) – Seconds a call may go without receiving any data before retrying stops and the failure surfaces. Bounds the wall-clock cost of a dead connection, which retries does not – it counts attempts, not seconds. Progress resets the clock; 0 disables the bound.

Examples

with dataretrieval.configure(Configuration(api_key=vault.read("usgs"))):
    df, md = waterdata.get_daily(monitoring_location_id="USGS-05114000")

dataretrieval.configuration.config_path() → Path[source]

Path to the configuration file, honoring DATARETRIEVAL_CONFIG.

Memoized on the raw DATARETRIEVAL_CONFIG value, because this sits on the per-request path via api_key() and building the default costs more than the stat it leads to (Path.home() alone dominates the whole resolution). Returning a stable object also lets _load_file() check its cache by identity instead of re-normalizing a fresh Path.

Returns:: The explicit path from DATARETRIEVAL_CONFIG if set, otherwise ~/.dataretrieval/config.toml. The file need not exist.
Return type:: pathlib.Path

dataretrieval.configuration.configure(*configurations: BaseConfiguration) → Iterator[None][source]

Apply configuration profiles for the duration of a with block.

The highest-precedence source. Takes configuration objects positionally, at most one per adapter, and nothing else:

with dataretrieval.configure(
    Configuration(api_key=secrets["usgs"]),
    WaterdataConfiguration.load("bulk"),
    NgwmnConfiguration(concurrency=4),
):
    df, md = waterdata.get_daily(monitoring_location_id=sites)

The adapter a configuration targets is a property of its class, so the caller never restates it – which is what keeps the adapter roster from being spelled once per call site. Naming two configurations for one adapter raises: they are the one pairing with no defined order between them.

Because the block is delivered through a ContextVar, a value set here applies to the current thread and to asyncio tasks started inside the block, and cannot leak into another thread, task, or unrelated call the way os.environ does – which is what makes it safe for a server or notebook handling several users’ credentials at once.

Blocks nest and merge per setting: an inner block that sets only concurrency keeps the outer block’s api_key, and an adapter configuration in an outer block loses to a package-wide value set by a block nested inside it, so the innermost block always decides.

Parameters:: *configurations (BaseConfiguration) – A package-wide Configuration and/or one configuration per adapter, in any order. Each adapter’s class lives in that adapter’s module – WaterdataConfiguration in dataretrieval.waterdata, NgwmnConfiguration in dataretrieval.ngwmn, and so on.
Yields:: None
Raises:: ConfigurationError – If an argument is not a configuration, or two of them target the same adapter. Raised on entry, before any request. A bad value raises earlier still, where the configuration was constructed.

Examples

# credentials from a secret store, no environment mutation
with dataretrieval.configure(
    Configuration(api_key=vault.read("usgs/pat"))
):
    df, md = waterdata.get_daily(monitoring_location_id="USGS-05114000")

# a big overnight pull, from a [waterdata.bulk] table in the file
with dataretrieval.configure(WaterdataConfiguration.load("bulk")):
    df, md = waterdata.get_daily(monitoring_location_id=many_sites)