dataretrieval.configuration

Layered configuration: a dataretrieval.configure(...) block holding one configuration per adapter, then the API_USGS_* environment variables, then the adapter’s table and the package-wide keys in ~/.dataretrieval/config.toml, then built-in defaults. A [<adapter>.<name>] table is a named profile, selected in code with <Adapter>Configuration.load("<name>"). See the configuration guide for the settings and worked examples.

Layered configuration resolution for dataretrieval.

Every setting – the Water Data API key, the fan-out concurrency cap, the retry count, and the progress line – resolves through one ordered chain so a caller never has to mutate os.environ to configure a single call.

Sources, highest precedence first:

A configuration passed to configure() – delivered through a ContextVar, so a setting applies to the current thread or asyncio task and cannot leak into another one.
The environment variable for that setting (API_USGS_PAT, API_USGS_CONCURRENT, API_USGS_RETRIES, API_USGS_PROGRESS).
The configuration file (TOML): ~/.dataretrieval/config.toml, or the path in DATARETRIEVAL_CONFIG. Top-level keys are the package-wide defaults; a [<adapter>] table is that adapter’s default profile, always in effect; a [<adapter>.<name>] table is a named profile, inert until a caller selects it with <Adapter>Configuration.load("<name>").
The built-in default.

Each of those four sources is one branch of _resolve(); ADR 0011 lists the same order in finer grain, as seven rungs – three of these sources hold two apiece. An adapter’s own default is not one of the four: a read site such as concurrency() passes it in as the default argument.

Precedence applies per setting, not per source: an environment that sets only API_USGS_PAT leaves a file-provided concurrency fully in effect. The environment ranks above the file (ADR 0009). ADR 0011 makes one exception: a profile named in code enters the chain through configure(), above the environment.

A caller configures by passing configuration objects, at most one per adapter:

with dataretrieval.configure(
    Configuration(api_key=vault.read("usgs/pat")),
    WaterdataConfiguration.load("bulk"),
    NgwmnConfiguration(concurrency=4),
):
    ...

Settings are scoped per adapter (ADR 0010): a [ngwmn] table in the file, or an NgwmnConfiguration, applies to NGWMN calls and no others. Precedence stays source-major (ADR 0010): an adapter-scoped value outranks a package-wide one only within the same source, and within the block source the innermost block decides.

Each adapter declares the settings it accepts on its own BaseConfiguration subclass, defined in the module that reads them (ADR 0011). The API key is not among them – it belongs to the gateway fronting a host, not to an adapter (ADR 0010).

This module is a leaf: it imports only the standard library plus the Python 3.10 tomli backport (ADR 0009). That is also why it holds the adapter names but never imports an adapter – see ADAPTERS. It centralizes each setting’s parser while retaining legacy environment behavior and stricter validation for the new Python/TOML surfaces.

class dataretrieval.configuration.BaseConfiguration[source]

Bases: object

A named set of settings for one adapter – a configuration profile.

Subclasses declare the settings their adapter reads as fields, and set adapter to that adapter’s module name. Every field is optional, so an empty configuration is legal and one can be built up conditionally.

Frozen, because a configuration is a value: two with the same settings are interchangeable, and one already passed to configure() must not change while the block that entered it is active.

Values are checked when the configuration is constructed, so a typo raises where it was written rather than at a later with statement or inside a request.

_label(name: str) → str[source]: How one of this configuration’s settings is named in an error.

_provenance() → str[source]

How show_configuration() reports a value this supplied.

The profile is named in the file’s own spelling – [waterdata.bulk] – so the report names the profile that set a value rather than only the block, and the name can be searched for in the file that defines it. A configuration written in code has no profile, so it names its adapter alone; the package-wide one narrows to nothing and names neither.

adapter: ClassVar[str | None] = None: The adapter this configuration targets, by the name of the module a caller imports. None on the package-wide Configuration, which every adapter reads. A ClassVar, not a field: the adapter is a property of the class, which is what stops the caller restating it at every call site and stops the roster being listed twice.

classmethod load(profile: str) → _C[source]

Read a named profile for this adapter from the configuration file.

[<adapter>.<profile>]. Only the keys that table names are included, so the profile still inherits the adapter’s default profile and the package-wide keys per setting from the rungs below.

Selecting a profile the file does not define raises: a name a caller just typed is a typo to report, not a fall-through to settings they did not ask for.

Parameters:: profile (str) – The name after the adapter, so [waterdata.bulk] is "bulk".
Returns:: An instance of the class it was called on, recording the profile it was read from so show_configuration() can name it.
Return type:: BaseConfiguration

profile: ClassVar[str | None] = None

The named profile these settings were read from, or None for a configuration written in code. Provenance rather than a setting: it records where the values came from, which is what lets show_configuration() name the profile that supplied each value instead of reporting every block alike.

A ClassVar shadowed per instance by load(), so it is neither a field nor part of equality – two configurations with the same settings stay interchangeable however each was written, which is what “a configuration is a value” means.

classmethod settings() → frozenset[str][source]: The setting names this configuration accepts.

validate() → None[source]

Check rules that span more than one setting.

Does nothing by default. Per-setting grammar is in this module’s parsers and is shared with the file and the environment, so a value means the same thing whichever source wrote it; override this only for a rule no single setting can express.

values() → dict[str, Any][source]

The settings supplied, omitting those left unset.

An omitted setting inherits from an outer block or a lower source; an explicit None suppresses them. Distinguishing the two is what the _UNSET default exists for, so it is done here rather than by every reader.

Bases: BaseConfiguration

Settings that apply to every adapter.

The package-wide profile: adapter stays None, so nothing narrows and every adapter reads what this sets unless its own configuration, or a block nested inside, overrides that setting.

Parameters:

api_key (str, optional) – Water Data API key, sent as X-Api-Key and only to api.waterdata.usgs.gov. Prefer reading it from a secret store, the environment, or the configuration file over writing a literal into a script. Pass None to make a call without an ambient key.
concurrency (int or str, optional) – Cap on simultaneous sub-requests: a positive integer, or "unbounded" to disable the cap.
retries (int, optional) – Retries attempted after a transient failure; 0 disables retrying.
progress (bool or str, optional) – Whether to draw the progress line. None leaves the automatic behavior (on for a TTY or Jupyter kernel, off otherwise).
parallel_chunks (int, optional) – Default optional fan-out for multi-value queries. It limits extra refinement, but URL-byte safety may already require more sub-requests. Sets the baseline that dataretrieval.parallel_chunks() overrides per call. Each sub-request spends rate-limit quota, so raise it only for pulls you know are large.
stall_timeout (float, optional) – Seconds a call may go without receiving any data before retrying stops and the failure is raised. Bounds the wall-clock cost of a dead connection, which retries does not – it counts attempts, not seconds. Progress resets the timer; 0 disables the bound.

Examples

with dataretrieval.configure(Configuration(api_key=vault.read("usgs"))):
    df, md = waterdata.get_daily(monitoring_location_id="USGS-05114000")

dataretrieval.configuration.config_path() → Path[source]

Path to the configuration file, applying DATARETRIEVAL_CONFIG.

Memoized on the raw DATARETRIEVAL_CONFIG value, because this is on the per-request path via api_key(). Returning a stable object also lets _load_file() check its cache by identity instead of re-normalizing a new Path.

Returns:: The explicit path from DATARETRIEVAL_CONFIG if set, otherwise ~/.dataretrieval/config.toml. The file need not exist.
Return type:: pathlib.Path

dataretrieval.configuration.configure(*configurations: BaseConfiguration) → Iterator[None][source]

Apply configuration profiles for the duration of a with block.

This is the highest-precedence source. Takes configuration objects positionally, at most one per adapter, and nothing else:

with dataretrieval.configure(
    Configuration(api_key=secrets["usgs"]),
    WaterdataConfiguration.load("bulk"),
    NgwmnConfiguration(concurrency=4),
):
    df, md = waterdata.get_daily(monitoring_location_id=sites)

The adapter a configuration targets is a property of its class, so the caller never restates it – which is what keeps the caller from restating the adapter roster at every call site. Naming two configurations for one adapter raises: they are the one pairing with no defined order between them.

Because the block is delivered through a ContextVar, a value set here applies to the current thread and to asyncio tasks started inside the block, and cannot leak into another thread, task, or unrelated call the way os.environ does – which is what makes it safe for a server or notebook handling several users’ credentials at once.

Blocks nest and merge per setting: an inner block that sets only concurrency keeps the outer block’s api_key, and an adapter configuration in an outer block is overridden by a package-wide value set by a block nested inside it, so the innermost block always decides.

Parameters:: *configurations (BaseConfiguration) – A package-wide Configuration and/or one configuration per adapter, in any order. Each adapter’s class is defined in that adapter’s module – WaterdataConfiguration in dataretrieval.waterdata, NgwmnConfiguration in dataretrieval.ngwmn, and so on.
Yields:: None
Raises:: ConfigurationError – If an argument is not a configuration, or two of them target the same adapter. Raised on entry, before any request. An invalid value raises earlier still, where the configuration was constructed.

Examples

# credentials from a secret store, no environment mutation
with dataretrieval.configure(
    Configuration(api_key=vault.read("usgs/pat"))
):
    df, md = waterdata.get_daily(monitoring_location_id="USGS-05114000")

# a large overnight pull, from a [waterdata.bulk] table in the file
with dataretrieval.configure(WaterdataConfiguration.load("bulk")):
    df, md = waterdata.get_daily(monitoring_location_id=many_sites)