ADR 0011: Configuration profiles, scoped to one adapter ======================================================== Status ------ Accepted. Supersedes two clauses of :doc:`0010-adapter-scoped-settings` -- decision 5 (adapter schemas held centrally as ``TypedDict``) and decision 8 (each adapter a named keyword on ``configure``) -- and three of :doc:`0009-layered-configuration`: the global ``[profiles.]`` table; the environment-above-file rule, inverted for a profile selected in code; and the refusal of a configuration object, which ADR 0010 had already narrowed to a preference about the payload's shape. The chain, the ``ContextVar`` delivery, host-scoped credentials and the leaf constraint stand. Context ------- ADR 0010 gave each adapter its own slice of the chain, so ``[ngwmn]`` narrows a setting to NGWMN. That covers "tune one service" but not the case a multi-service caller actually has: - **Several named configurations per adapter.** A caller with an overnight bulk shape and a polite daytime shape for Water Data cannot store both. The only named construct is ``[profiles.]``, which switches *every* service at once. - **Composing them.** The two mechanisms do not compose: ``[profiles.bulk.ngwmn]`` raises, so a profile cannot carry per-service detail. That refusal was recorded in ADR 0010 on the grounds that layering them needed a fourth precedence rule nobody had asked for. Someone has now asked for it, and it is the primary use case. Two further problems ADR 0010 left open feed into the same decision. The adapter roster is spelled in four places, only one of which is derived -- adding an adapter needs coordinated edits, and forgetting one leaves a schema no call site can reach, which happened to three adapters and shipped undetected until a fitness test was written. And a setting's definition lives in ``config`` rather than in the module that reads it, so adding a Water Data setting edits a file that knows nothing about Water Data. Decision -------- **A configuration profile is a named set of settings for one adapter.** The file gains named profiles beside each adapter's default profile:: concurrency = 16 # package-wide defaults [waterdata] # waterdata's DEFAULT profile: always active concurrency = 32 [waterdata.bulk] # a NAMED profile: only when selected parallel_chunks = 8 [ngwmn.gentle] concurrency = 4 A named profile never enters the chain unless a caller selects it. The global ``[profiles.]`` table and ``DATARETRIEVAL_PROFILE`` are retired; nothing has shipped, so nothing is deprecated. **``configure()`` takes configuration objects.** Positionally, one per adapter, and nothing else:: with dataretrieval.configure( Configuration(api_key=vault.read("usgs/pat")), WaterdataConfiguration.load("bulk"), NgwmnConfiguration(concurrency=4), ): ... The adapter an instance targets is a property of its class, so the caller never restates it -- which is what removes the roster duplication. Naming two configurations for one adapter raises: they would be the one pairing with no defined order. Keyword settings are removed, so ``configure(api_key=...)`` no longer works. This is the most-typed line the feature exists to enable, and making it wordier is a real cost, accepted deliberately for one shape everywhere. **Schemas live with their adapter; names live centrally.** ``configuration`` is a standard-library-only leaf every adapter may import, so it cannot import adapters. It holds the tuple of adapter *names*, which is what parsing a file needs (is ``[ngwmn]`` a table or a typo?). Each adapter package owns its subclass, which is what a setting's definition needs to be local to the service that reads it. Registration at import alone would not do: ``dataretrieval`` imports six of seven adapters eagerly, but NLDI is deliberately on demand for the geopandas extra, so a registry built from imports would reject a valid ``[nldi]`` table until something imported it, and the report would vary by what a caller had touched. **Precedence**, highest first: 1. A configuration instance passed to ``configure()`` 2. A profile selected in code, ``WaterdataConfiguration.load("bulk")`` 3. The setting's environment variable (package-wide settings only) 4. The adapter's default profile in the file 5. Package-wide defaults in the file 6. The adapter's built-in preference in code 7. The package built-in default Each level overrides the one below **per key**, so a named profile still inherits its adapter's default profile and the package-wide keys. Positions 1 and 2 are both code and both target one adapter, so the same-adapter rule means they cannot tie. Position 2 above 3 inverts ADR 0009's environment-above-file rule for this one case. A profile named in code is a more deliberate act than a variable inherited from a shell, and losing to that variable is the behaviour a caller would file a bug about. Everything the caller did *not* name in code still follows the original rule. **Validation is lazy.** A file's structure is checked when it is parsed; a table's keys are checked when that adapter first resolves a setting. This keeps the blast-radius rule ADR 0010 established -- a malformed ``[nldi]`` table must not fail a Water Data call -- and it is what allows the schema to live in a module the parser cannot import. **Base URLs may be configured, from code only.** An adapter's configuration may carry its base URL, settable in a ``configure()`` block and rejected from the file and the environment. A file that silently redirects a data-retrieval library to another host is a supply-chain-shaped hazard; an in-code block keeps the redirect where a reader sees it. **The module is renamed** ``dataretrieval.config`` to ``dataretrieval.configuration``, and ADR 0009's rule reserving ``config`` as an abbreviation for the module and the file is withdrawn. The path has never been released, so no alias is needed. **Credentials are unchanged, and measurement settled why.** The API key stays one package-wide setting scoped to the single host that honours it. Probing the live services: .. list-table:: :header-rows: 1 * - Host - No key - With key - Bad key * - ``api.waterdata.usgs.gov`` (waterdata, ngwmn) - no limit header - ``x-ratelimit-limit: 4000`` - 403 * - ``api.water.usgs.gov`` (nwdc) - ``1000`` - ``1000`` - 403 * - ``api.water.usgs.gov`` (nldi) - ``3600`` - ``3600`` - 403 NWDC and NLDI meter by address and report the *same* limit with or without a key; the gateway validates one only if present. Sending the key there would gain nothing and would turn a stale key into 403s on calls that work anonymously today. The three hosts also keep independent counters, so ADR 0010's "one key, one quota pool" is true of waterdata and ngwmn only. Consequences ------------ - **The multi-service case gets a spelling**, which is the point. One block, several adapters, at most one configuration each, any of them from the file or from code. - **The roster stops being duplicated.** An adapter declares itself once. The failure mode where a schema exists that nothing passes becomes impossible by construction rather than caught by a fitness test. - **A setting's definition moves next to the code that reads it.** Adding a Water Data setting no longer edits a service-neutral module. - **``configure(api_key=...)`` breaks.** The README, the configuration guide, the PR description and ADR 0009's examples all use it and all must change in the same commit. - **``show_configuration()`` can only resolve the settings an adapter accepts once that adapter has been imported.** It names the adapters it could not check rather than omitting them silently, which is the honest cost of lazy validation. The *profile list* is not import-limited: what a profile is called is a fact about the file, so every ``[.]`` table it defines is listed, imported or not -- withholding one would make the section's answer depend on which optional extras happened to be installed. - **Two names differ only by case** -- the ``configuration`` module and the ``Configuration`` class. The module stays out of the package's public exports so the confusing import line cannot arise. - **Separate quota pools are still not modelled.** Three exist. Nothing in the library needs to know yet. - **``ssl_check`` is unaffected** and remains a per-call argument, for the reasons in ADR 0010. Compliance ---------- Satisfied. In ``tests/configuration_test.py``: - ``test_several_named_profiles_are_selected_independently`` -- one block, a different profile per adapter. - ``test_a_named_profile_layers_per_key_over_the_tiers_below`` -- a profile inherits its adapter's default profile and the package-wide keys per key. - ``test_adding_a_named_profile_changes_nothing_until_it_is_selected`` -- a named profile is inert until something selects it. - ``test_two_configurations_for_one_adapter_raise``. - ``test_a_code_selected_profile_outranks_the_environment``, plus a case per rung of the seven-rung ladder above, each written against one file that populates every rung with a distinct value. - ``test_inner_block_can_lower_a_setting_an_outer_block_scoped`` -- the innermost block wins, including over an adapter-scoped outer one. - ``test_a_table_for_an_unimported_adapter_stays_valid`` and ``test_a_malformed_table_does_not_fail_another_adapters_call`` -- the blast-radius rule under lazy validation. - ``test_base_url_applies_from_code_and_is_refused_from_the_file``, with ``test_base_url_is_refused_from_the_environment`` for the other source, and ``test_a_code_base_url_redirects_every_water_data_endpoint_family`` -- one public-getter contract covering OGC, Samples, Statistics, and Ratings. The endpoint module exposes request-time acquisition functions rather than raw usable endpoints, so a family module gets the active scoped root without a wrapper obligation at every use site. - ``test_adapter_roster_names_real_modules_that_register_themselves`` and ``test_every_adapter_is_actually_wired_to_a_read_site`` -- the roster resolves, and no configuration exists that nothing reads. An adapter name the code does not recognize now raises out of ``_resolve`` rather than falling through to the package-wide value, so the grep is a backstop rather than the only guard. ``tests/architecture_test.py::test_config_is_a_standard_library_only_leaf`` asserts the module imports no adapter -- ``dataretrieval.exceptions`` is its only first-party import -- and ``lint-imports`` keeps ``configuration`` below ``credentials``. Notes ----- - Live measurements taken 2026-08-11 against ``api.waterdata.usgs.gov`` and ``api.water.usgs.gov``. - Open, not decided here: whether ``parallel_chunks`` is renamed. ``fan_out`` was suggested and conflicts with the glossary, where fan-out is *executing* chunks concurrently -- which ``concurrency`` already governs -- while ``parallel_chunks`` asks the planner to *divide* more finely. ADR 0009 rejected ``parallelism`` and ``chunk_parallelism`` for the same conflation. ``chunk_count`` or ``target_chunks`` would stay on the correct side of it.