Python Equivalents to R Vignette Examples
[1]:
from dataretrieval import nwis, waterdata, wqp
The dataretrieval Python package was created as an equivalent to the R dataRetrieval package.
The following shows Python equivalents for the methods outlined in the R dataRetrieval vignette, with the equivalent R code in comments.
[2]:
"""
{r getSite, echo=TRUE, eval=FALSE}
siteNumbers <- c("01491000","01645000")
siteINFO <- readNWISsite(siteNumbers)
"""
siteNumbers = ["USGS-01491000", "USGS-01645000"]
siteINFO, md = waterdata.get_monitoring_locations(
monitoring_location_id=siteNumbers, skip_geometry=True
)
Retrieving: monitoring-locations · 1 page · 2 rows
No API key detected — register for higher rate limits at https://api.waterdata.usgs.gov/signup/
[3]:
"""
# Continuing from the previous example:
# This pulls out just the daily, mean data:
dailyDataAvailable <- whatNWISdata(siteNumbers,
service="dv", statCd="00003")
"""
dailyDataAvailable, md = waterdata.get_time_series_metadata(
monitoring_location_id=siteNumbers, statistic_id="00003", skip_geometry=True
)
Retrieving: time-series-metadata · 1 page · 11 rows
[4]:
"""
# Choptank River near Greensboro, MD:
siteNumber <- "01491000"
parameterCd <- "00060" # Discharge
startDate <- "2009-10-01"
endDate <- "2012-09-30"
discharge <- readNWISdv(siteNumber, parameterCd, startDate, endDate)
"""
# Choptank River near Greensboro, MD:
siteNumber = "USGS-01491000"
parameterCd = "00060" # Discharge
discharge, md = waterdata.get_daily(
monitoring_location_id=siteNumber,
parameter_code=parameterCd,
time="2009-10-01/2012-09-30",
)
Retrieving: daily · 1 page · 1,096 rows
[5]:
"""
siteNumber <- "01491000"
parameterCd <- c("00010","00060") # Temperature and discharge
statCd <- c("00001","00003") # Maximum and mean
startDate <- "2012-01-01"
endDate <- "2012-05-01"
temperatureAndFlow <- readNWISdv(siteNumber, parameterCd, startDate, endDate, statCd=statCd)
"""
siteNumber = "USGS-01491000"
parameterCd = ["00010", "00060"] # Temperature and discharge
statisticId = ["00001", "00003"] # Maximum and mean
temperatureAndFlow, md = waterdata.get_daily(
monitoring_location_id=siteNumber,
parameter_code=parameterCd,
statistic_id=statisticId,
time="2012-01-01/2012-05-01",
)
---------------------------------------------------------------------------
RateLimited Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/fanout.py:612, in FanOut._run.<locals>.track(index, item)
611 """One chunk (with retry) + result-store + progress tick."""
--> 612 result = await _retry(
613 lambda: self.fetch(item), self.retry_policy, gate=semaphore
614 )
615 self._chunks[index] = result
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/retry.py:294, in retry_async(afn, policy, gate)
293 try:
--> 294 return await attempt_once()
295 except Exception as exc: # noqa: BLE001 - re-raised unless retryable
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/retry.py:290, in retry_async.<locals>.attempt_once()
289 credit_wait(time.monotonic() - started)
--> 290 return await afn()
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/engine.py:422, in _fetch_once(args, build_request, geopd, include_geometry, row_cap)
421 req = build_request(**args)
--> 422 return await _walk_pages(
423 geopd=geopd,
424 req=req,
425 include_geometry=include_geometry,
426 row_cap=row_cap,
427 )
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/engine.py:203, in _walk_pages(geopd, req, client, include_geometry, row_cap)
201 return await sess.request(method, cursor, headers=headers, content=content)
--> 203 return await paginate(
204 req,
205 parse_response=functools.partial(
206 _ogc_parse_response,
207 geopd=geopd,
208 include_geometry=include_geometry,
209 ),
210 follow_up=follow_up,
211 client=client,
212 raise_for_status=_raise_for_non_200,
213 row_cap=row_cap,
214 )
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/pagination.py:106, in paginate(initial_req, parse_response, follow_up, raise_for_status, client, row_cap)
105 response = await session.send(initial_req)
--> 106 raise_for_status(response)
107 initial_response = response
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/errors.py:135, in _raise_for_non_200(resp)
134 return
--> 135 raise error_for_status(
136 status,
137 _error_body(resp) + _url_suffix(resp),
138 retry_after=_parse_retry_after(resp.headers.get("Retry-After")),
139 )
RateLimited: 429: Too many requests made. Please obtain an API token or try again later. (URL: https://api.waterdata.usgs.gov/ogcapi/v0/collections/daily/items?monitoring_location_id=USGS-01491000¶meter_code=00010%2C00060&statistic_id=00001%2C00003&time=2012-01-01%2F2012-05-01&limit=50000)
The above exception was the direct cause of the following exception:
QuotaExhausted Traceback (most recent call last)
Cell In[5], line 14
10 siteNumber = "USGS-01491000"
11 parameterCd = ["00010", "00060"] # Temperature and discharge
12 statisticId = ["00001", "00003"] # Maximum and mean
13
---> 14 temperatureAndFlow, md = waterdata.get_daily(
15 monitoring_location_id=siteNumber,
16 parameter_code=parameterCd,
17 statistic_id=statisticId,
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/waterdata/time_series.py:256, in get_daily(monitoring_location_id, parameter_code, statistic_id, properties, time_series_id, daily_id, approval_status, unit_of_measure, qualifier, value, last_modified, skip_geometry, time, bbox, limit, filter, filter_lang, convert_type, max_rows, **queryables)
253 # Build argument dictionary, omitting None values
254 args = _get_args(locals(), exclude={"max_rows"})
--> 256 return get_ogc_data(args, collection, max_rows=max_rows)
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/waterdata/utils.py:240, in get_ogc_data(args, collection, output_id, max_rows, cql_body, spatial)
238 if output_id is None:
239 output_id = _OUTPUT_ID_BY_COLLECTION[collection]
--> 240 return _facade_get_ogc_data(
241 args,
242 collection,
243 output_id,
244 max_rows=max_rows,
245 # Endpoint acquisition resolves the active ContextVar at request time;
246 # the documented ``OGC_API_URL`` constant remains the default-value
247 # compatibility path rather than a production request destination.
248 base_url=ogc_api_url(),
249 spatial=spatial,
250 extra_id_cols=_EXTRA_ID_COLS,
251 dialect=WATERDATA_DIALECT,
252 cql_body=cql_body,
253 # Which settings table these calls read. Declared here, in the one
254 # wrapper every Water Data getter goes through, rather than derived
255 # from ``base_url``: NGWMN is served from the same host, so a URL
256 # cannot tell the two adapters apart (ADR 0010).
257 adapter="waterdata",
258 )
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/engine.py:396, in get_ogc_data(args, collection, output_id, base_url, spatial, max_rows, extra_id_cols, dialect, cql_body, adapter)
391 run = chunking.multi_value_chunked(build_request=build_request, adapter=adapter)(
392 fetch
393 )
394 # No progress block here: the executor that emits the events owns the line
395 # (see :meth:`~dataretrieval.transport.fanout.FanOut.resume`).
--> 396 return run(args, finalize=finalize)
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/chunking.py:285, in multi_value_chunked.<locals>.decorator.<locals>.wrapper(args, finalize)
271 retry_policy = RetryPolicy.from_configuration(adapter=adapter)
272 # The concurrency cap is resolved inside ``resume()`` through the
273 # configuration chain; ``1`` is a sequential gather,
274 # ``total <= 1`` a one-element gather — no special branch.
275 return ChunkedCall(
276 plan,
277 fetch,
278 retry_policy,
279 finalize,
280 canonical_url=plan.canonical_url,
281 # The collection name, for the progress line the executor
282 # opens. ``get_ogc_data`` puts it in ``args``.
283 service=args.get("collection"),
284 adapter=adapter,
--> 285 ).resume()
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/fanout.py:525, in FanOut.resume(self)
517 concurrency = _configuration.concurrency(
518 self.default_concurrent, adapter=self.adapter
519 )
520 with start_blocking_portal() as portal:
521 # ``portal.call`` returns ``Any`` because ``functools.partial``
522 # erases ``_run``'s return type; restore the declared tuple.
523 return cast(
524 "tuple[pd.DataFrame, Any]",
--> 525 portal.call(functools.partial(self._run, concurrency)),
526 )
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/anyio/from_thread.py:338, in BlockingPortal.call(self, func, *args)
323 def call(
324 self,
325 func: Callable[[Unpack[PosArgsT]], Awaitable[T_Retval] | T_Retval],
326 *args: Unpack[PosArgsT],
327 ) -> T_Retval:
328 """
329 Call the given function in the event loop thread.
330
(...) 336
337 """
--> 338 return cast(T_Retval, self.start_task_soon(func, *args).result())
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/concurrent/futures/_base.py:460, in Future.result(self, timeout)
458 raise CancelledError()
459 elif self._state == FINISHED:
--> 460 return self.__get_result()
461 else:
462 raise TimeoutError()
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/concurrent/futures/_base.py:402, in Future.__get_result(self)
400 if self._exception is not None:
401 try:
--> 402 raise self._exception
403 finally:
404 # Break a reference cycle with the exception in self._exception
405 self = None
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/anyio/from_thread.py:263, in BlockingPortal._call_func(self, func, args, kwargs, future)
261 with CancelScope() as scope:
262 future.add_done_callback(callback)
--> 263 retval = await retval_or_awaitable
264 else:
265 retval = retval_or_awaitable
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/fanout.py:663, in FanOut._run(self, max_concurrent)
660 if interrupted is None:
661 # Unreachable: classified as transient just above.
662 raise self._normalize_failure(first_transient)
--> 663 raise interrupted from first_transient
665 return self.finalize(*self._combine_raw())
QuotaExhausted: HTTP 429 after 0/1 chunks; catch QuotaExhausted (or FanOutInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many requests made. Please obtain an API token or try again later. (URL: https://api.waterdata.usgs.gov/ogcapi/v0/collections/daily/items?monitoring_location_id=USGS-01491000¶meter_code=00010%2C00060&statistic_id=00001%2C00003&time=2012-01-01%2F2012-05-01&limit=50000)
[6]:
"""
parameterCd <- "00060" # Discharge
startDate <- "2012-05-12"
endDate <- "2012-05-13"
dischargeUnit <- readNWISuv(siteNumber, parameterCd, startDate, endDate)
"""
siteNumber = "USGS-01491000"
parameterCd = "00060" # Discharge
dischargeUnit, md = waterdata.get_continuous(
monitoring_location_id=siteNumber,
parameter_code=parameterCd,
time="2012-05-12/2012-05-13",
)
---------------------------------------------------------------------------
RateLimited Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/fanout.py:612, in FanOut._run.<locals>.track(index, item)
611 """One chunk (with retry) + result-store + progress tick."""
--> 612 result = await _retry(
613 lambda: self.fetch(item), self.retry_policy, gate=semaphore
614 )
615 self._chunks[index] = result
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/retry.py:294, in retry_async(afn, policy, gate)
293 try:
--> 294 return await attempt_once()
295 except Exception as exc: # noqa: BLE001 - re-raised unless retryable
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/retry.py:290, in retry_async.<locals>.attempt_once()
289 credit_wait(time.monotonic() - started)
--> 290 return await afn()
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/engine.py:422, in _fetch_once(args, build_request, geopd, include_geometry, row_cap)
421 req = build_request(**args)
--> 422 return await _walk_pages(
423 geopd=geopd,
424 req=req,
425 include_geometry=include_geometry,
426 row_cap=row_cap,
427 )
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/engine.py:203, in _walk_pages(geopd, req, client, include_geometry, row_cap)
201 return await sess.request(method, cursor, headers=headers, content=content)
--> 203 return await paginate(
204 req,
205 parse_response=functools.partial(
206 _ogc_parse_response,
207 geopd=geopd,
208 include_geometry=include_geometry,
209 ),
210 follow_up=follow_up,
211 client=client,
212 raise_for_status=_raise_for_non_200,
213 row_cap=row_cap,
214 )
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/pagination.py:106, in paginate(initial_req, parse_response, follow_up, raise_for_status, client, row_cap)
105 response = await session.send(initial_req)
--> 106 raise_for_status(response)
107 initial_response = response
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/errors.py:135, in _raise_for_non_200(resp)
134 return
--> 135 raise error_for_status(
136 status,
137 _error_body(resp) + _url_suffix(resp),
138 retry_after=_parse_retry_after(resp.headers.get("Retry-After")),
139 )
RateLimited: 429: Too many requests made. Please obtain an API token or try again later. (URL: https://api.waterdata.usgs.gov/ogcapi/v0/collections/continuous/items?monitoring_location_id=USGS-01491000¶meter_code=00060&time=2012-05-12%2F2012-05-13&limit=50000)
The above exception was the direct cause of the following exception:
QuotaExhausted Traceback (most recent call last)
Cell In[6], line 10
6 """
7 siteNumber = "USGS-01491000"
8 parameterCd = "00060" # Discharge
9
---> 10 dischargeUnit, md = waterdata.get_continuous(
11 monitoring_location_id=siteNumber,
12 parameter_code=parameterCd,
13 time="2012-05-12/2012-05-13",
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/waterdata/time_series.py:454, in get_continuous(monitoring_location_id, parameter_code, statistic_id, properties, time_series_id, continuous_id, approval_status, unit_of_measure, qualifier, value, last_modified, time, limit, filter, filter_lang, convert_type, max_rows, **queryables)
451 # Build argument dictionary, omitting None values
452 args = _get_args(locals(), exclude={"max_rows"})
--> 454 return get_ogc_data(args, collection, max_rows=max_rows)
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/waterdata/utils.py:240, in get_ogc_data(args, collection, output_id, max_rows, cql_body, spatial)
238 if output_id is None:
239 output_id = _OUTPUT_ID_BY_COLLECTION[collection]
--> 240 return _facade_get_ogc_data(
241 args,
242 collection,
243 output_id,
244 max_rows=max_rows,
245 # Endpoint acquisition resolves the active ContextVar at request time;
246 # the documented ``OGC_API_URL`` constant remains the default-value
247 # compatibility path rather than a production request destination.
248 base_url=ogc_api_url(),
249 spatial=spatial,
250 extra_id_cols=_EXTRA_ID_COLS,
251 dialect=WATERDATA_DIALECT,
252 cql_body=cql_body,
253 # Which settings table these calls read. Declared here, in the one
254 # wrapper every Water Data getter goes through, rather than derived
255 # from ``base_url``: NGWMN is served from the same host, so a URL
256 # cannot tell the two adapters apart (ADR 0010).
257 adapter="waterdata",
258 )
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/engine.py:396, in get_ogc_data(args, collection, output_id, base_url, spatial, max_rows, extra_id_cols, dialect, cql_body, adapter)
391 run = chunking.multi_value_chunked(build_request=build_request, adapter=adapter)(
392 fetch
393 )
394 # No progress block here: the executor that emits the events owns the line
395 # (see :meth:`~dataretrieval.transport.fanout.FanOut.resume`).
--> 396 return run(args, finalize=finalize)
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/chunking.py:285, in multi_value_chunked.<locals>.decorator.<locals>.wrapper(args, finalize)
271 retry_policy = RetryPolicy.from_configuration(adapter=adapter)
272 # The concurrency cap is resolved inside ``resume()`` through the
273 # configuration chain; ``1`` is a sequential gather,
274 # ``total <= 1`` a one-element gather — no special branch.
275 return ChunkedCall(
276 plan,
277 fetch,
278 retry_policy,
279 finalize,
280 canonical_url=plan.canonical_url,
281 # The collection name, for the progress line the executor
282 # opens. ``get_ogc_data`` puts it in ``args``.
283 service=args.get("collection"),
284 adapter=adapter,
--> 285 ).resume()
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/fanout.py:525, in FanOut.resume(self)
517 concurrency = _configuration.concurrency(
518 self.default_concurrent, adapter=self.adapter
519 )
520 with start_blocking_portal() as portal:
521 # ``portal.call`` returns ``Any`` because ``functools.partial``
522 # erases ``_run``'s return type; restore the declared tuple.
523 return cast(
524 "tuple[pd.DataFrame, Any]",
--> 525 portal.call(functools.partial(self._run, concurrency)),
526 )
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/anyio/from_thread.py:338, in BlockingPortal.call(self, func, *args)
323 def call(
324 self,
325 func: Callable[[Unpack[PosArgsT]], Awaitable[T_Retval] | T_Retval],
326 *args: Unpack[PosArgsT],
327 ) -> T_Retval:
328 """
329 Call the given function in the event loop thread.
330
(...) 336
337 """
--> 338 return cast(T_Retval, self.start_task_soon(func, *args).result())
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/concurrent/futures/_base.py:460, in Future.result(self, timeout)
458 raise CancelledError()
459 elif self._state == FINISHED:
--> 460 return self.__get_result()
461 else:
462 raise TimeoutError()
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/concurrent/futures/_base.py:402, in Future.__get_result(self)
400 if self._exception is not None:
401 try:
--> 402 raise self._exception
403 finally:
404 # Break a reference cycle with the exception in self._exception
405 self = None
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/anyio/from_thread.py:263, in BlockingPortal._call_func(self, func, args, kwargs, future)
261 with CancelScope() as scope:
262 future.add_done_callback(callback)
--> 263 retval = await retval_or_awaitable
264 else:
265 retval = retval_or_awaitable
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/fanout.py:663, in FanOut._run(self, max_concurrent)
660 if interrupted is None:
661 # Unreachable: classified as transient just above.
662 raise self._normalize_failure(first_transient)
--> 663 raise interrupted from first_transient
665 return self.finalize(*self._combine_raw())
QuotaExhausted: HTTP 429 after 0/1 chunks; catch QuotaExhausted (or FanOutInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many requests made. Please obtain an API token or try again later. (URL: https://api.waterdata.usgs.gov/ogcapi/v0/collections/continuous/items?monitoring_location_id=USGS-01491000¶meter_code=00060&time=2012-05-12%2F2012-05-13&limit=50000)
[7]:
"""
# Dissolved Nitrate parameter codes:
parameterCd <- c("00618","71851")
startDate <- "1985-10-01"
endDate <- "2012-09-30"
dfLong <- read_USGS_samples(monitoringLocationIdentifier=sprintf("USGS-%s", siteNumber),
usgsPCode=parameterCd, activityStartDateLower=startDate, activityStartDateUpper=endDate)
"""
siteNumber = "USGS-01491000"
parameterCd = ["00618", "71851"]
dfLong, md = waterdata.get_samples(
monitoringLocationIdentifier=siteNumber,
usgsPCode=parameterCd,
activityStartDateLower="1985-10-01",
activityStartDateUpper="2012-09-30",
)
/tmp/ipykernel_6675/2092726829.py:12: DeprecationWarning: The 'activityStartDateLower' argument is deprecated and will be removed from `dataretrieval` in a future release; use 'activity_start_date_lower' instead.
dfLong, md = waterdata.get_samples(
/tmp/ipykernel_6675/2092726829.py:12: DeprecationWarning: The 'activityStartDateUpper' argument is deprecated and will be removed from `dataretrieval` in a future release; use 'activity_start_date_upper' instead.
dfLong, md = waterdata.get_samples(
/tmp/ipykernel_6675/2092726829.py:12: DeprecationWarning: The 'usgsPCode' argument is deprecated and will be removed from `dataretrieval` in a future release; use 'usgs_pcode' instead.
dfLong, md = waterdata.get_samples(
/tmp/ipykernel_6675/2092726829.py:12: DeprecationWarning: The 'monitoringLocationIdentifier' argument is deprecated and will be removed from `dataretrieval` in a future release; use 'monitoring_location_id' instead.
dfLong, md = waterdata.get_samples(
[8]:
"""
siteNumber <- '01594440'
peakData <- readNWISpeak(siteNumber)
"""
peakData, md = waterdata.get_peaks(
monitoring_location_id="USGS-01594440", parameter_code="00060"
)
---------------------------------------------------------------------------
RateLimited Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/fanout.py:612, in FanOut._run.<locals>.track(index, item)
611 """One chunk (with retry) + result-store + progress tick."""
--> 612 result = await _retry(
613 lambda: self.fetch(item), self.retry_policy, gate=semaphore
614 )
615 self._chunks[index] = result
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/retry.py:294, in retry_async(afn, policy, gate)
293 try:
--> 294 return await attempt_once()
295 except Exception as exc: # noqa: BLE001 - re-raised unless retryable
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/retry.py:290, in retry_async.<locals>.attempt_once()
289 credit_wait(time.monotonic() - started)
--> 290 return await afn()
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/engine.py:422, in _fetch_once(args, build_request, geopd, include_geometry, row_cap)
421 req = build_request(**args)
--> 422 return await _walk_pages(
423 geopd=geopd,
424 req=req,
425 include_geometry=include_geometry,
426 row_cap=row_cap,
427 )
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/engine.py:203, in _walk_pages(geopd, req, client, include_geometry, row_cap)
201 return await sess.request(method, cursor, headers=headers, content=content)
--> 203 return await paginate(
204 req,
205 parse_response=functools.partial(
206 _ogc_parse_response,
207 geopd=geopd,
208 include_geometry=include_geometry,
209 ),
210 follow_up=follow_up,
211 client=client,
212 raise_for_status=_raise_for_non_200,
213 row_cap=row_cap,
214 )
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/pagination.py:106, in paginate(initial_req, parse_response, follow_up, raise_for_status, client, row_cap)
105 response = await session.send(initial_req)
--> 106 raise_for_status(response)
107 initial_response = response
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/errors.py:135, in _raise_for_non_200(resp)
134 return
--> 135 raise error_for_status(
136 status,
137 _error_body(resp) + _url_suffix(resp),
138 retry_after=_parse_retry_after(resp.headers.get("Retry-After")),
139 )
RateLimited: 429: Too many requests made. Please obtain an API token or try again later. (URL: https://api.waterdata.usgs.gov/ogcapi/v0/collections/peaks/items?monitoring_location_id=USGS-01594440¶meter_code=00060&limit=50000)
The above exception was the direct cause of the following exception:
QuotaExhausted Traceback (most recent call last)
Cell In[8], line 5
1 """
2 siteNumber <- '01594440'
3 peakData <- readNWISpeak(siteNumber)
4 """
----> 5 peakData, md = waterdata.get_peaks(
6 monitoring_location_id="USGS-01594440", parameter_code="00060"
7 )
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/waterdata/measurements.py:368, in get_peaks(monitoring_location_id, parameter_code, time_series_id, unit_of_measure, time, last_modified, water_year, year, month, day, peak_since, properties, skip_geometry, bbox, limit, filter, filter_lang, convert_type, max_rows, **queryables)
364 collection = "peaks"
366 args = _get_args(locals(), exclude={"max_rows"})
--> 368 return get_ogc_data(args, collection, max_rows=max_rows)
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/waterdata/utils.py:240, in get_ogc_data(args, collection, output_id, max_rows, cql_body, spatial)
238 if output_id is None:
239 output_id = _OUTPUT_ID_BY_COLLECTION[collection]
--> 240 return _facade_get_ogc_data(
241 args,
242 collection,
243 output_id,
244 max_rows=max_rows,
245 # Endpoint acquisition resolves the active ContextVar at request time;
246 # the documented ``OGC_API_URL`` constant remains the default-value
247 # compatibility path rather than a production request destination.
248 base_url=ogc_api_url(),
249 spatial=spatial,
250 extra_id_cols=_EXTRA_ID_COLS,
251 dialect=WATERDATA_DIALECT,
252 cql_body=cql_body,
253 # Which settings table these calls read. Declared here, in the one
254 # wrapper every Water Data getter goes through, rather than derived
255 # from ``base_url``: NGWMN is served from the same host, so a URL
256 # cannot tell the two adapters apart (ADR 0010).
257 adapter="waterdata",
258 )
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/engine.py:396, in get_ogc_data(args, collection, output_id, base_url, spatial, max_rows, extra_id_cols, dialect, cql_body, adapter)
391 run = chunking.multi_value_chunked(build_request=build_request, adapter=adapter)(
392 fetch
393 )
394 # No progress block here: the executor that emits the events owns the line
395 # (see :meth:`~dataretrieval.transport.fanout.FanOut.resume`).
--> 396 return run(args, finalize=finalize)
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/chunking.py:285, in multi_value_chunked.<locals>.decorator.<locals>.wrapper(args, finalize)
271 retry_policy = RetryPolicy.from_configuration(adapter=adapter)
272 # The concurrency cap is resolved inside ``resume()`` through the
273 # configuration chain; ``1`` is a sequential gather,
274 # ``total <= 1`` a one-element gather — no special branch.
275 return ChunkedCall(
276 plan,
277 fetch,
278 retry_policy,
279 finalize,
280 canonical_url=plan.canonical_url,
281 # The collection name, for the progress line the executor
282 # opens. ``get_ogc_data`` puts it in ``args``.
283 service=args.get("collection"),
284 adapter=adapter,
--> 285 ).resume()
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/fanout.py:525, in FanOut.resume(self)
517 concurrency = _configuration.concurrency(
518 self.default_concurrent, adapter=self.adapter
519 )
520 with start_blocking_portal() as portal:
521 # ``portal.call`` returns ``Any`` because ``functools.partial``
522 # erases ``_run``'s return type; restore the declared tuple.
523 return cast(
524 "tuple[pd.DataFrame, Any]",
--> 525 portal.call(functools.partial(self._run, concurrency)),
526 )
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/anyio/from_thread.py:338, in BlockingPortal.call(self, func, *args)
323 def call(
324 self,
325 func: Callable[[Unpack[PosArgsT]], Awaitable[T_Retval] | T_Retval],
326 *args: Unpack[PosArgsT],
327 ) -> T_Retval:
328 """
329 Call the given function in the event loop thread.
330
(...) 336
337 """
--> 338 return cast(T_Retval, self.start_task_soon(func, *args).result())
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/concurrent/futures/_base.py:460, in Future.result(self, timeout)
458 raise CancelledError()
459 elif self._state == FINISHED:
--> 460 return self.__get_result()
461 else:
462 raise TimeoutError()
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/concurrent/futures/_base.py:402, in Future.__get_result(self)
400 if self._exception is not None:
401 try:
--> 402 raise self._exception
403 finally:
404 # Break a reference cycle with the exception in self._exception
405 self = None
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/anyio/from_thread.py:263, in BlockingPortal._call_func(self, func, args, kwargs, future)
261 with CancelScope() as scope:
262 future.add_done_callback(callback)
--> 263 retval = await retval_or_awaitable
264 else:
265 retval = retval_or_awaitable
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/fanout.py:663, in FanOut._run(self, max_concurrent)
660 if interrupted is None:
661 # Unreachable: classified as transient just above.
662 raise self._normalize_failure(first_transient)
--> 663 raise interrupted from first_transient
665 return self.finalize(*self._combine_raw())
QuotaExhausted: HTTP 429 after 0/1 chunks; catch QuotaExhausted (or FanOutInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many requests made. Please obtain an API token or try again later. (URL: https://api.waterdata.usgs.gov/ogcapi/v0/collections/peaks/items?monitoring_location_id=USGS-01594440¶meter_code=00060&limit=50000)
[9]:
"""
ratingData <- readNWISrating(siteNumber, "base")
attr(ratingData, "RATING")
"""
# get_ratings returns a dict keyed by "<id>.<file_type>.rdb"
ratings_data = waterdata.get_ratings(
monitoring_location_id="USGS-01594440", file_type="base"
)
list(ratings_data.keys())
Retrieving: ratings · 1 page · 1 rows
---------------------------------------------------------------------------
RateLimited Traceback (most recent call last)
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/fanout.py:612, in FanOut._run.<locals>.track(index, item)
611 """One chunk (with retry) + result-store + progress tick."""
--> 612 result = await _retry(
613 lambda: self.fetch(item), self.retry_policy, gate=semaphore
614 )
615 self._chunks[index] = result
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/retry.py:294, in retry_async(afn, policy, gate)
293 try:
--> 294 return await attempt_once()
295 except Exception as exc: # noqa: BLE001 - re-raised unless retryable
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/retry.py:290, in retry_async.<locals>.attempt_once()
289 credit_wait(time.monotonic() - started)
--> 290 return await afn()
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/waterdata/ratings.py:385, in _download_all.<locals>.fetch(feature)
384 try:
--> 385 df, response = await _fetch_rating(feature, file_path)
386 except (DataRetrievalError, LookupError, ValueError) as e:
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/waterdata/ratings.py:337, in _fetch_rating(feature, file_path)
336 response = await session.get(href, headers=headers)
--> 337 _raise_for_non_200(response)
339 if file_path is not None:
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/ogc/errors.py:135, in _raise_for_non_200(resp)
134 return
--> 135 raise error_for_status(
136 status,
137 _error_body(resp) + _url_suffix(resp),
138 retry_after=_parse_retry_after(resp.headers.get("Retry-After")),
139 )
RateLimited: 429: Too many requests made. Please obtain an API token or try again later. (URL: https://api.waterdata.usgs.gov/stac-files/ratings/USGS.01594440.base.rdb)
The above exception was the direct cause of the following exception:
QuotaExhausted Traceback (most recent call last)
Cell In[9], line 6
2 ratingData <- readNWISrating(siteNumber, "base")
3 attr(ratingData, "RATING")
4 """
5 # get_ratings returns a dict keyed by "<id>.<file_type>.rdb"
----> 6 ratings_data = waterdata.get_ratings(
7 monitoring_location_id="USGS-01594440", file_type="base"
8 )
9 list(ratings_data.keys())
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/waterdata/ratings.py:208, in get_ratings(monitoring_location_id, file_type, file_path, time, bbox, limit, download_and_parse, ssl_check)
205 if file_path is not None:
206 os.makedirs(file_path, exist_ok=True)
--> 208 return _download_all(matching, file_path, ssl_check)
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/waterdata/ratings.py:413, in _download_all(features, file_path, ssl_check)
398 out[fid] = df
399 return df, _inert_response(
400 response.status_code, str(response.url), response.headers
401 )
403 FanOut(
404 features,
405 fetch,
406 RetryPolicy.from_configuration(adapter="waterdata"),
407 client_options={"verify": ssl_check},
408 # No single URL expresses "all of these assets" -- the aggregate
409 # reports the first, matching what a single-feature call would show.
410 canonical_url=_asset_href(features[0]),
411 service="ratings",
412 adapter="waterdata",
--> 413 ).resume()
414 return out
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/fanout.py:525, in FanOut.resume(self)
517 concurrency = _configuration.concurrency(
518 self.default_concurrent, adapter=self.adapter
519 )
520 with start_blocking_portal() as portal:
521 # ``portal.call`` returns ``Any`` because ``functools.partial``
522 # erases ``_run``'s return type; restore the declared tuple.
523 return cast(
524 "tuple[pd.DataFrame, Any]",
--> 525 portal.call(functools.partial(self._run, concurrency)),
526 )
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/anyio/from_thread.py:338, in BlockingPortal.call(self, func, *args)
323 def call(
324 self,
325 func: Callable[[Unpack[PosArgsT]], Awaitable[T_Retval] | T_Retval],
326 *args: Unpack[PosArgsT],
327 ) -> T_Retval:
328 """
329 Call the given function in the event loop thread.
330
(...) 336
337 """
--> 338 return cast(T_Retval, self.start_task_soon(func, *args).result())
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/concurrent/futures/_base.py:460, in Future.result(self, timeout)
458 raise CancelledError()
459 elif self._state == FINISHED:
--> 460 return self.__get_result()
461 else:
462 raise TimeoutError()
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/concurrent/futures/_base.py:402, in Future.__get_result(self)
400 if self._exception is not None:
401 try:
--> 402 raise self._exception
403 finally:
404 # Break a reference cycle with the exception in self._exception
405 self = None
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/anyio/from_thread.py:263, in BlockingPortal._call_func(self, func, args, kwargs, future)
261 with CancelScope() as scope:
262 future.add_done_callback(callback)
--> 263 retval = await retval_or_awaitable
264 else:
265 retval = retval_or_awaitable
File /opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/transport/fanout.py:663, in FanOut._run(self, max_concurrent)
660 if interrupted is None:
661 # Unreachable: classified as transient just above.
662 raise self._normalize_failure(first_transient)
--> 663 raise interrupted from first_transient
665 return self.finalize(*self._combine_raw())
QuotaExhausted: HTTP 429 after 0/1 chunks; catch QuotaExhausted (or FanOutInterrupted) to access .partial_frame or .call.resume() once the rate-limit window has rolled over. Cause: RateLimited: 429: Too many requests made. Please obtain an API token or try again later. (URL: https://api.waterdata.usgs.gov/stac-files/ratings/USGS.01594440.base.rdb)
[10]:
"""
discharge_stats <- readNWISstat(siteNumbers=c("02319394"),
parameterCd=c("00060"),
statReportType="annual")
"""
discharge_stats, md = waterdata.get_stats_date_range(
monitoring_location_id="USGS-02319394",
parameter_code="00060",
computation_type="arithmetic_mean",
)
Retrieving: observationIntervals · 1 page · 344 rows
[11]:
# R: readNWISdata(service="dv", stateCd="WI", parameterCd="00060",
# drainAreaMin="50", statCd="00003")
#
# The Water Data API serves daily values per monitoring location. To assemble a
# state-wide set, first find the locations (optionally filtering by drainage
# area) with waterdata.get_monitoring_locations(state_name="Wisconsin", ...),
# then pass their ids to waterdata.get_daily(parameter_code="00060",
# statistic_id="00003").
[12]:
# '''
# sitesNJ <- whatWQPsites(statecode="US:34",
# characteristicName="Chloride")
# '''
# sitesNJ, md = wqp.what_sites(statecode="US:34", characteristicName="Chloride")
[13]:
# '''
# dataPH <- readWQPdata(statecode="US:55",
# characteristicName="pH")
# '''
# dataPH, md = wqp.what_sites(statecode="US:55", characteristicName="pH")
[14]:
# '''
# type <- "Stream"
# sites <- whatWQPdata(countycode="US:55:025",siteType=type)
# '''
# streamType = "Stream"
# sites, md = wqp.get_results(countycode="US:55:025", siteType=streamType)
[15]:
"""site <- whatWQPsamples(siteid="USGS-01594440")"""
site, md = wqp.what_sites(siteid="USGS-01594440")
/opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/wqp.py:664: DataCurrencyWarning: This function call will return the legacy WQX format, which means USGS data have not been updated since March 2024. Please review the dataretrieval-python documentation for more information on updated WQX3.0 profiles. Setting `legacy=False` will remove this warning.
_warn_legacy_use()
[16]:
"""
type <- "Stream"
sites <- whatWQPmetrics(countycode="US:55:025",siteType=type)
"""
streamType = "Stream"
sites, md = wqp.what_sites(countycode="US:55:025", siteType=streamType)
/opt/hostedtoolcache/Python/3.13.15/x64/lib/python3.13/site-packages/dataretrieval/wqp.py:664: DataCurrencyWarning: This function call will return the legacy WQX format, which means USGS data have not been updated since March 2024. Please review the dataretrieval-python documentation for more information on updated WQX3.0 profiles. Setting `legacy=False` will remove this warning.
_warn_legacy_use()
Embedded Metadata
Most waterdata and wqp service methods return a tuple of the requested data (a pandas DataFrame) and a metadata object.
md is an object with the following attributes:
Metadata
url # the URL used to query the service
query_time # how long the query took
header # the response headers
Note: USGS water use data has no Water Data API equivalent yet. The legacy nwis.get_water_use() service has been decommissioned and now raises a “defunct” error, so there is currently no runnable way to retrieve water-use data through dataretrieval.