header_tag.html

Skip to contents

dataRetrieval Status

December 03, 2024

Source: vignettes/Status.Rmd
Status.Rmd

This page will be updated frequently with information about the status of dataRetrieval services.

Contact with additional questions.

Overview

On March 11, 2024, NWIS discrete water quality services were “frozen”: any public data retrieval using readNWISqw() did not include any new data. As of dataRetrieval v2.7.17, readNWISqw() has been retired and replaced by readWQPqw(). Learn more about the change and where to find the new samples data in our blog.

If you have additional questions about the NWIS qw data service, email .

Locating USGS data using the Water Quality Portal

New USGS data (post March 11, 2024) are temporarily not accessible on the main Water Quality Portal (WQP) page (www.waterqualitydata.us). Data are still being collected, but are not available on this webpage. This limited availability is expected to last a few months.

However, new USGS data are accessible in a pre-release (beta) version of the WQP web page and new wqx3 web services. Data are available in the “WQX version 3.0 format” (WQX = Water Quality Exchange) for these new “Data Profiles” (how the data is formatted by the WQP):

  • Monitoring Location
  • Results - Narrow
  • Results - Full Physical Chemical
  • Results - Basic Physical Chemical
  • Sampling Activity

Guidance on how to use the new web page and web services are available in the User Guide and Web Services Guide. Additional profiles will continue to be added over time.

Disclaimer: During the beta period, users may encounter bugs or identify issues with the implementation of the WQX 3.0 format: we welcome (and encourage!) your feedback to help improve these offerings, just send an email to .

The current WQP data profiles (available on the main Water Quality Portal web pages and from the current web services, https://www.waterqualitydata.us) deliver data in “WQX version 2.0” (what we’re referring to as the “legacy”) format. These will remain available for a period of time after the rollout of version 3.0. Eventually they will be retired, but there is not yet an estimated time line.

What to expect: dataRetrieval specific

Function Status
readWQPdata Set to legacy options by default. WQX3 options available.
readWQPqw Set to legacy options by default. WQX3 options available.
whatWQPsites Set to legacy options by default. WQX3 options available.
whatWQPmetrics Currently only available via legacy services.
whatWQPsamples Currently only available via legacy services.
whatWQPdata Currently only available via legacy services.
readNWISqw Retired.
readWQPsummary Does not have accurate information for USGS data.
whatNWISdata Does not have accurate information for qw data.

readWQPqw

The readWQPqw() function is generally advertised as a user-friendly function since it only works with a known list of sites, parameter codes or characteristic names, and start/end dates.

As of dataRetrieval 2.7.17, this function will use the default WQX version 2 dataProfile, specified by the legacy = TRUE argument. Setting legacy = FALSE will return the WQX 3.0 “narrow” dataProfile. Keep in mind the 2.0 profiles will eventually be retired. For any more flexibility, users will need to use the readWQPdata() function.

An example of a WQX 3.0 return:

library(dataRetrieval)
rawPcode <- readWQPqw(siteNumbers = "USGS-01594440",
                      parameterCd = "01075",
                      legacy = FALSE)
## GET: https://www.waterqualitydata.us/wqx3/Result/search?siteid=USGS-01594440&pCode=01075&mimeType=csv&dataProfile=basicPhysChem
## WQX3 services are in-development, use with caution.

Compared to using the WQX 2.0 legacy results:

rawPcode_legacy <- readWQPqw(siteNumbers = "USGS-01594440",
                      parameterCd = "01075",
                      legacy = TRUE)
## GET: https://www.waterqualitydata.us/data/Result/search?siteid=USGS-01594440&pCode=01075&mimeType=csv
## NEWS: Data does not include USGS data newer than March 11, 2024. More details:
## https://doi-usgs.github.io/dataRetrieval/articles/Status.html

readWQPdata

The readWQPdata() function is the most flexible function to get WQP data. Currently there are 11 legacy options and 5 options that use the new WQX 3.0 profiles. Note that readWQPdata() does not leverage a legacy argument to specify which profile version the user would like returned, but instead relies on the user’s specification of service and dataProfile arguments.

WQX 3.0

There are currently three WQX 3.0 “services” available: ResultWQX, StationWQX and ActivityWQX. The “ResultWQX” service has multiple available “dataProfiles”.

Service dataProfile
StationWQX
ResultWQX fullPhysChem
ResultWQX basicPhysChem
ResultWQX narrow
ActivityWQX

Examples:

data_full <- readWQPdata(siteid = "USGS-04024315",
                         characteristicName = "pH",
                         dataProfile = "fullPhysChem",
                         service = "ResultWQX3")
## GET: https://www.waterqualitydata.us/wqx3/Result/search?siteid=USGS-04024315&characteristicName=pH&dataProfile=fullPhysChem&mimeType=csv
## Trying again!
## GET: https://www.waterqualitydata.us/wqx3/Result/search?siteid=USGS-04024315&characteristicName=pH&dataProfile=fullPhysChem&mimeType=csv&try=1
## GET: https://www.waterqualitydata.us/wqx3/Station/search?siteid=USGS-04024315&characteristicName=pH&mimeType=csv
## WQX3 services are in-development, use with caution.
ncol(data_full)
## [1] 182
data_basic <- readWQPdata(siteid = "USGS-04024315",
                          characteristicName = "pH",
                          dataProfile = "basicPhysChem",
                          service = "ResultWQX3")
## GET: https://www.waterqualitydata.us/wqx3/Result/search?siteid=USGS-04024315&characteristicName=pH&dataProfile=basicPhysChem&mimeType=csv
## GET: https://www.waterqualitydata.us/wqx3/Station/search?siteid=USGS-04024315&characteristicName=pH&mimeType=csv
## WQX3 services are in-development, use with caution.
ncol(data_basic)
## [1] 100
data_narrow <- readWQPdata(siteid = "USGS-04024315",
                           characteristicName = "pH",
                           dataProfile = "narrow",
                           service = "ResultWQX3")
## GET: https://www.waterqualitydata.us/wqx3/Result/search?siteid=USGS-04024315&characteristicName=pH&dataProfile=narrow&mimeType=csv
## GET: https://www.waterqualitydata.us/wqx3/Station/search?siteid=USGS-04024315&characteristicName=pH&mimeType=csv
## WQX3 services are in-development, use with caution.
ncol(data_narrow)
## [1] 65
data_sites <- readWQPdata(siteid = "USGS-04024315",
                          characteristicName = "pH",
                          service = "StationWQX3")
## GET: https://www.waterqualitydata.us/wqx3/Station/search?siteid=USGS-04024315&characteristicName=pH&mimeType=csv
## GET: https://www.waterqualitydata.us/wqx3/Station/search?siteid=USGS-04024315&characteristicName=pH&mimeType=csv
## WQX3 services are in-development, use with caution.
ncol(data_sites)
## [1] 56

WQX 2.0 - Legacy

There are 8 services available from the legacy WQP. The Station and Result legacy services can still be accessed, but users should move to StationWQX, ResultWQX, and ActivityWQX. As other former services become available in WQX 3.0, we will update these documents.

Service dataProfile WQX 3.0 service “analog”
Station StationWQX
Result resultPhysChem ResultWQX
Result biological
Result narrowResult ResultWQX
Activity activityAll ActivityWQX
ActivityMetric
Project
ProjectMonitoringLocationWeighting
ResultDetectionQuantitationLimit
BiologicalMetric

Examples:

# Data profiles: "Organization Data" (legacy)
org_data <- readWQPdata(
  statecode = "WI",
  countycode = "Dane",
  service = "Organization"
)

# Data profiles: "Project Data"  (legacy)
project_data <- readWQPdata(
  statecode = "WI",
  countycode = "Dane",
  service = "Project"
)

# Data profiles: "Project Monitoring Location Weighting Data"  (legacy)
proj_mlwd <- readWQPdata(
  statecode = "WI",
  countycode = "Dane",
  service = "ProjectMonitoringLocationWeighting"
)

# Data profiles: "Sample Results (physical/chemical metadata)"  (legacy)
samp_data <- readWQPdata(
  siteid = "USGS-04024315",
  dataProfile = "resultPhysChem",
  service = "Result"
)

# Data profiles: "Sample Results (biological metadata)"  (legacy)
samp_bio <- readWQPdata(
  siteid = "USGS-04024315",
  dataProfile = "biological",
  service = "Result"
)

# Data profiles: "Sample Results (narrow)" (legacy)
samp_narrow <- readWQPdata(
  siteid = "USGS-04024315",
  dataProfile = "narrowResult",
  service = "Result"
)

# Data profiles: "Sampling Activity"  (legacy)
samp_activity <- readWQPdata(
  siteid = "USGS-04024315",
  dataProfile = "activityAll",
  service = "Activity"
)


# Data profile: "Sampling Activity Metrics"  (legacy)
act_metrics <- readWQPdata(
  statecode = "WI",
  countycode = "Dane",
  service = "ActivityMetric"
)

# Data profile: "Result Detection Quantitation Limit Data"  (legacy)
dl_data <- readWQPdata(
  siteid = "USGS-04024315",
  service = "ResultDetectionQuantitationLimit"
)

whatNWISdata

NWIS discrete water quality services are “frozen”: the returned data availability will also be frozen ONLY for “qw” data_type_cd results. All other data types should not be affected.

When the NWIS services are decommissioned (likely in 2025): there will no longer be any “qw” information provided in the output of whatNWISdata. Discrete water-quality availability will be available via WQP services. More information will be provided as we learn more.

Here’s an example of what will change:

what_NWIS <- whatNWISdata(siteNumber = "05114000")
nrow(what_NWIS)
[1] 407
nrow(what_NWIS[what_NWIS$data_type_cd == "qw",])
[1] 381

So for site “05114000”, there are 381 NWIS qw parameters that have been measured. Since mid-March 2024, the data availability for those 381 parameters are frozen…even if new data are collected. Eventually those 381 rows of data will not be returned, only 26 rows of data will be returned (407-381).

New services/functions are being developed to replace the lost functionality so check back here for updated information.

Known Dependencies

EGRET

The readNWISSample function will be deprecated or updated. Plan to switch to readWQPSample.

HASP

The data_available function will need to be updated.

If you know of additional R package dependencies that will be affected by these changes, please email .