Skip to contents

dataRetrieval Status

July 25, 2024

Source: vignettes/Status.Rmd
Status.Rmd

This page will be updated frequently with information about the status of dataRetrieval services.

Contact with additional questions!

Overview

As of March 11, 2024, NWIS discrete water quality services are “frozen”: any public data retrieval will not include any new data.

NWIS qw services

Discrete water quality samples data are undergoing modernization, and NWIS services pulling that discrete water quality data (‘qw’) are no longer being updated as of March 11, 2024. A full decommission is expected approximately 6 months later. Learn more about the upcoming change and where to find the new samples data in our blog.

If you have additional questions about the NWIS qw data service, email .

WQP: USGS Data

New USGS data: temporarily not accessible on the main Water Quality Portal (WQP) page (www.waterqualitydata.us). Data are still being collected, but are not publicly available. This limited availability is expected to last a few months.

New USGS data are accessible in a pre-release (beta) version of the WQP web page and new wqx3 web services. Data are available in the “WQX version 3.0 format” (WQX = Water Quality Exchange) for three new “Data Profiles” (how the data is formatted by the WQP):

  • Monitoring Location
  • Narrow results
  • Full Physical Chemical results
  • Basic Physical Chemical

Guidance on how to use the new web page and web services are available in the User Guide and Web Services Guide. Additional profiles will continue to be added to this beta release in the coming weeks.

During the beta period users may encounter bugs or identify issues with the implementation of the WQX 3.0 format: we welcome (and encourage!) your feedback to help improve these offerings, just send an email to .

The current WQP data profiles (available on the main Water Quality Portal web pages and from the current web services, https://www.waterqualitydata.us) deliver data in “WQX version 2.0” format. These will remain available for a period of time after the rollout of version 3.0. Eventually they will be retired, but there is not yet an estimated time line.

What to expect: dataRetrieval specific

Function Status
readWQPdata Set to WQX3 full profile by default. ‘legacy’ option exists.
readWQPqw Set to WQX3 narrow profile by default. ‘legacy’ option exists.
whatWQPsites Set to WQX3 station profile by default. ‘legacy’ option exists.
whatWQPmetrics Currently only available via legacy services.
whatWQPsamples Currently only available via legacy services.
whatWQPdata Currently only available via legacy services.
readNWISqw Deprecated: should not be continued to use.
readWQPsummary Does not have accurate information for USGS data.
whatNWISdata Does not have accurate information for qw data.

readWQPqw

This function is generally advertised as a user-friendly function since it only works with a known list of sites, parameter codes or characterisitic names, and start/end dates.

As of dataRetrieval 2.7.15.1, this function will use the WQX 3.0 “narrow” dataProfile. There is an argument “legacy” that allows the user to switch back to the WQX 2.0 profile. Keep in mind the 2.0 profiles will eventually be retired. For any more flexibility, users will need to use the “readWQPdata” function.

An example of a WQX 3.0 return:

library(dataRetrieval)
rawPcode <- readWQPqw(siteNumbers = "USGS-01594440",
                      parameterCd = "01075")
attr(rawPcode, "url")
## [1] "https://www.waterqualitydata.us/wqx3/Result/search?siteid=USGS-01594440&pCode=01075&mimeType=csv&dataProfile=narrow"

Compared to using the WQX 2.0 legacy results:

rawPcode_legacy <- readWQPqw(siteNumbers = "USGS-01594440",
                      parameterCd = "01075",
                      legacy = TRUE)
## NEWS: Legacy data profiles will be retired. Please begin converting
## workflows to the WQX 3.0 profiles. Also, data from legacy profiles do not
## include USGS data newer than March 11, 2024. More details:
## https://doi-usgs.github.io/dataRetrieval/articles/Status.html
attr(rawPcode_legacy, "url")
## [1] "https://www.waterqualitydata.us/data/Result/search?siteid=USGS-01594440&pCode=01075&mimeType=csv"

readWQPdata

The “readWQPdata” function is the most flexible function to get WQP data. Currently there are 4 options that use the new WQX 3.0 profiles, and 11 legacy options.

WQX 3.0

There are now 2 “services” available: StationWQX and ResultWQX. The “ResultWQX” service has multiple available “dataProfiles”.

Service dataProfile
StationWQX
ResultWQX fullPhysChem
ResultWQX basicPhysChem
ResultWQX narrow

Examples:

data_full <- readWQPdata(siteid = "USGS-04024315",
                         characteristicName = "pH",
                         dataProfile = "fullPhysChem",
                         service = "ResultWQX3")
ncol(data_full)
## [1] 182
attr(data_full, "url")
## [1] "https://www.waterqualitydata.us/wqx3/Result/search?siteid=USGS-04024315&characteristicName=pH&dataProfile=fullPhysChem&mimeType=csv"
data_basic <- readWQPdata(siteid = "USGS-04024315",
                          characteristicName = "pH",
                          dataProfile = "basicPhysChem",
                          service = "ResultWQX3")
ncol(data_basic)
## [1] 100
attr(data_basic, "url")
## [1] "https://www.waterqualitydata.us/wqx3/Result/search?siteid=USGS-04024315&characteristicName=pH&dataProfile=basicPhysChem&mimeType=csv"
data_narrow <- readWQPdata(siteid = "USGS-04024315",
                           characteristicName = "pH",
                           dataProfile = "narrow",
                           service = "ResultWQX3")
ncol(data_narrow)
## [1] 65
attr(data_narrow, "url")
## [1] "https://www.waterqualitydata.us/wqx3/Result/search?siteid=USGS-04024315&characteristicName=pH&dataProfile=narrow&mimeType=csv"
data_sites <- readWQPdata(siteid = "USGS-04024315",
                          characteristicName = "pH",
                          service = "StationWQX3")
ncol(data_sites)
## [1] 56
attr(data_sites, "url")
## [1] "https://www.waterqualitydata.us/wqx3/Station/search?siteid=USGS-04024315&characteristicName=pH&mimeType=csv"

WQX 2.0 - Legacy

There are 8 services available from the legacy WQP. The Station and Result legacy services can still be accessed, but users should move to StationWQX and ResultWQX. As other former services become available in WQX 3.0, we will update these documents.

Service dataProfile WQX 3.0 option?
Station StationWQX
Result resultPhysChem ResultWQX
Result biological
Result narrowResult ResultWQX
Activity activityAll
ActivityMetric
Project
ProjectMonitoringLocationWeighting
ResultDetectionQuantitationLimit
BiologicalMetric

Examples:

# Data profiles: "Organization Data" (legacy)
org_data <- readWQPdata(
  statecode = "WI",
  countycode = "Dane",
  service = "Organization"
)

# Data profiles: "Project Data"  (legacy)
project_data <- readWQPdata(
  statecode = "WI",
  countycode = "Dane",
  service = "Project"
)

# Data profiles: "Project Monitoring Location Weighting Data"  (legacy)
proj_mlwd <- readWQPdata(
  statecode = "WI",
  countycode = "Dane",
  service = "ProjectMonitoringLocationWeighting"
)

# Data profiles: "Sample Results (physical/chemical metadata)"  (legacy)
samp_data <- readWQPdata(
  siteid = "USGS-04024315",
  dataProfile = "resultPhysChem",
  service = "Result"
)

# Data profiles: "Sample Results (biological metadata)"  (legacy)
samp_bio <- readWQPdata(
  siteid = "USGS-04024315",
  dataProfile = "biological",
  service = "Result"
)

# Data profiles: "Sample Results (narrow)" (legacy)
samp_narrow <- readWQPdata(
  siteid = "USGS-04024315",
  dataProfile = "narrowResult",
  service = "Result"
)

# Data profiles: "Sampling Activity"  (legacy)
samp_activity <- readWQPdata(
  siteid = "USGS-04024315",
  dataProfile = "activityAll",
  service = "Activity"
)


# Data profile: "Sampling Activity Metrics"  (legacy)
act_metrics <- readWQPdata(
  statecode = "WI",
  countycode = "Dane",
  service = "ActivityMetric"
)

# Data profile: "Result Detection Quantitation Limit Data"  (legacy)
dl_data <- readWQPdata(
  siteid = "USGS-04024315",
  service = "ResultDetectionQuantitationLimit"
)

whatNWISdata

NWIS services are “frozen”: the returned data availability will also be frozen ONLY for “qw” data_type_cd results. All other data types should not be affected.

When the NWIS services are decommissioned (possible September 2024): there will no longer be any “qw” information provided in the output of whatNWISdata. Discrete water-quality availability will be available via WQP services. More information will be provided as we learn more.

Here’s an example of what will change:

what_NWIS <- whatNWISdata(siteNumber = "05114000")
nrow(what_NWIS)
[1] 407
nrow(what_NWIS[what_NWIS$data_type_cd == "qw",])
[1] 381

So for site “05114000”, there are 381 NWIS qw parameters that have been measured. Starting mid-March 2024, the data availability for those 381 parameters will be frozen…even if new data is collected. Eventually those 381 rows of data will not be returned, only 26 rows of data will be returned (407-381).

New services/functions are being developed to replace the lost functionality so check back here for updated information.

Known Dependencies

EGRET

The readNWISSample function will be deprecated or updated. Plan to switch to readWQPSample.

HASP

The data_available function will need to be updated.

If you know of additional R package dependencies that will be affected by these changes, please email .