Skip to contents

Daily and Sample retrieval updates

Matthew Diebel and Laura DeCicco

2026-04-30

Source: vignettes/data_retrieval_updates.Rmd
data_retrieval_updates.Rmd

Introduction

The functions for retrieving daily and sample data were revised in EGRET 3.0.12 to use new USGS water data services (https://waterservices.usgs.gov/). As part of these revisions, additional arguments were added to readNWISDaily to allow more flexibility for dealing with missing, zero, and negative discharge values. Columns were also added to the Sample data frames returned by readNWISSample and readWQPSample to help identify potential inconsistencies among samples, such as samples with different collection methods. Messages printed by readNWISDaily, readNWISSample, and readWQPSample were revised to alert users to key features of the resulting data frames. This vignette details these revisions and provides suggestions for dealing with anomalies.

readNWISDaily

The readNWISDaily function retrieves mean daily values from the USGS water data service using the dataRetrieval function read_waterdata_daily. While any parameter code with daily values can be retrieved with this function, the default is “00060” for discharge in cubic feet per second (ft3/s). The default argument convert = TRUE converts the values to cubic meters per second (m3/s) for use in EGRET. The new arguments in EGRET 3.0.12 are adjust and fill. When adjust is TRUE (default) and when there are any zero values but no negative values, a small constant (0.001 times the mean discharge) will be added to all of the daily discharge values to allow the log transformation of discharge used in WRTDS. If there are any negative values or if adjust is FALSE, no adjustment will be made. When the argument fill is TRUE (default is FALSE), linear interpolation will be used to fill missing values, including discontinous daily records and values recorded as NA or -999999. Messages summarize the number and date ranges of missing values and whether the interpolation was performed. Both adjust and fill should only be used when there are very few zero or missing values.

Here is an example of using readNWISDaily on a daily discharge dataset from the Little Auglaize River at Melrose OH - USGS-04191058. First, we will run it with adjust = TRUE and fill = FALSE (the defaults):

library(EGRET)
Daily <- readNWISDaily(siteNumber = "04191058",
                       parameterCd = "00060",
                       startDate = "2015-04-15",
                       endDate = "2024-10-01",
                       verbose = TRUE,
                       convert = TRUE,
                       adjust = TRUE,
                       fill = FALSE)
## There are 272 negative flow days.
## Many EGRET functions will not work with negative values.
## Adjust is TRUE but there are 272 negative flow days.
## Discharge was not adjusted.
## NA ranges in Q (1 total NA value across 1 run):
##   2016-11-09 to 2016-11-09 (1 day)
## Many EGRET functions will not work with missing values.

There are 272 negative flow days, so even though adjust = TRUE, discharge is not adjusted because the adjustment method will not adjust negative flows. Also, there is one missing value. We can fill this value by interpolation by making fill = TRUE. If we also make adjust = FALSE, the resulting Daily data frame will not be usable in most EGRET functions, but could be analyzed with functions in WRTDSplus, as shown in this example [link].

Daily <- readNWISDaily(siteNumber = "04191058",
                       parameterCd = "00060",
                       startDate = "2015-04-15",
                       endDate = "2024-10-01",
                       verbose = TRUE,
                       convert = TRUE,
                       adjust = FALSE,
                       fill = TRUE)
## There are 272 negative flow days.
## Many EGRET functions will not work with negative values.
## NA ranges in Q (1 total NA value across 1 run):
##   2016-11-09 to 2016-11-09 (1 day)
## Many EGRET functions will not work with missing values.
## All NA values filled by linear interpolation.

Here is an example for Nile Ditch at Arnold Hwy NR Jasper, MI - USGS-04176063. Here, we start with the default arguments. The messages indicate that there are 6 zero flow days and no negative flow days. All days had 5e-04 cms added to the discharge value. The small constant (0.001 times mean discharge) that was added to all daily values will allow the log transformation in WRTDS and will add a very small positive bias to the flux estimates.

Daily <- readNWISDaily(siteNumber = "04176063",
                       parameterCd = "00060",
                       startDate = "2020-10-01",
                       endDate = "2024-09-01",
                       verbose = TRUE,
                       convert = TRUE,
                       adjust = TRUE,
                       fill = FALSE)
## There are 6 zero flow days and no negative flow days.
## All days had 5e-04 cms added to the discharge value.

readNWISSample

The readNWISSample function retrieves USGS sample data from the USGS water data service using the dataRetrieval function read_waterdata_samples. There are no new arguments for this function in EGRET 3.0.12, but the function now returns several additional columns with information about the sampling activity, collection method, and result characteristics. It also prints a message to alert users to potential inconsistencies among samples, such as samples from different media (water, sediment, biological tissue) or sample collection methods. It is then the user’s responsibility to remove inappropriate samples before further analysis in EGRET.

Here is an example of using readNWISSample on a total phosphorus dataset from the Detroit River at Fort Wayne at Detroit, MI - USGS-04165710. This site has samples collected with a variety of methods.

Sample <- readNWISSample(siteNumber = "04165710",
                         parameterCd = "00665",
                         verbose = TRUE)
## 186 samples retrieved.
## Multiple values for some sample characteristics:
## ActivityTypeCode: 'Sample-Routine', 'Sample-Integrated Horizontal Profile'
## SampleCollectionMethod: 'EDI', 'EWI', 'Point sample', '', 'Grab/dip', 'Single vertical DI', 'Other', 'Multiple verticals'

readWQPSample

Here is an example of using readWQPSample to retrieve the same data. Note that siteNumber must contain the agency code prefix with readWQPSample because WQP contains data from multiple agencies.

Sample <- readWQPSample(siteNumber = "USGS-04165710",
                        characteristicName = "00665",
                        verbose = TRUE)
## Please use readNWISSample for USGS data
## 186 samples retrieved.
## Multiple values for some sample characteristics:
## ActivityTypeCode: 'Sample-Routine', 'Sample-Integrated Horizontal Profile'
## SampleCollectionMethod: 'EDI', 'EWI', 'Point sample', '', 'Grab/dip', 'Single vertical DI', 'Other', 'Multiple verticals'