Getting started with rnz
rnz exposes one RNetCDF-shaped API over both NetCDF
files (via RNetCDF)
and Zarr stores (via pizzarr).
On the Zarr side it follows the NZ-1.0 convention —
the structural bridge from Zarr v3 to NUG-compatible
semantics — and remains backward compatible with Zarr v2 stores using
xarray’s _ARRAY_DIMENSIONS attribute. The package README carries the
longer framing; this vignette walks the inquiry and read functions
side-by-side on a single dataset in both its NetCDF and Zarr forms, so
the two backends can be compared call-for-call.
The demo dataset is derived from https://gdo-dcp.ucllnl.org/downscaled_cmip_projections/ and is bundled in both NetCDF and Zarr form.
# these files are in `z_demo()` as a file store
# and via "https://raw.githubusercontent.com/DOI-USGS/rnz/main/inst/extdata/bcsd.zarr/" as an http store.
gsub(normalizePath(dirname(rnz::z_demo())), "",
list.files(rnz::z_demo(), recursive = TRUE, all.files = TRUE),
fixed = TRUE)
#> [1] ".zattrs" ".zgroup" ".zmetadata"
#> [4] "latitude/.zarray" "latitude/.zattrs" "latitude/0"
#> [7] "longitude/.zarray" "longitude/.zattrs" "longitude/0"
#> [10] "pr/.zarray" "pr/.zattrs" "pr/0.0.0"
#> [13] "tas/.zarray" "tas/.zattrs" "tas/0.0.0"
#> [16] "time/.zarray" "time/.zattrs" "time/0"
(z <- rnz::open_nz(rnz::z_demo()))
#> <ZarrGroup> /
#> Store type : DirectoryStore
#> Zarr format : 2
#> Read-only : TRUE
#> No. members : 5
cat(c(capture.output(rnz::nzdump(rnz::z_demo()))[1:25], "..."), sep = "\n")
#> zarr {
#> dimensions:
#> latitude = 33 ;
#> longitude = 81 ;
#> time = 12 ;
#> variables:
#> <f4 latitude(latitude) ;
#> latitude:_CoordinateAxisType = Lat ;
#> latitude:axis = Y ;
#> latitude:bounds = latitude_bnds ;
#> latitude:long_name = Latitude ;
#> latitude:standard_name = latitude ;
#> latitude:units = degrees_north ;
#> <f4 longitude(longitude) ;
#> longitude:_CoordinateAxisType = Lon ;
#> longitude:axis = X ;
#> longitude:bounds = longitude_bnds ;
#> longitude:long_name = Longitude ;
#> longitude:standard_name = longitude ;
#> longitude:units = degrees_east ;
#> <f4 pr(time, latitude, longitude) ;
#> pr:coordinates = time latitude longitude ;
#> pr:long_name = monthly_sum_pr ;
#> pr:name = pr ;
#> pr:units = mm/m ;
#> ...
nc_file <- rnz::z_demo(format = "netcdf")
(nc <- RNetCDF::open.nc(nc_file))
#> [1] 65536
#> attr(,"handle_ptr")
#> <pointer: 0x5576ea439380>
#> attr(,"class")
#> [1] "NetCDF"
cat(c(capture.output(RNetCDF::print.nc(nc))[1:25], "..."), sep = "\n")
#> netcdf classic {
#> dimensions:
#> latitude = 33 ;
#> longitude = 81 ;
#> time = UNLIMITED ; // (12 currently)
#> variables:
#> NC_FLOAT latitude(latitude) ;
#> NC_CHAR latitude:standard_name = "latitude" ;
#> NC_CHAR latitude:long_name = "Latitude" ;
#> NC_CHAR latitude:units = "degrees_north" ;
#> NC_CHAR latitude:axis = "Y" ;
#> NC_CHAR latitude:bounds = "latitude_bnds" ;
#> NC_CHAR latitude:_CoordinateAxisType = "Lat" ;
#> NC_FLOAT longitude(longitude) ;
#> NC_CHAR longitude:standard_name = "longitude" ;
#> NC_CHAR longitude:long_name = "Longitude" ;
#> NC_CHAR longitude:units = "degrees_east" ;
#> NC_CHAR longitude:axis = "X" ;
#> NC_CHAR longitude:bounds = "longitude_bnds" ;
#> NC_CHAR longitude:_CoordinateAxisType = "Lon" ;
#> NC_FLOAT pr(longitude, latitude, time) ;
#> NC_CHAR pr:long_name = "monthly_sum_pr" ;
#> NC_CHAR pr:units = "mm/m" ;
#> NC_FLOAT pr:_FillValue = 100000002004087734272 ;
#> NC_CHAR pr:name = "pr" ;
#> ...“Inquire” about elements of a Zarr store
We can do all the normal inquire stuff using 0-indexed ids or names.
Inquire about a store
rnz::inq_nz_source(nc)
#> $ndims
#> [1] 3
#>
#> $nvars
#> [1] 5
#>
#> $ngatts
#> [1] 30
#>
#> $unlimdimid
#> [1] 2
#>
#> $format
#> [1] "classic"
#>
#> $libvers
#> [1] "4.9.2 of Mar 31 2024 08:09:36 $"
rnz::inq_nz_source(z)
#> $ndims
#> [1] 3
#>
#> $nvars
#> [1] 5
#>
#> $ngatts
#> [1] 30
#>
#> $format
#> [1] "DirectoryStore"Inquire about a group
TODO: get rnz working with more than root groups?
rnz::inq_grp(nc)
#> $self
#> [1] 65536
#> attr(,"handle_ptr")
#> <pointer: 0x5576ea439380>
#> attr(,"class")
#> [1] "NetCDF"
#>
#> $grps
#> list()
#>
#> $name
#> [1] "/"
#>
#> $fullname
#> [1] "/"
#>
#> $dimids
#> [1] 0 1 2
#>
#> $unlimids
#> [1] 2
#>
#> $varids
#> [1] 0 1 2 3 4
#>
#> $typeids
#> integer(0)
#>
#> $ngatts
#> [1] 30
rnz::inq_grp(z)
#> $grps
#> list()
#>
#> $name
#> [1] "/"
#>
#> $fullname
#> [1] "/"
#>
#> $dimids
#> [1] 0 1 2
#>
#> $varids
#> [1] 0 1 2 3 4
#>
#> $ngatts
#> [1] 30Inquire about a dimension
rnz::inq_dim(nc, 2)
#> $id
#> [1] 2
#>
#> $name
#> [1] "time"
#>
#> $length
#> [1] 12
#>
#> $unlim
#> [1] TRUE
rnz::inq_dim(z, 2)
#> $id
#> [1] 2
#>
#> $name
#> [1] "time"
#>
#> $length
#> [1] 12
rnz::inq_dim(nc, "time")
#> $id
#> [1] 2
#>
#> $name
#> [1] "time"
#>
#> $length
#> [1] 12
#>
#> $unlim
#> [1] TRUE
rnz::inq_dim(z, "time")
#> $id
#> [1] 2
#>
#> $name
#> [1] "time"
#>
#> $length
#> [1] 12Inquire about a variable
rnz::inq_var(nc, 4)
#> $id
#> [1] 4
#>
#> $name
#> [1] "time"
#>
#> $type
#> [1] "NC_DOUBLE"
#>
#> $ndims
#> [1] 1
#>
#> $dimids
#> [1] 2
#>
#> $natts
#> [1] 4
rnz::inq_var(z, 4)
#> $id
#> [1] 4
#>
#> $name
#> [1] "time"
#>
#> $type
#> [1] "<f8"
#>
#> $ndims
#> [1] 1
#>
#> $dimids
#> [1] 2
#>
#> $natts
#> [1] 4
rnz::inq_var(nc, "time")
#> $id
#> [1] 4
#>
#> $name
#> [1] "time"
#>
#> $type
#> [1] "NC_DOUBLE"
#>
#> $ndims
#> [1] 1
#>
#> $dimids
#> [1] 2
#>
#> $natts
#> [1] 4
rnz::inq_var(z, "time")
#> $id
#> [1] 4
#>
#> $name
#> [1] "time"
#>
#> $type
#> [1] "<f8"
#>
#> $ndims
#> [1] 1
#>
#> $dimids
#> [1] 2
#>
#> $natts
#> [1] 4Inquire about an attribute
rnz::inq_att(nc, "time", "units")
#> $id
#> [1] 1
#>
#> $name
#> [1] "units"
#>
#> $type
#> [1] "NC_CHAR"
#>
#> $length
#> [1] 30
rnz::inq_att(z, "time", "units")
#> $id
#> [1] 3
#>
#> $name
#> [1] "units"
#>
#> $type
#> [1] "character"
#>
#> $length
#> [1] 1
rnz::inq_att(nc, 4, 3)
#> $id
#> [1] 3
#>
#> $name
#> [1] "_CoordinateAxisType"
#>
#> $type
#> [1] "NC_CHAR"
#>
#> $length
#> [1] 4
rnz::inq_att(z, 4, 3)
#> $id
#> [1] 3
#>
#> $name
#> [1] "units"
#>
#> $type
#> [1] "character"
#>
#> $length
#> [1] 1Get a variable
rnz::get_var(nc, "time")
#> [1] 17927 17955 17986 18016 18047 18077 18108 18139 18169 18200 18230 18261
rnz::get_var(z, "time")
#> [1] 17927 17955 17986 18016 18047 18077 18108 18139 18169 18200 18230 18261
rnz::get_var(nc, var = "pr",
start = c(1,1,5), count = c(3,3,1))
#> [,1] [,2] [,3]
#> [1,] 69.27 69.59 69.51
#> [2,] 60.19 65.76 68.04
#> [3,] 49.68 55.63 55.45
rnz::get_var(z, var = "pr",
start = c(1,1,5), count = c(3,3,1))
#> [,1] [,2] [,3]
#> [1,] 134.90 137.65 149.01
#> [2,] 59.46 73.97 79.23
#> [3,] 101.86 101.06 96.12
# TODO: figure out why aperm #3
pr <- rnz::get_var(z, "pr") |>
aperm(c(3,2,1))
# TODO: figure out NA handling
pr[pr > 1000] <- NA
image(pr[,,1], col = hcl.colors(12, "PuBuGn", rev = TRUE),
useRaster = TRUE, axes = FALSE)
pr <- rnz::get_var(nc, "pr")
image(pr[,,1], col = hcl.colors(12, "PuBuGn", rev = TRUE),
useRaster = TRUE, axes = FALSE)
Remote and cloud stores
Zarr was designed for data that sits in object storage rather than on
a local disk, and most published Zarr stores are hosted that way. The
calls above do not change when the data moves — open_nz()
takes a URL as readily as a path, and everything downstream of it works
on what the store reports rather than on where the store lives.
Catalog entries for cloud-hosted data usually advertise a bucket address paired with an endpoint. gridMET, a daily gridded meteorology dataset for the continental United States, is republished by the USGS on an Open Storage Network pod and listed this way:
s3://mdmf/gdp/gridMET.zarr/ endpoint https://usgs.osn.mghpcc.org/That pairing — a bucket URL plus a host that is not Amazon — is what sends most people looking for documentation. In rnz the endpoint is the one setting you need, and it comes from the environment rather than a function argument, because the underlying store handle is cached and shared across calls:
Sys.setenv(AWS_ENDPOINT = "https://usgs.osn.mghpcc.org")
z_s3 <- rnz::open_nz("s3://mdmf/gdp/gridMET.zarr")
rnz::inq_nz_source(z_s3)
#> $ndims
#> [1] 3
#>
#> $nvars
#> [1] 12
#>
#> $ngatts
#> [1] 0
#>
#> $format
#> [1] "S3Store"Inquiry proceeds exactly as it did against the bundled demo data:
rnz::inq_dim(z_s3, 0)
#> $id
#> [1] 0
#>
#> $name
#> [1] "lat"
#>
#> $length
#> [1] 585
rnz::inq_var(z_s3, "precipitation_amount")
#> $id
#> [1] 7
#>
#> $name
#> [1] "precipitation_amount"
#>
#> $type
#> [1] "<i2"
#>
#> $ndims
#> [1] 3
#>
#> $dimids
#> [1] 2 0 1
#>
#> $natts
#> [1] 12
rnz::inq_att(z_s3, "lat", "units")
#> $id
#> [1] 4
#>
#> $name
#> [1] "units"
#>
#> $type
#> [1] "character"
#>
#> $length
#> [1] 1Reading is the same call as well, with one thing to keep in mind.
precipitation_amount is 17369 days by 585 rows by 1386
columns, stored in chunks of 2190 by 150 by 150. A chunk is the smallest
unit the store will return, so asking for a single value still transfers
the block that contains it. Slice along chunk boundaries and keep
requests small in the finely chunked dimensions:
rnz::get_var(z_s3, "lat", start = 1, count = 5)
#> [1] 49.40000 49.35833 49.31667 49.27500 49.23333
rnz::get_var(z_s3, "precipitation_amount",
start = c(1, 1, 1), count = c(2, 3, 4))
#> , , 1
#>
#> [,1] [,2] [,3]
#> [1,] 32767 32767 32767
#> [2,] 32767 32767 32767
#>
#> , , 2
#>
#> [,1] [,2] [,3]
#> [1,] 32767 32767 32767
#> [2,] 32767 32767 32767
#>
#> , , 3
#>
#> [,1] [,2] [,3]
#> [1,] 32767 32767 32767
#> [2,] 32767 32767 32767
#>
#> , , 4
#>
#> [,1] [,2] [,3]
#> [1,] 32767 32767 32767
#> [2,] 32767 32767 32767Store handles are cached by URL, so a changed endpoint has no effect
until the handle is dropped. close_nz() drops it:
rnz::close_nz(z_s3)Google Cloud Storage works the same way, with one wrinkle: it has no
anonymous mode by default, and a world-readable bucket will still fail
without either credentials or an explicit instruction to skip signing.
GOOGLE_SKIP_SIGNATURE is that instruction:
Sys.setenv(GOOGLE_SKIP_SIGNATURE = "true")
z_gs <- rnz::open_nz("gs://pangeo-data/ECCO_basins.zarr")
rnz::inq_var(z_gs, "basin_mask")
#> $id
#> [1] 0
#>
#> $name
#> [1] "basin_mask"
#>
#> $type
#> [1] ">f4"
#>
#> $ndims
#> [1] 3
#>
#> $dimids
#> [1] 0 1 2
#>
#> $natts
#> [1] 0
rnz::get_var(z_gs, "basin_mask", start = c(1, 1, 1), count = c(2, 3, 4))
#> , , 1
#>
#> [,1] [,2] [,3]
#> [1,] 0 0 0
#> [2,] 2 2 2
#>
#> , , 2
#>
#> [,1] [,2] [,3]
#> [1,] 0 0 0
#> [2,] 2 2 2
#>
#> , , 3
#>
#> [,1] [,2] [,3]
#> [1,] 0 0 0
#> [2,] 2 2 2
#>
#> , , 4
#>
#> [,1] [,2] [,3]
#> [1,] 0 0 0
#> [2,] 2 2 2
rnz::close_nz(z_gs)Both stores above are Zarr v2, but nothing in the calls depends on
that. A v3 store reached the same way behaves identically, because the
two versions differ only in where they put the information rnz reads: v2
labels dimensions with an _ARRAY_DIMENSIONS attribute,
while v3 gives them a dimension_names field of their own,
and rnz prefers the latter where both appear.
The s3:// and gs:// routes need the
r-universe pizzarr build, which compiles the zarrs Rust backend with
cloud support; the CRAN build is pure R and cannot take them. And rnz
can only inquire a cloud store that publishes consolidated metadata —
every array’s metadata gathered into one document, which is a
.zmetadata key on v2 and a
consolidated_metadata member of the root
zarr.json on v3 — because the object store APIs offer no
way to list a store’s contents. Both stores above publish it, which is
why the calls can find the variables without being told their names.
Where neither holds, most of these services also serve their objects
over plain HTTPS, and that address works with no configuration at
all:
z_http <- rnz::open_nz("https://usgs.osn.mghpcc.org/mdmf/gdp/gridMET.zarr")
rnz::inq_dim(z_http, 0)
#> $id
#> [1] 0
#>
#> $name
#> [1] "lat"
#>
#> $length
#> [1] 585That’s all for now.