Data Leaf Metadata File
GSPy uses metadata files (YAML or JSON) to ingest required and ancillary information and build the Data Tree.
The Data Leaf Metadata File must contain the required sections dataset_attrs and coordinates.
The sections dimensions and variables are often commonly required, depending on the data type.
Lastly, sections with the suffix _system are automatically identified by GSPy and added to the Data Leaf group as a Supplementary Leaflet. Systems can also be included in the Survey metadata file, for easy access to copy systems through when adding data leaves later on. However, when writing to netCDF, all systems attached to the ‘/survey’ group directly are dropped.
Any additional sections are optional at the user’s discretion.
Dataset Attributes
Global attributes for the Data Leaf group are passed through the dataset attributes, dataset_attrs. The GS standard requires and recommends multiple specific keys, explained on this page.
The GSPy package automatically adds the structure key when reading in data files based on the type of file (tabular vs. raster).
1dataset_attrs:
2 content: raw data
3 comment: This dataset includes minimally processed (raw) AEM and raw/processed magnetic data provided by SkyTEM
4 type: data
5 structure: tabular
6 mode: airborne
7 method: electromagnetic, time domain
8 instrument: skytem
9 property: ''
Coordinates
The four standardized coordinate variables (x, y, z, and t) are specificed through the coordinates dictionary, which is simply a key mapping to the variables the user wants to designate as the coordinates.
11coordinates:
12 x: E_Nad83
13 y: N_Nad83
14 z: DEM
15 t: DateTime
GSPy automatically adds the required CF attribute axis with the values ‘X’, ‘Y’, ‘Z’, or ‘T’ to these coordinate variables for universal identification.
The vertical z and time t coordinates both require the attribute datum, and the z coordinate additionally requires positive indicating ‘up’ or ‘down’, per CF conventions. These attributes are passed through the variables dictionary (see further below).
Dimensions
Underneath the “dimensions” header, any subsection will become a dimension coordinate with the title corresponding to its variable name (e.g. “layer_depth”). GSPy will generate the values of the dimension coordinate through multiple options:
- if “centers” are passed, without “bounds”:
the 1-D list becomes the coordinate values
if “discrete: True” is passed, then no bounds are calculated, otherwise GSPy with automatically generate a “bounds” variable based on the spacing of the centers.
- if “bounds” are passed, without centers:
the “bounds” should be a 2 x N list, where N is the length of the dimension
the “centers” are automatically generated based on the widths of the bounds
- if “centers” and “bounds” are both passed:
both “bounds” and “centers” are defined directly based on the values passed. “centers” should be a 1-D list of length N, and “bounds” should be a 2-D list and can be either (2, N) or (N, 2).
- Using the fields “length”, “increment”, and “origin”:
GSPy with automatically generate a 1-D array corresponding to the centers of the dimension starting at the origin and incrementing evenly to the total length defined.
If only “length” is passed the “origin” is assumed to be zero and the “increment” is one.
As before, if “discrete: True” is passed, then no bounds are calculated, otherwise a “bounds” variable will be generated based on the spacing of the centers
18dimensions:
19 layer_depth:
20 standard_name: layer_depth
21 long_name: layer depth below surface
22 units: meters
23 missing_value: not_defined
24 length: 31
25 increment: 5.0
26 origin: 2.5
27 positive: down
28 datum: ground surface
Variables
Variable metadata is primarily ingested through the “variables” section. Here, each variable must have at a minimum the required attributes. Furthermore, attributes such as dimensions, raw_data_columns, and system_couplet should be added where relevant. Additional attributes may optionally be included.
dimensions:should be a list of dimension names. For tabular datasets the first dimension should always be “index” and the second should correspond to either a dimension defined in the early “dimensions” section or within a “systems” section (see examples of “systems” below).
raw_data_columns:should be a list of column names from the source data file that are meant to be combined into a single 2-D variable.
system_couplet:the “couplet_label” for the specific data variable. In system definitions, couplets are unique source (transmitter) - receiver pairs. Any data variable containing measurements corresponding to an individual couplet should have the attribute
system_couplet.
In this example below, notice that the variables corresponding to the coordinate variables z and t in the earlier coordinates section have additional required attributes.
30variables:
31 E_Nad83:
32 standard_name: easting_nad83
33 long_name: Easting, Wisconsin Transverse Mercator (WTM), North American Datum of 1983 (NAD83)
34 units: meter
35 missing_value: not_defined
36
37 N_Nad83:
38 standard_name: northing_nad83
39 long_name: Northing, Wisconsin Transverse Mercator (WTM), North American Datum of 1983 (NAD83)
40 units: meter
41 missing_value: not_defined
42
43 DateTime:
44 standard_name: time
45 long_name: Time, decimal days
46 units: day
47 missing_value: not_defined
48 datum: January 1, 1900
49
50 DEM:
51 standard_name: dem
52 long_name: Digital elevation model
53 units: meter
54 missing_value: not_defined
55 positive: up
56 datum: North American Vertical Datum of 1988 (NAVD88)
57
58 Fid:
59 standard_name: fiducial
60 long_name: Unique fiducial number
61 units: not_defined
62 missing_value: not_defined
63
64 Flight:
65 standard_name: flight_number
66 long_name: Flight name, yyyymmdd.ff
67 units: not_defined
68 missing_value: not_defined
69
70 LM_X:
71 standard_name: em_data_lmx
72 long_name: EM data, low moment x-component
73 units: picoVolt per Ampere per meter^4
74 missing_value: -9999.99
75 system_couplet: lm_x
76 dimensions: [index, lm_gate_times]
77
78 LM_Z:
79 standard_name: em_data_lmz
80 long_name: EM data, low moment z-component
81 units: picoVolt per Ampere per meter^4
82 missing_value: -9999.99
83 system_couplet: lm_z
84 dimensions: [index, lm_gate_times]
85
86 HM_X:
87 standard_name: em_data_hmx
88 long_name: EM data, high moment x-component
89 units: picoVolt per Ampere per meter^4
90 missing_value: -9999.99
91 system_couplet: hm_x
92 dimensions: [index, hm_gate_times]
93
94 HM_Z:
95 standard_name: em_data_hmz
96 long_name: EM data, high moment z-component
97 units: picoVolt per Ampere per meter^4
98 missing_value: -9999.99
99 system_couplet: hm_z
100 dimensions: [index, hm_gate_times]
101
102 RMF:
103 standard_name: residual_magnetic_field
104 long_name: Residual magnetic field, IGRF corrected based on 2015 model
105 units: nanoTesla
106 missing_value: not_defined
107 system_couplet: passive_scalar_magnetometer
108
109 TMI:
110 standard_name: total_magnetic_intensity
111 long_name: Diurnally corrected filtered total magnetic intensity
112 units: nanoTesla
113 missing_value: not_defined
114 system_couplet: passive_scalar_magnetometer
Systems
Many geophysical methods entail complex system configurations with detailed ancillary data and metadata information. These system data are often critical for accurate inversion and data handling. GSPy adds this information in a standardized structure and located at the Supplementary Leaflet tier beneath corresponding Data Leaf groups.
1nominal_system:
2 type: system
3 mode: airborne
4 method: electromagnetic, time domain
5 instrument: SkyTEM 304M
6
7 dimensions:
8
9 lm_gate_times:
10 standard_name: lm_gate_times
11 long_name: calibrated low moment gate times
12 units: seconds
13 missing_value: not_defined
14 bounds: [[-1.420000e-06, -8.500000e-07],
15 [-4.200000e-07, 1.150000e-06],
16 [ 1.580000e-06, 3.150000e-06],
17 [ 3.580000e-06, 5.150000e-06],
18 [ 5.580000e-06, 7.150000e-06],
19 [ 7.580000e-06, 9.150000e-06],
20 [ 9.580000e-06, 1.115000e-05],
21 [ 1.158000e-05, 1.415000e-05],
22 [ 1.458000e-05, 1.815000e-05],
23 [ 1.858000e-05, 2.315000e-05],
24 [ 2.358000e-05, 2.915000e-05],
25 [ 2.958000e-05, 3.715000e-05],
26 [ 3.758000e-05, 4.715000e-05],
27 [ 4.758000e-05, 6.015000e-05],
28 [ 6.056500e-05, 7.616500e-05],
29 [ 7.656500e-05, 9.616500e-05],
30 [ 9.656500e-05, 1.211650e-04],
31 [ 1.215650e-04, 1.521650e-04],
32 [ 1.525650e-04, 1.921650e-04],
33 [ 1.925650e-04, 2.431650e-04],
34 [ 2.435650e-04, 3.061650e-04],
35 [ 3.065650e-04, 3.871650e-04],
36 [ 3.875650e-04, 4.881650e-04],
37 [ 4.885650e-04, 6.151650e-04],
38 [ 6.155650e-04, 7.761650e-04],
39 [ 7.765650e-04, 9.781650e-04],
40 [ 9.785650e-04, 1.233165e-03],
41 [ 1.233565e-03, 1.555165e-03]]
42 centers: [-1.135000E-06, 3.650000E-07, 2.365000E-06, 4.365000E-06, 6.365000E-06, 8.365000E-06, 1.036500E-05, 1.286500E-05, 1.636500E-05, 2.086500E-05, 2.636500E-05, 3.336500E-05, 4.236500E-05, 5.386500E-05, 6.836500E-05, 8.636500E-05, 1.088650E-04, 1.368650E-04, 1.723650E-04, 2.178650E-04, 2.748650E-04, 3.468650E-04, 4.378650E-04, 5.518650E-04, 6.958650E-04, 8.773650E-04, 1.105865E-03, 1.394365E-03]
43 hm_gate_times:
44 standard_name: hm_gate_times
45 long_name: calibrated high moment gate times
46 units: seconds
47 missing_value: not_defined
48 bounds: [[2.85800e-05, 2.91500e-05],
49 [2.95800e-05, 3.11500e-05],
50 [3.15800e-05, 3.31500e-05],
51 [3.35800e-05, 3.51500e-05],
52 [3.55800e-05, 3.71500e-05],
53 [3.75800e-05, 3.91500e-05],
54 [3.95800e-05, 4.11500e-05],
55 [4.15800e-05, 4.41500e-05],
56 [4.45800e-05, 4.81500e-05],
57 [4.85800e-05, 5.31500e-05],
58 [5.35800e-05, 5.91500e-05],
59 [5.95800e-05, 6.71500e-05],
60 [6.75800e-05, 7.71500e-05],
61 [7.75800e-05, 9.01500e-05],
62 [9.05800e-05, 1.06150e-04],
63 [1.06580e-04, 1.26150e-04],
64 [1.26580e-04, 1.51150e-04],
65 [1.51580e-04, 1.82150e-04],
66 [1.82580e-04, 2.22150e-04],
67 [2.22580e-04, 2.73150e-04],
68 [2.73580e-04, 3.36150e-04],
69 [3.36580e-04, 4.17150e-04],
70 [4.17580e-04, 5.18150e-04],
71 [5.18580e-04, 6.45150e-04],
72 [6.45580e-04, 8.06150e-04],
73 [8.06580e-04, 1.00815e-03],
74 [1.00858e-03, 1.26315e-03],
75 [1.26358e-03, 1.58515e-03],
76 [1.58558e-03, 1.99115e-03],
77 [1.99158e-03, 2.50215e-03],
78 [2.50258e-03, 3.14815e-03],
79 [3.14858e-03, 3.94015e-03]]
80 centers: [2.886500E-05, 3.036500E-05, 3.236500E-05, 3.436500E-05, 3.636500E-05, 3.836500E-05, 4.036500E-05, 4.286500E-05, 4.636500E-05, 5.086500E-05, 5.636500E-05, 6.336500E-05, 7.236500E-05, 8.386500E-05, 9.836500E-05, 1.163650E-04, 1.388650E-04, 1.668650E-04, 2.023650E-04, 2.478650E-04, 3.048650E-04, 3.768650E-04, 4.678650E-04, 5.818650E-04, 7.258650E-04, 9.073650E-04, 1.135865E-03, 1.424365E-03, 1.788365E-03, 2.246865E-03, 2.825365E-03, 3.544365E-03]
81
82 n_loop_vertices:
83 standard_name: number_of_loop_vertices
84 long_name: number of loop vertices
85 units: not_defined
86 missing_value: not_defined
87 length: 8
88
89 xyz:
90 standard_name: xyz_coordinates
91 long_name: coordinates of the loop vertices
92 units: not_defined
93 missing_value: not_defined
94 length: 3
95
96 variables:
97
98 data_normalized: True
99 skytem_skb_gex_available: True
100 reference_frame: right-handed positive down
101 coil_orientations: X, Z
102
103 transmitter:
104 label: [LM, HM]
105 number_of_turns: [1, 4]
106 coordinates:
107 values: [[[-12.64,-2.10,0.00],[-6.14,-8.58,0.00],[6.14,-8.58,0.00],[11.41,-3.31,0.00],[11.41,3.31,0.00],[6.14,8.58,0.00],[-6.14,8.58,0.00],[-12.64,2.10,0.00]],
108 [[-12.64,-2.10,0.00],[-6.14,-8.58,0.00],[6.14,-8.58,0.00],[11.41,-3.31,0.00],[11.41,3.31,0.00],[6.14,8.58,0.00],[-6.14,8.58,0.00],[-12.64,2.10,0.00]]]
109 dimensions: ['n_transmitter', 'n_loop_vertices', 'xyz']
110 area: [342, 342]
111 waveform_type: [trapezoid, trapezoid]
112 waveform_time:
113 values: [[-3.1810E-003, -3.1019E-003, -2.9844E-003, -2.3810E-003, -2.3781E-003, -2.3779E-003, -2.3776E-003, -2.3763E-003, -8.0000E-004, -7.2093E-004, -6.0345E-004, 0.0000E+000, 3.0000E-008, 7.0000E-008, 2.7200E-006, 2.8000E-006, 2.9000E-006, 3.0100E-006, 3.1300E-006, 3.4100E-006, 4.7400E-006],
114 [-6.9167E-02, -6.9157E-02, -6.9153E-02, -6.9150E-02, -6.9143E-02, -6.9122E-02, -6.9118E-02, -6.9114E-02, -6.9107E-02, -6.9083E-02, -6.8159E-02, -6.6667E-02, -6.6627E-02, -6.6626E-02, -6.6622E-02, -2.5000E-03, -2.4899E-03, -2.4862E-03, -2.4830E-03, -2.4767E-03, -2.4637E-03, -2.4547E-03, -2.4510E-03, -2.4475E-03, -2.4442E-03, -2.4406E-03, -2.4159E-03, -2.2328E-03, -1.4913E-03, 0.0000E+00, 6.4270E-07, 8.9870E-07, 1.4267E-05, 4.0291E-05, 4.1331E-05, 4.4539E-05]]
115 long_name: waveform time
116 missing_value: not_defined
117 units: s
118 waveform_current:
119 values: [[-0.0000E+000, -1.4067E-001, -3.0174E-001, -1.0000E+000, -7.5094E-003, 2.2879E-002, 3.7669E-002, -0.0000E+000, 0.0000E+000, 1.4063E-001, 3.0168E-001, 1.0000E+000, 9.9851E-001, 9.8817E-001, 5.9260E-002, 3.2392E-002, 7.5094E-003, -1.2284E-002, -2.6411E-002, -3.8086E-002, 0.0000E+000],
120 [-0.0000E+00, -3.3580E-02, -6.8755E-02, -1.0992E-01, -2.4885E-01, -7.3516E-01, -8.1234E-01, -8.6553E-01, -9.0296E-01, -9.2188E-01, -9.6364E-01, -1.0000E+00, -8.2124E-03, 7.2510E-03, -0.0000E+00, 0.0000E+00, 3.3780E-02, 6.5400E-02, 1.0996E-01, 2.3303E-01, 5.4048E-01, 7.4152E-01, 8.1301E-01, 8.6142E-01, 8.8900E-01, 9.0249E-01, 9.2195E-01, 9.3742E-01, 9.6367E-01, 1.0000E+00, 9.9562E-01, 9.8391E-01, 6.4740E-01, 9.9177E-04, -1.1094E-02, 0.0000E+00]]
121 dimensions: ['n_transmitter', 'waveform_time']
122 current_scale_factor: 1.0
123 peak_current: [9.0, 110.0]
124 base_frequency: [210.0, 75.0]
125 on_time: [800E-06, 2500e-6]
126 off_time: [1581E-06, 4167e-6]
127 orientation: [z, z]
128
129 receiver:
130 label: [z, x]
131 orientation: [z, x]
132 coil_low_pass_filter: [628000.0, 250000.0]
133 instrument_low_pass_filter: [500000.0, 500000.0]
134 area:
135 values: [105.0, 115.0]
136 units: m^2
137
138 couplet:
139 transmitters: [lm, hm, lm, hm]
140 receivers: [z, z, x, x]
141 txrx_dx: [-13.25, -13.25, -14.65, -14.65]
142 txrx_dy: [0.0, 0.0, 0.0, 0.0]
143 txrx_dz: [-2.0, 0.0, -2.0, 0.0]
144 data_type: [dBdt, dBdt, dBdt, dBdt]
145 gate_times: [LM_gate_times, HM_gate_times, LM_gate_times, HM_gate_times]
146
147magnetic_system:
148 type: system
149 mode: airborne
150 method: magnetic
151 instrument: Geometrics G-822A cesium‑vapor magnetometer
152
153 prefixes: ['base_magnetometer']
154
155 dimensions:
156 base_mag_locations:
157 standard_name: base_mag_locations
158 long_name: Base Magnetometer Location Index Numbers
159 units: not_defined
160 missing_value: not_defined
161 centers: [1, 2]
162 discrete: True
163
164 variables:
165
166 transmitter:
167 label: passive
168 description: No artificial magnetic transmitter was used. The system measures the scalar Larmor precession frequency induced by the Earth's magnetic field.
169
170 receiver:
171 label: scalar_magnetometer
172 sensor_type: cesium-vapor split-beam
173 sensor_model: G-822A
174 sensor_manufacturer: Geometrics
175 description: Scalar cesium-vapor magnetometer mounted in the aircraft tail stinger. Measures total magnetic field through Larmor precession frequency.
176 orientation: Tail-stinger mounted; scalar measurement independent of orientation.
177 coordinates: not_reported
178 lag_correction: Lag was negligible and no lag correction was applied
179 heading_correction: Heading was negligible and no heading correction was applied
180
181 couplet:
182 passive_scalar_pair:
183 transmitters: [passive]
184 receivers: [scalar_magnetometer]
185 description: The magnetic measurement system consists of the Earth's field as a passive transmitter and a single scalar magnetometer mounted in the tail stinger.
186
187 base_magnetometer:
188 label: base_magnetometer
189 description: The base station magnetometer was placed in a location of low magnetic gradient, away from electrical transmission lines and moving metallic objects, such as motor vehicles and aircrafts.
190
191 location_names:
192 values: ["Door County", "Manitowoc"]
193 dimensions: 'base_mag_locations'
194
195 values:
196 values: [54538, 54194.7]
197 units: nT
198 dimensions: 'base_mag_locations'
199
200 latitude:
201 values: [44.849335, 44.127998]
202 long_name: Latitude in WGS84
203 units: decial degrees
204 dimensions: 'base_mag_locations'
205
206 longitude:
207 values: [87.422440, 87.685524]
208 long_name: Longitude in WGS84
209 units: decial degrees
210 dimensions: 'base_mag_locations'
211
212 elevation:
213 values: [178.1, 164.4]
214 long_name: Elevation
215 datum: WGS84
216 units: m
217 dimensions: 'base_mag_locations'
218
219 diurnal_correction: Diurnal signal removed using 3 second Fraser low-pass filter and subtracting base-station magnetometer values.
220 tieline_levelling: No tie line-leveling were applied
221 microlevelling: No micro-levelling were applied
222 igrf_model_date: "2015, 15th generation"
223 igrf_model_location: variable according to GPS WGS84 longitude and latitude
224 igrf_model_height: variable according to magnetic sensor altitude derived from DGPS data
See the page on Systems for more information.
Parameters
Additional metadata information, such as inversion settings and parameters, can be added as a Supplementary Leaflet group.
1inversion_parameters:
2 dataset_attrs:
3 type: parameters
4 method: electromagnetic, time domain
5 instrument: 30Hz Tempest
6 mode: airborne
7 property: electrical conductivity
8
9 variables:
10 software: GALEISBSTDEM 1-D time-domain deterministic inversion software
11 software_reference: "Brodie, R. C., 2015, GALEISBSTDEM: A deterministic algorithm for 1D sample by sample inversion of time-domain AEM data – theoretical details, accessed May 1, 2020, at https://github.com/GeoscienceAustralia/ga-aem/blob/master/docs/GALEISBSTDEM%20Inversion%20Algorithm%20Theoretical%20Details%20.pdf."
12 description: Inversions were done using a multilayered smooth model formulation in which 30 layer thicknesses were fixed and layer conductivities were solved for. Horizontal (X) and vertical (Z) components of the data were inverted separately. A vertical conductivity smoothing constraint, alpha_s = 1000, was applied. The inversion reference model used a half-space conductivity of 0.04 Siemens per meter (S/m) with a standard deviation of 1 S/m. The relative importance of the reference conductivity model, alpha_c, was set to 1.0. The horizontal and vertical separation between transmitter and receiver was given a lateral and vertical standard deviation constraint of 0.5 meters (m) in the reference model. The receiver pitch was also included with a 0.5 m standard deviation. These steps were repeated using the same inversion parameters but for reference models of higher (0.2 S/m) and lower (0.008 S/m) conductivity representing lower (5 Ohm-m) and higher (125) resistivity, respectively. A number of inversions were conducted with various homogenous prior model values, and constraints on resistivity. The final model parameters described above were selected because they best represent the physical understanding of the system and minimized data misfit. Final inverted resistivity values for each layer, layer thicknesses, and the uncertainty associated with these values can be found in the model dataset.
13 doi_calculation: The depth of investigation (DOI) for each model location was calculated using the difference between the low and high reference conductivity model results. Using the approach from Oldenburg and Li (1999), models from the low and high reference inversions were divided and rescaled producing a metric of their similarity. Models were similar where constrained by the data (shallow depths) and diverge back to their distinct reference model values when no longer constrained by the data. Therefore, the DOI was calculated as the threshold below which models were no longer informed by the data.
14 phid_cut: Individual models with a data misfit, "PhiD", less than or equal to 1.5 were accepted for final outputs and products. A new channel, "ACCEPT_FLAG" was added to the data file representing this misfit cutoff, with 0 = rejected models and 1 = accepted models.
See the page on Parameters for more information.