EVOLUTION-MANAGER

Edit File: frmt_netcdf_vector.html

<html>
<head>
<title>NetCDF network Common Data Form - Vector</title>
</head>

<h1>NetCDF: Network Common Data Form - Vector</h1>

Starting with GDAL 2.1, the netCDF driver support read and write (creation from
scratch and append operations) to vector datasets (you can find documentation for the
<a href="frmt_netcdf.html">raster side</a>)<p>

NetCDF is an interface for
array-oriented data access and is used for representing scientific data.<p>

The driver handles the "point" and "profile"
<a href="http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html#_features_and_feature_types">feature types</a>
of the CF 1.6 convention. It also supports a more custom approach for non-point
geometries.<p>

<h2>Mapping of concepts</h2>

<h3>Field types</h3>

On creation of netCDF files, the mapping between OGR field types and netCDF type is the following :<p>

<table border="1">
<tr><th>OGR field type</th>     <th>netCDF type</th></tr>
<tr><td>String(1)</td>          <td>char</td></tr>
<tr><td>String</td>             <td>char (bi-dimensional), or string for NC4</td></tr>
<tr><td>Integer</td>            <td>int</td></tr>
<tr><td>Integer(Boolean)</td>   <td>byte</td></tr>
<tr><td>Integer(Int16)</td>     <td>short</td></tr>
<tr><td>Integer64</td>          <td>int64 for NC4, or double for NC3 as a fallback</td></tr>
<tr><td>Real</td>               <td>double</td></tr>
<tr><td>Real(Float32)</td>      <td>float</td></tr>
<tr><td>Date</td>               <td>int (with units="days since 1970-1-1")</td></tr>
<tr><td>DateTime</td>           <td>double (with units="seconds since 1970-1-1 0:0:0")</td></tr>
</table>

<p>The driver also writes the following attributes for each OGR fields / netCDF variables.</p>
<ul>
<li><i>ogr_field_name</i>: OGR field name (useful if the netCDF variable name is different, due to collision)</li>
<li><i>ogr_field_type</i>: OGR field type (such as String,Integer,Date,DateTime,etc...)</li>
<li><i>ogr_field_width</i>: OGR field width. Only set if it is non-zero, except for strings</li>
<li><i>ogr_field_precision</i>: OGR field precision. Only set if it is non-zero</li>
</ul>
<p>
They are written by default (unless the <a href="#WRITE_GDAL_TAGS">WRITE_GDAL_TAGS</a>
dataset creation option is set to NO). They are not required for reading, but
may help to better identify field characteristics
<p>
On reading, the mapping is the following :
</p>

<table border="1">
<tr><th>netCDF type</th>        <th>OGR field type</th></tr>
<tr><td>byte</td>               <td>Integer</td></tr>
<tr><td>ubyte (NC4 only)</td>   <td>Integer</td></tr>
<tr><td>char (mono dimensional) <td>String(1)</td></tr>
<tr><td>char (bi dimensional)   <td>String</td></tr>
<tr><td>string (NC4 only)       <td>String</td></tr>
<tr><td>short</td>              <td>Integer(Int16)</td></tr>
<tr><td>ushort (NC4 only)</td>  <td>Integer</td></tr>
<tr><td>int</td>                <td>Integer</td></tr>
<tr><td>int or double (with units="days since 1970-1-1")</td><td>Date</td>
<tr><td>uint (NC4 only)</td>    <td>Integer64</td></tr>
<tr><td>int64 (NC4 only)</td>   <td>Integer64</td></tr>
<tr><td>uint64 (NC4 only)</td>  <td>Real</td></tr>
<tr><td>float</td>              <td>Real(Float32)</td>
<tr><td>double</td>             <td>Real</td>
<tr><td>double (with units="seconds since 1970-1-1 0:0:0")</td><td>DateTime</td>
</table>

<h3>Layers</h3>

<p>
Generally a single netCDF file is viewed as a single OGR layer, provided that
it contains only mon-dimensional variables, indexed by the same dimension (or
bi-dimensional variables of type char). For netCDF v4 files with multiple groups,
each group may be seen as a separate OGR layer.</p>

<p>On writing, the <a href="#MULTIPLE_LAYERS">MULTIPLE_LAYERS</a> dataset creation
option can be used to control whether multiple layers is disabled, or if multiple
layers should go in separate files, or separate groups.</p>

<h3>Strings</h3>

<p>Variable length strings are not natively supported in netCDF v3 format. To work around that,
OGR uses bi-dimensional char variables, whose first dimension is the record dimension,
and second dimension the maximum width of the string. By default, OGR implements a
"auto-grow" mode in writing, where the maximum width of the variable used to store a
OGR string field is extended when needed. Note that this leads to a full rewrite of
already written records : this is transparent for the user, but can slow down the
creation process in non-linear ways. A similar mechanism is used to handle layers
with geometry types other than point to store the ISO WKT representation of the geometries.</p>

<p>When using a netCDF v4 output format (NC4), strings will be by default written
as netCDF v4 variable length strings.</p>

<h3>Geometry</h3>

<p>Layers with a geometry type of Point or Point25D will cause the implicit creation of
x,y(,z) variables for projected coordinate system, or lon,lat(,z) variables for
geographic coordinate systems. For other geometry type, a variable "ogc_wkt" (
bi-dimensional char for NC3 output, or string for NC4 output) is created and
used to store the geometry as a ISO WKT string.</p>

<h3><a name="profile">"Profile" feature type</a></h3>

<p>The driver can handle "profile" feature type, i.e. phenomenons that happen at
a few positions along a vertical line at a fixed horizontal position. In that representation,
some variables are indexed by the profile, and others by the observation.<p>

<p>More precisely, the driver supports reading and writing profiles organized accordingly
with the "<a href="http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html#_indexed_ragged_array_representation_of_profiles">Indexed ragged array representation</a>"
of profiles.</p>

<p>On reading, the driver will collect values of variables indexed by the profile
dimension and expose them as long as variables indexed by the observation dimension,
based on a variable such as "parentIndex" with an attribute "instance_dimension"
pointing to the profile dimension.</p>

<p>On writing, the <a href="#FEATURE_TYPE">FEATURE_TYPE</a>=PROFILE layer creation
option must be set and the driver will need to be instructed which OGR fields are indexed
either by the profile or by the observation dimension. The list of fields indexed by the
profile can be specified with the <a href="#PROFILE_VARIABLES">PROFILE_VARIABLES</a>
layer creation options (other fields are assumed to be indexed by the observation dimension).
Fields indexed by the profile are the horizontal geolocation (created implicitly), and
other user attributes such as the location name, etc. Care should be taken into selecting
which variables are indexed by the profile dimension: given 2 OGR features (taking into
account only the variables indexed by the profile dimension), if they have different
values for such variables, they will be considered to belong to different profiles.</p>

<p>In the below example, the station_name and time variables may be indexed by the
profile dimension (the geometry is assumed to be also indexed by the profile dimension),
since all records that have the same value for one of those variables have same values
for the other ones, whereas temparature and Z should be indexed by the default dimension.
</p>

<p>
<table border="1">
<tr><th>station_name</th> <th>time</th>                  <th>geometry</th>            <th>temperature</th> <th>Z</th></tr>
<tr><td>Paris</td>        <td>2016-03-01T00:00:00Z</td>  <td>POINT (2 49)</td>        <td>25</td>          <td>100</td></tr>
<tr><td>Vancouver</td>    <td>2016-04-01T12:00:00Z</td>  <td>POINT (-123 49.25)</td>  <td>5</td>           <td>100</td></tr>
<tr><td>Paris</td>        <td>2016-03-01T00:00:00Z</td>  <td>POINT (2 49)</td>        <td>3</td>           <td>500</td></tr>
<tr><td>Vancouver</td>    <td>2016-04-01T12:00:00Z</td>  <td>POINT (-123 49.25)</td>  <td>-15</td>         <td>500</td></tr>
</table>
</p>

<p>An integer field, with the name of the profile dimension (whose default name is "profile", which
can be altered with the <a href="#PROFILE_DIM_NAME">PROFILE_DIM_NAME</a> layer creation option),
will be used to store the automatically computed id of profile sites (unless a integer OGR field with the
same name exits).</p>

<p>The size of the profile dimension defaults to 100 for non-NC4 output format,
and is extended automatically in case of additional profiles (with similar performance
issues as growing strings). For NC4 output format, the profile dimension is of
unlimited size by default.</p>

<h2>Dataset creation options</h2>

<ul>
<li><b>FORMAT</b>=NC/NC2/NC4/NC4C: netCDF format. NC is the classic netCDF format
(compatible of netCDF v3.X and 4.X libraries). NC2 is the extension of NC for
files larger than 4 GB. NC4 is the netCDF v4 format, using a HDF5 container, offering
new capabilities (new types, concept of groups, etc...) only available in netCDF v4 library.
NC4C is a restriction of the NC4 format to the concepts supported by the classic netCDF
format. Default is NC.</li>
<li><b><a name="WRITE_GDAL_TAGS">WRITE_GDAL_TAGS</a></b>=YES/NO: Whether to write GDAL specific information as
netCDF attributes. Default is YES.</li>
<li><b><a name="MULTIPLE_LAYERS">MULTIPLE_LAYERS</a></b>=NO/SEPARATE_FILES/SEPARATE_GROUPS. Default is NO, i.e a
dataset can contain only a single OGR layer. SEPARATE_FILES can be used to put the content
of each OGR layer in a single netCDF file, in which case the name passed at dataset
creation is used as the directory, and the layer name is used as the basename of the netCDF
file. SEPARATE_GROUPS may be used when FORMAT=NC4 to put each OGR layer in a separate
netCDF group, inside the same file.</li>
<li><b>CONFIG_FILE</b>=string. Path to a <a href="#XML_config">XML configuration file</a>
(or its content inlined) for precise control of the output</li>
</ul>

<h2>Layer creation options</h2>

<ul>
<li><b>RECORD_DIM_NAME</b>=string. Name of the unlimited dimension that index features.
Defaults to "record"</li>
<li><b>STRING_DEFAULT_WIDTH</b>=int. Default width of strings (when using
bi-dimensional char variables). Default is 10 in autogrow mode, 80 otherwise.</li>
<li><b>USE_STRING_IN_NC4</b>=YES/NO. Whether to use NetCDF string type for strings
in NC4 format. If NO, bidimensional char variable are used. Default to YES when
FORMAT=NC4.</li>
<li><b>WKT_DEFAULT_WIDTH</b>=int. Default width of WKT strings (when using
bi-dimensional char variables). Default is 1000 in autogrow mode, 10000 otherwise.</li>
<li><b>AUTOGROW_STRINGS</b>=YES/NO. Whether to auto-grow string fields of non-fixed width,
or ogc_wkt special field, when serialized as bidimensional char variables.
Default is YES. When set to NO, if the string is larger than its maximum initial
width (set by STRING_DEFAULT_WIDTH), it is truncated. For a geometry, it is completely
discarded.</li>
<li><b><a name="FEATURE_TYPE">FEATURE_TYPE</a></b>=AUTO/POINT/PROFILE. Select the CF FeatureType. Defaults to
AUTO where FeatureType=Point is selected if the layer geometry type is Point, otherwise
the custom approach involving the "ogc_wkt" field is used. Can be set to <a href="#profile">PROFILE</a> so
as to select the creation of an indexed ragged array representation of profiles.</li>
<li><b><a name="PROFILE_DIM_NAME">PROFILE_DIM_NAME</a></b>=string. Name of the
profile dimension and variable. Defaults to "profile". Only used when FEATURE_TYPE=PROFILE.</li>
<li><b>PROFILE_DIM_INIT_SIZE</b>=int or string. Initial size of profile dimension,
or UNLIMITED for NC4 files. Defaults to 100 when FORMAT != NC4 and to UNLIMITED when
FORMAT = NC4. Only used when FEATURE_TYPE=PROFILE.</li>
<li><b><a name="PROFILE_VARIABLES">PROFILE_VARIABLES</a></b>=string. Comma separated
list of field names that must be indexed by the profile dimension. Only used when FEATURE_TYPE=PROFILE.</li>
</ul>

<h2><a name="XML_config">XML configuration file</a></h2>

<p>
A XML configuration file conforming to the following
<a href="https://svn.osgeo.org/gdal/trunk/gdal/data/netcdf_config.xsd">schema</a>
can be used for very precise control on the output format, in particular to set
all needed attributes (such as units) to conform to the
<a href="http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html">NetCDF CF-1.6 convention</a>.</p>

<p>It has been designed in particular, but not exclusively, to be usable in use cases involving the
<a href="http://mapserver.org/output/ogr_output.html">MapServer OGR output</a>.</p>

<p>Such a file can be used to :
<ul>
<li>set dataset and layer creation options.</li>
<li>set global netCDF attributes.</li>
<li>map OGR field names to netCDF variable names.</li>
<li>set netCDF attributes attached to netCDF variables.</li>
</ul>
</p>

<p>The scope of effect is either globally, when elements are defined as direct
children of the root &lt;Configuration&gt; node, or specifically to a given layer,
when defined as children of a &lt;Layer&gt; node.<p>

<p>The filename is specified with the CONFIG_FILE dataset creation option. Alternatively,
the content of the file can be specifid inline as the value of the option (it must
then begin strictly with the "&lt;Configuration" characters)</p>

<p>The following example shows all possibilities and precedence rules:</p>
<pre>
&lt;Configuration&gt;
    &lt;DatasetCreationOption name="FORMAT" value="NC4"/&gt;
    &lt;DatasetCreationOption name="MULTIPLE_LAYERS" value="SEPARATE_GROUPS"/&gt;
    &lt;LayerCreationOption name="RECORD_DIM_NAME" value="observation"/&gt;
&lt;!-- applies to all layers --&gt;
    &lt;Attribute name="copyright" value="Copyright(C) 2016 Example"/&gt;
    &lt;Field name="weight"&gt;  &lt;!-- edit user field/variable --&gt;
        &lt;Attribute name="units" value="kg"/&gt; 
        &lt;Attribute name="maximum" value="10" type="double"/&gt;
    &lt;/Field&gt;
    &lt;Field netcdf_name="z"&gt; &lt;!-- edit predefined variable --&gt;
        &lt;Attribute name="long_name" value="Elevation"/&gt; 
    &lt;/Field&gt;
&lt;!-- start of layer specific definitions --&gt;
    &lt;Layer name="1st_layer" netcdf_name="firstlayer"&gt; &lt;!-- OGR layer "1st_layer" is renamed as "firstlayer" netCDF group --&gt;
        &lt;LayerCreationOption name="FEATURE_TYPE" value="POINT"/&gt;
        &lt;Attribute name="copyright" value="Public domain"/&gt; &lt;!-- override global one --&gt;
        &lt;Attribute name="description" value="This is my first layer"/&gt; &lt;!-- additional attribute --&gt;
        &lt;Field name="1st_field" netcdf_name="firstfield"/&gt; &lt;!-- rename OGR field "1st_field" as the "firstfield" netCDF variable --&gt;
        &lt;Field name="weight"/&gt; &lt;!-- cancel above global customization --&gt;
        &lt;Field netcdf_name="lat"&gt; &lt;!-- edit predefined variable --&gt;
            &lt;Attribute name="long_name" value=""/&gt; &lt;!-- remove predefined attribute --&gt;
        &lt;/Field&gt;
    &lt;/Layer&gt;
    &lt;Layer name="sounding"&gt;
        &lt;LayerCreationOption name="FEATURE_TYPE" value="PROFILE"/&gt;
        &lt;Field name="station_name" main_dim="profile"/&gt; &lt;!-- the corresponding netCDF variable will be indexed against the profile dimension, instead of the observation dimension --&gt;
        &lt;Field name="time" main_dim="profile"/&gt; &lt;!-- the corresponding netCDF variable will be indexed against the profile dimension, instead of the observation dimension --&gt;
    &lt;/Layer&gt;
&lt;/Configuration&gt;
</pre>

<p>The effect on the output can be checked by running the <b>ncdump</b> utility</p>

<h2>See Also:</h2>
<ul>
<li><a href="frmts_netcdf.html">Raster side of the netCDF driver.</a></li>
<li><a href="http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html">NetCDF CF-1.6 convention</a></li>
<li><a href="http://www.unidata.ucar.edu/downloads/netcdf/index.jsp">NetCDF compiled libraries</a></li>
<li><a href="http://www.unidata.ucar.edu/software/netcdf/docs/">NetCDF Documentation</a></li>
</ul>

<h2>Credits</h2>

<p>Development of the read/write vector capabilities for netCDF was funded by
<a href="https://www.ec.gc.ca/meteo-weather/">Meteorological Service of Canada</a>
and <a href="http://woudc.org">World Ozone and Ultraviolet Radiation Data Centre</a>.</p>

</body>
</html>