EDEX D2D HDF5 File Format

March 28, 2012

This document describes the format of the HDF5 file that is:

read by EDEX as a D2D database

If you are interested in the format of the netCDF files generated from the ifpnetCDF product formatter, or read by the iscMosaic program , then refer to the netCDF File Format Document.

When data arrives over the SBN, the EDEX JVM ingest thread reads the next available message in AMQP queue from the QPID message broker. It compares against the distribution XML files loaded into memory on EDEX startup from /awips2/edex/data/utility/edex_static/base/distribution. If no pattern matches, nothing is done. Otherwise a metadata record is generated in the appropriate table of the metadata database, and an HDF5 file is written to /awips2/edex/data/hdf5/grib/MODEL_NAME on dx1. SmartInit is kicked off by EDEX and generates data in /awips2/edex/data/hdf5/gfe/SITE_ID. You can access the metadata database on dx1 as user awips. You can view the contents of HDF5 files using either the free HDFView utility, or h5dump (installed in /awips2/tools/bin).

About h5dump

OPTIONS

-h, --help Print a usage message and exit -B, --userblock Print the content of the user block -H, --header Print the header only; no data is displayed -i, --object-ids Print the object ids -V, --version Print version number and exit -a P, --attribute=P Print the specified attribute -d P, --dataset=P Print the specified dataset -g P, --group=P Print the specified group and all members -l P, --soft-link=P Print the value(s) of the specified soft link -o F, --output=F Output raw data into file F -t P, --datatype=P Print the specified named data type -w N, --width=N Set the number of columns of output -x, --xml Output in XML -D U, --xml-dtd=U Use the DTD at U

Subsetting is available by using the following options with a dataset attribute. Subsetting is done by selecting a hyperslab from the data. Thus, the options mirror those for performing a hyperslab selection. The START and COUNT parameters are mandatory if you do subsetting. The STRIDE and BLOCK parameters are optional and will default to 1 in each dimension.

-s L, --start=L Offset of start of subsetting selection -S L, --stride=L Hyperslab stride -c L, --count=L Number of blocks to include in selection -k L, --block=L Size of block in hyperslab P - is the full path from the root group to the object. F - is a filename. N - is an integer greater than 1. L - is a list of integers the number of which are equal to the number of dimensions in the dataspace being queried U - is a URI reference Examples: 1) Attribute foo of the group /bar_none in file quux.h5 h5dump -a /bar_none/foo quux.h5 2) Selecting a subset from dataset /foo in file quux.h5 h5dump -d /foo -s "0,1" -S "1,1" -c "2,3" -k "2,2" quux.h5

Diagnosing Model Problems

If you point the volume browser to a D2D directory and the model is not appearing, verify that:

EDEX started up without any errors that would affect gfe, i.e., all configurations loaded normally.
Ingest processes started normally and are up and running.
PyPies are up and running on dx1.
the file name is in the standard format of yyyymmdd_hhmm, which is the model base time
that the file contains all of the required variables, dimensions, and attributes as outlined above
that there isn't more than ONE directory in the D2DDIRS configuration containing files that have an variable called "model" with the same name

If all else fails, you can turn on the debugging logging option in EDEX. In this mode EDEX will log all problems it finds with model data files. To turn on this feature, perform the following steps:

Edit the file /awips2/edex/conf/log4j-ingest.xml to set level value = "DEBUG" for the appender ThreadBasedLog. Refer to the log files document for details.
Stop and restart EDEX.
Look in the server log files. It will show the problems for all files it finds. If it doesn't identify your file or directory, re-check your D2DDIRS to be sure you have set it up correctly.