ifpnetCDF Formatter User's Guide

January 6, 2012


Table of Contents

Overview

Running the ifpnetCDF Program

Organization of the Gridded Data

Additional Data Fields

Converting to real values from the "-k krunch" option

Configuration Interval File

Overview

The ifpnetCDF program reads gridded data from EDEX that you specify and writes the data to a netCDF file. In addtion to the forecast grids, a topography grid, latitude grid, or longitude grid can be written to the file using the -g option. Virtually all of the data describing each weather element is written to the file as well, including the grid size, projection and domain information, units, and valid times for each grid. The following sections describe how to use this program and document the structure of the resulting netCDF file. A special switch is available (-k) to "squish" down the formats of the data file to as compact as possible.

A special configuration file may be used to "sample" the grids and not take each grid in the inventory.  This has the benefit of dramatically cutting down on the bandwidth required to transmit the netCDF file when only a portion of the inventory is required.


Running the ifpnetCDF Program

The program is capable of both storing and retrieving grids from EDEX.
The command line syntax is:
ifpnetCDF -o outputFile -h hostname -r rpcPortNumber -d databaseID [-p parmID] [-s startTime] [-e endTime] [-m editAreaName] [-u user] [-g] [-t] [-C configurationIntervalName]
A sample command line is:
ifpnetCDF -o myGridFile -h dx3-oax -r 9581 -d TEST_GRID__Fcst_00000000_0000 -p T -s 19980604_1200 -e 19980605_0000 -m BOU -g -c -t -C cdfSample

Option Syntax Optional or Mandatory Description
-o filename
Optional
 Specifies the name of the output file. If no output file is specified the data will be written to the file ifpnetCDFFile.cdf
-h hostname
Mandatory (See Note)
 Specifies the EDEX host from which the data will be extracted.
-r port
Mandatory (See Note)
 Specifies the EDEX host from which the data will be extracted.
-u user
Optional
 Specifies the user for connection to the server. Defaults to SITE. Important only when using edit area masks and the mask hasn't been defined in BASE or SITE.
-d databaseID
Mandatory
 Source database identifier from which to get the data.  There may be only one databaseID specified.  The format of the database identifier is: site_GRID_optType_modelName_modelRunTime. For example: DEN_GRID__eta_19980604_1200. (Note double underscore after GRID for databases that don't have the optional type)
-p weatherElementName
Optional
 If no -p switches are present, then all weather elements in the specified databases will be processed.  There may be several -p switches present if desired. For example: -p T -p Td -p Wind.  The weather element name refers to the SFC level, unless the weather element name contains an underscore.  To specify a level other than SFC, use the format weatherElementName_level, such as T_3K.
-s startTime
Optional
 Specifies the start time for the range of grids to extract. If no time is specified, then assume "from the beginning of time". Format is yyyyMMDD_HHMM or 20010604_1200.
-e endTime
Optional
 Specifies the ending time for the range of grids to extract. If no time is specified, then assume "to the end of time". Format is yyyyMMDD_HHMM or 20010604_1200.
 -m mask
Optional
 Specifies an edit area outside of which the grids are clipped with a fill value. If the edit area is not specified or is not found in EDEX, the entire grid will be written to the file unclipped.
 -g
Optional
 When specified, a grid of topography, latitude, and longitude are also written to the netCDF file.
-c
Optional
 When specified, the output will be compressed. The compression factor to gzip defaults to 6; this can be changed through the -f switch.
-f factor
Optional
 When specified with the -c switch, this defines the gzip factor to be applied to the compression. Defaults to 6. Can range from 1 to 9 with 9 being the maximum, but slowest, compression.
-t
Optional
 When specified, data values will be "trimmed" or rounded to the precision of the data, as defined by EDEX.
-k
Optional
 The -t switch must also be given if the -k functionality is desired.  The -k switch reformats the data values into smaller units, as as converting a 32-bit float into a 8-bit byte, in order to save space.  When the -k switch is given and the data has been reformatted, two new attributes provided (dataMultipler and dataOffset) which are used to convert the data back into real numbers.
-C configIntervalName
 Optional
 Specifies a configuration interval file which controls the interval/spacing of the grids.  The name identifies a file within the EDEX textUtilities directory and must be a Python file in the correct format.
Note: The -h and -r switches are predefined if you are running ifpnetCDF in the GFESuite installed environment.  The values specified in the installation are picked up automatically. You only need to specify them if you want to connect to a different EDEX.   If environment variables ${CDSHOST} or ${CDSPORT} are defined, then the default server and port will be determined from the environment variables, unless overridden with the user specified -h and -r switches.


Organization of the Gridded Data in the netCDF file

The organization of the data is described in the netCDF format documentation.

Additional Data Fields

Topography

The ifpnetCDF program will optionally store a topography grid along with the weather elements that you specify. This grid contains similar attributes and is in units of feet above sea level.

Latitude, Longitude

The ifpnetCDF program will optionally store grids of latitude and longitude at each grid point.  Remember that longitude is expressed as a negative number, if the domain is in the Western Hemisphere.  If the -k switch is given, then you will need to multiply the data values by the dataMultiplier and add the dataOffset, per the format of the netCDF attributes.


Converting to real values from the "-k krunch" option

If the -k krunch value has been given, then the data values in the file need to be converted to actual values through the use of a formula.  The attributes dataMultiplier and dataOffset provide the required information.  If these values are not present, then you may interpret the values directly.

data value = (netCDF value * dataMultiplier) + dataOffset

Configuration Interval File

The configuration interval file is an optional capability of ifpnetCDF.   It can be used to select certain grids to be placed in the ifpnetCDF output file, rather than all grids in the inventory.  For example, you can choose to only include 3-hrly temperature grids out to 24 hours, then 6-hrly temperature grids out to 72 hours, and then no temperature grids past 72 hours.   You can control this capability on a per weather element basis.  The definition determines a set of explicit times.  If there is a grid that contains that explicit time, then the grid is included in the output.

The configuration interval file is a python file and must reside in the EDEX textUtilities directory.  You can create the file through the use of the GFE, with the GFE->Define Text Products menu, or by using a conventional text editor and the ifpServerText utility.

Here is an example configuration interval file for the ifpnetCDF program, which samples the dew point (Td) with basetimes of 3z and 15z, which takes 6 hrly grids of Td up to 72 hours, then no Td grids after that point.  All other weather elements use basetimes of 0z and 12z, and take 3hr grids out to 24hr, 6 hrly grids out to 96 hr, and then 12 hrly grids past this point:

The format is:
SampleDef[weNameAndLevel] = ([listOfbasetimes], [list of offset/intervals]), where
listOfBasetimes is [basetime1, basetime2, basetime3, ...] and
listOfOffset/Intervals is [(offset1, interval1), (offset2, interval2), (offset3, interval3), ...]

HR=3600

SampleDef = {}
SampleDef['default'] = (
   [0*HR, 12*HR],       #first tuple is basetimes
    [                   #2nd tuple is list of offset from basetime, interval
     (0*HR,  3*HR),            #start at basetime, take every 3 hours
     (24*HR, 6*HR),           #24h+ from basetime, take every 6 hours
     (96*HR, 12*HR),          #96h+ from basetime, take every 12 hours
    ])

SampleDef['Td_SFC'] = (
   [3*HR, 15*HR],        #basetimes of 3z and 15z
    [
     (0*HR,  6*HR),            #start at basetime, take every 6 hours
     (72*HR, 1000*HR),         #don't take anything past 72Hr
    ])

The file is a Python dictionary.  A mandatory key is 'default', which applies to all weather elements that are not explicitly specified.  Each entry consists of a tuple.  The first tuple is a list of basetimes, the second tuple is a list of (offset, interval) values.

Item
 Example
 Description
SampleDef = {}
 SampleDef = {}
 Mandatory sampling definition, defines the dictionary.  The ifpnetCDF program looks for the SampleDef variable name.
BaseTimes
 [0*HR, 12*HR]
 List of times relative to 0000z.  The values are specified in seconds, thus be sure to convert the value to hours.  The basetime values should not be negative and must be less than 24*HR.   The basetimes define the "base" of the offset / interval calculations.  For example, if the basetime is set to 0000z, and the offset/interval is set to (0*HR,3*HR), that includes grids that are aligned with 0z, 3z, 6z, 9z, ...   If the basetime was set to 0200z, and the offset/interval set to (0*HR, 3*HR), then grids would be included that are aligned with 2z, 5z, 8z, 11z, ...   There may be more than one basetime specified.
Offset/Interval Times
 [(0*HR, 3*HR), (24*HR, 6*HR)]
 Defines the interval as a series of offset and interval tuples.  The offset is the starting time, offset from the most recent basetime.  The 2nd portion of the tuple defines the interval to use for the grids.
'default'
 SampleDef['default'] = (basetimeList, offset/Interval list))
Defines the interval for all weather elements, unless specified weather elements have their own definition.
'Td_SFC'
 SampleDef['Td_SFC'] = (basetimeList, offset/interval list)
 Defines the interval for weather element "Td_SFC", which is a weather element name, followed by an underscore character, followed by the weather element level.

Common Offset/Interval Examples


Definition
 Basetime Definition
 Offset/Interval Definition
Hourly Grids up to 72 hr from basetime, then no more grids.
 [0*HR]
 [(0*HR, 1*HR), (72*HR, 1000*HR)]
Hourly Grids up to 12 hr, 3 hrly grids to 24 hr, 6 hrly grids after 24hr
 [0*HR]
 [(0*HR, 1*HR), (12*HR, 3*HR), (24*HR, 6*HR)]
No grids for first 24 hours, then 3 hrly grids through 72 hours, then 6 hrly grids.
 [0*HR]
 [(24*HR, 3*HR), (72*HR, 6*HR)]