diff --git a/docs/edex/new-grid.md b/docs/edex/new-grid.md index a9c301cc5b..bcf7c627c0 100644 --- a/docs/edex/new-grid.md +++ b/docs/edex/new-grid.md @@ -1,21 +1,29 @@ +# Ingest a New Grid Unrecognized grids can be decoded by EDEX simply by dropping `*.grib` or `*.grib2` files into `/awips2/data_store/ingest/` To add support for a new grid, two edits must be made: -* **Geospatial projection** must be defined in a *grid navigation file* -* **Grid name**, **center**, **subcenter**, and **process ID** must be defined in a *model definition file*. +* **Geospatial projection** must be defined in a [***grid navigation file***](#create-grid-projection-file) +* **Grid name**, **center**, **subcenter**, and **process ID** must be defined in a [***model definition file***](#create-model-definition) + +If the parameters in the grib file haven't been previously specified, another change *may* be needed as well: + +* **Center**, **subcenter**, **discipline**, **category**, and possibly **parameter ID** information may need to be defined in a [***table***](#adding-a-table) +--- ## Ingest an Unsupported Grid +### Grib Products + 1. Download an example grib1 file and rename to a `*.grib` extension, then copy to the manual ingest point `/awips2/data_store/ingest/` - wget https://www.unidata.ucar.edu/staff/mjames/14102318_nmm_d01.GrbF00600 -O wrf.grib + wget https://www.unidata.ucar.edu/software/awips2/14102318_nmm_d01.GrbF00600 -O wrf.grib cp wrf.grib /awips2/data_store/ingest/ - Remember that the data distribution file (`/awips2/edex/data/utility/edex_static/base/distribution/grib.xml`) will match filename which have the `*.grib` extension. + Remember that the data distribution file (`/awips2/edex/data/utility/common_static/base/distribution/grib.xml`) will match filenames which have the `*.grib*` extension. 2. Confirm that the grib file decodes in the grib log file: @@ -33,9 +41,48 @@ To add support for a new grid, two edits must be made: ls -latr /awips2/edex/data/hdf5/grid/GribModel:7:0:89 Though the grib file has been decoded, it has been given a generic name with its center, subcenter, and process IDs (7, 0, 89, respectively). + +### Grib2 Products + +1. Download an example grib2 file and rename to a `*.grib2` extension, then copy to the manual ingest point `/awips2/data_store/ingest/` + + wget https://www.unidata.ucar.edu/software/awips2/CPTI_00.50_20180502-000144.grib2 -O cpti.grib2 + + cp cpti.grib2 /awips2/data_store/ingest/ + + Remember that the data distribution file (`/awips2/edex/data/utility/common_static/base/distribution/grib.xml`) will match filenames which have the `*.grib*` extension. + +2. Confirm that the grib file decodes in the grib log file: + + edex log grib + + INFO [Ingest.GribDecode] /awips2/data_store/ingest/cpti.grib2 processed in: 0.1200 (sec) Latency: 21.8080 (sec) + INFO [Ingest.GribDecode] /awips2/data_store/ingest/cpti.grib2 processed in: 0.1180 (sec) Latency: 21.8140 (sec) + INFO [Ingest.GribDecode] /awips2/data_store/ingest/cpti.grib2 processed in: 0.4230 (sec) Latency: 21.8360 (sec) + INFO [Ingest.GribDecode] /awips2/data_store/ingest/cpti.grib2 processed in: 0.2240 (sec) Latency: 21.9140 (sec) + + ... + + **Note:** This step will likely fail, because the parameter is not yet defined. The error will look like: + + INFO 2020-07-20 20:34:17,710 2565 [GribPersist-1] GridDao: EDEX - Discarding record due to missing or unknown parameter mapping: /grid/2018-05-02_00:01:44.0_(0)/GribModel:161:0:97/null/null/403/Missing/FH/500.0/-999999.0 + INFO 2020-07-20 20:34:17,710 2566 [GribPersist-1] Ingest: EDEX: Ingest - grib2:: /awips2/data_store/ingest/CPTI_00.50_20180502-000144.grib2 processed in: 2.3550 (sec) + INFO 2020-07-20 20:34:17,827 2567 [Ingest.GribDecode-6] grib: EDEX - No parameter information for center[161], subcenter[0], tableName[4.2.209.3], parameter value[61] + + In order to successfully ingest the example file, [**define the appropriate table**](#grib2-products_4). + +3. Check that the hdf5 data directory exists for our unnamed grid + + ls -latr /awips2/edex/data/hdf5/grid/GribModel:161:0:97 + + Though the grib file has been decoded, it has been given a generic name with its center, subcenter, and process IDs (161, 0, 97, respectively). + +--- ## Determine Grid Projection +### Grib Products + When the grid was ingested a record was added to the `grid_coverage` table with its navigation information: psql metadata @@ -46,26 +93,66 @@ When the grid was ingested a record was added to the `grid_coverage` table with 201 | 155 | 4.29699993133545 | 4.29699993133545 | 6378160 | 6356775 | 42.2830009460449 | -72.3610000610352 | -67.0770034790039 | 45.3680000305176 | 45.3680000305176 (1 row) -Compare with the projection info returned by wgrib on the original file: +Compare with the projection info returned by wgrib on the original file (look at the bolded sections below and make sure they match up with the corresponding entries returned from the database above): + +
+wgrib -V wrf.grib +rec 799:27785754:date 2014102318 ALBDO kpds5=84 kpds6=1 kpds7=0 levels=(0,0) grid=255 sfc 6hr fcst: bitmap: 736 undef + ALBDO=Albedo [%] + timerange 0 P1 6 P2 0 TimeU 1 nx 201 ny 155 GDS grid 3 num_in_ave 0 missing 0 + center 7 subcenter 0 process 89 Table 2 scan: WE:SN winds(grid) + Lambert Conf: Lat1 42.283000 Lon1 -72.361000 Lov -67.077000 + Latin1 45.368000 Latin2 45.368000 LatSP 0.000000 LonSP 0.000000 + North Pole (201 x 155) Dx 4.297000 Dy 4.297000 scan 64 mode 8 + min/max data 5 21.9 num bits 8 BDS_Ref 50 DecScale 1 BinScale 0 +- wgrib -V wrf.grib +Notice that our grib file has a **Lambert Conformal** projection. We will need these values for the next step. Note that **there is a tolerance of +/- 0.1 degrees** to keep in mind when defining your coverage area. + +### Grib2 Products + +When the grid was ingested a record was added to the `grid_coverage` table with its navigation information: + + psql metadata - rec 799:27785754:date 2014102318 ALBDO kpds5=84 kpds6=1 kpds7=0 levels=(0,0) grid=255 sfc 6hr fcst: bitmap: 736 undef - ALBDO=Albedo [%] - timerange 0 P1 6 P2 0 TimeU 1 nx 201 ny 155 GDS grid 3 num_in_ave 0 missing 0 - center 7 subcenter 0 process 89 Table 2 scan: WE:SN winds(grid) - Lambert Conf: Lat1 42.283000 Lon1 -72.361000 Lov -67.077000 - Latin1 45.368000 Latin2 45.368000 LatSP 0.000000 LonSP 0.000000 - North Pole (201 x 155) Dx 4.297000 Dy 4.297000 scan 64 mode 8 - min/max data 5 21.9 num bits 8 BDS_Ref 50 DecScale 1 BinScale 0 + metadata=# select nx,ny,dx,dy,majoraxis,minoraxis,la1,lo1,lov,latin1,latin2 from gridcoverage where id=(select distinct(location_id) from grid_info where datasetid='GribModel:161:0:97'); + + nx | ny | dx | dy | majoraxis | minoraxis | la1 | lo1 | lov | latin1 | latin2 + -----+-----+-------+-------+-----------+-----------+-----------+-----+-----+--------+-------- + 600 | 640 | 0.005 | 0.005 | | | 40.799999 | 261 | | | + (1 row) + Compare with the projection info returned by wgrib2 on the original file (look at the bolded sections below and make sure they match up with the corresponding entries returned from the database above): + +
+wgrib2 -grid -nxny cpti.grib2 +1:0:grid_template=0:winds(N/S): + lat-lon grid:(600 x 640) units 1e-06 input WE:NS output WE:SN res 48 + lat 40.799999 to 37.599999 by 0.005000 + lon 260.999999 to 263.999999 by 0.005000 #points=384000:(600 x 640) + ... ++Notice that our grib2 file has a **Lat/lon Grid** projection. Where: -Notice that our grib1 file is a **Lambert Conformal** projection. We will need these values for the next step. Note that **there is a tolerance of +/- 0.1 degrees** to keep in mind when defining your coverage area. +* **nx** is **600** +* **ny** is **640** +* **dx** is **0.005** +* **dy** is **0.005** +* **la1** is **40.799999** +* **lo1** is **261** + +We will need these values for the next step. Note that **there is a tolerance of +/- 0.1 degrees** to keep in mind when defining your coverage (la1 and lo1) area. + +--- ## Create Grid Projection File -Grid projection files are stored in `/awips2/edex/data/utility/edex_static/base/grib/grids/` and there are four grid coverage types available: +Grid projection files are stored in `/awips2/edex/data/utility/common_static/base/grib/grids/` and there are four grid coverage types available: -1. **lambertConformalGridCoverage** example +1. **lambertConformalGridCoverage** (example: `RUCIcing.xml`)
+wgrib -V wrf.grib +rec 799:27785754:date 2014102318 ALBDO kpds5=84 kpds6=1 kpds7=0 levels=(0,0) grid=255 sfc 6hr fcst: bitmap: 736 undef + ALBDO=Albedo [%] + timerange 0 P1 6 P2 0 TimeU 1 nx 201 ny 155 GDS grid 3 num_in_ave 0 missing 0 + center 7 subcenter 0 process 89 Table 2 scan: WE:SN winds(grid) + Lambert Conf: Lat1 42.283000 Lon1 -72.361000 Lov -67.077000 + Latin1 45.368000 Latin2 45.368000< LatSP 0.000000 LonSP 0.000000 + North Pole (201 x 155) Dx 4.297000 Dy 4.297000 scan 64 mode 8 + min/max data 5 21.9 num bits 8 BDS_Ref 50 DecScale 1 BinScale 0 ++ +For our example, the process is **89** and table is **2**. Next, take a look in: + + /awips2/edex/data/utility/common_static/base/grid/grib1ParameterConvTable.xml + +And find the entry that has grib1 data with TableVersion 2 and Value 89: + +
+wgrib2 -varX cpti.grib2 + 1:0:var209_255_1_161_3_61 +... ++ Where the 4th argument after "var" is the center id, in this case **161**. + +To get the subcenter, simply run: +
+wgrib2 -subcenter cpti.grib2 + 1:0:subcenter=0 +... ++The subcenter of this file is **0**. + +So based on the center and subcenter, the corresponding directory is: + + /awips2/edex/data/utility/common_static/base/grib/tables/161/0/ + +To find the exact table, we need the discipline and category: + +
+wgrib2 -full_name cpti.grib2 + 1:0:var209_3_61.500_m_above_mean_sea_level + ... ++ +In this case the **discipline is 209** and **category is 3**, so the corresponding table is: + + 4.2.209.3.table + +So, the full path to the corresponding table would be: + + /awips2/edex/data/utility/common_static/base/grib/tables/161/0/4.2.209.3.table + +The parameter ID was also listed in that output as **61**. Make sure that specific parameter information is defined in the table: +
+... +56:56:Reflectivity at -20C:dBZ:ReflectivityM20C +57:57:Reflectivity At Lowest Altitude (RALA):dBZ:ReflectivityAtLowestAltitude +58:58:Merged Reflectivity At Lowest Altitude (RALA):dBZ:MergedReflectivityAtLowestAltitude +59:59:CPTI 80mph+:%:CPTI80mph +61:61:CPTI 110mph+:%:CPTI110mph ++ +You will have to restart ingestGrib for the changes to take place: + + sudo service edex_camel restart ingestGrib + +Now you can try [re-ingesting the grib2 file](#grib2-products). + +--- + +## Troubleshooting Grib Ingest + +* Make sure the latitude and longitude entries in your coverage specification file match those of your ingested raw grib file. There is a tolerance of +/- 0.1 degree to keep in mind when defining your coverage area. + +* If some of the information is unknown, using a grib utility application such as *wgrib* and *wgrib2* can be useful in determining the information that must be added to correctly process a new grib file. + +* If you are experiencing `Segmentation fault` errors when running wgrib2, it may be best to install the latest version using the following command: + + yum install wgrib2 + And then you may either need to change where `wgrib2` points to, or use `/bin/wgrib2` to run the recently downloaded version.