169 lines
9.6 KiB
Markdown
169 lines
9.6 KiB
Markdown
|
|
AWIPS uses a plugin-based purge strategy for HDF5 data, allowing the user to change the purge frequency for each plugin individually, and specific products for a particular plugin.
|
|
|
|
> Purge is triggered by a quartz timer event that fires at 30 minutes after each hour.
|
|
|
|
Purging rules are defined in XML files in the Localization Store, accessible from the CAVE localization perspective. On EDEX, most are located in `/awips2/edex/data/utility/common_static/base/purge`, and follow the **base/site** localization pattern (e.g. site purge files are in `site/XXX/purge` rather than `base/purge`, where XXX is the site identifier.
|
|
|
|
## Time-base purge
|
|
|
|
If a plugin has no XML file, the default rule of 1 day (24 hours) is used, from `/awips2/edex/data/utility/common_static/base/purge/defaultPurgeRules.xml`
|
|
|
|
<purgeRuleSet>
|
|
<defaultRule>
|
|
<period>01-00:00:00</period>
|
|
</defaultRule>
|
|
</purgeRuleSet>
|
|
|
|
Time-based purging uses the *reference time* of the data, and dtermination of the reference time is decoder based.
|
|
|
|
|
|
## 30-day NEXRAD3 Example
|
|
|
|
Modify `/awips2/edex/data/utility/common_static/base/purge/radarPurgeRules.xml` to increase the data retention period from 7 to 31 days:
|
|
|
|
<purgeRuleSet>
|
|
<defaultRule>
|
|
<period>31-00:00:00</period>
|
|
</defaultRule>
|
|
</purgeRuleSet>
|
|
|
|
**Note**: you do NOT have to restart EDEX when you change a purge rule!
|
|
|
|
|
|
## Frame-based purge
|
|
|
|
Some plugins use frame-base purging, retaining and certain number of prpduct "versions".
|
|
|
|
`/awips2/edex/data/utility/common_static/base/purge/satellitePurgeRules.xml`
|
|
|
|
<purgeRuleSet>
|
|
<key>sectorID</key>
|
|
<key>physicalElement</key>
|
|
<defaultRule>
|
|
<versionsToKeep>196</versionsToKeep>
|
|
</defaultRule>
|
|
<rule>
|
|
<keyValue>NEXRCOMP</keyValue>
|
|
<versionsToKeep>288</versionsToKeep>
|
|
</rule>
|
|
<rule>
|
|
<keyValue>Alaska National</keyValue>
|
|
<versionsToKeep>48</versionsToKeep>
|
|
</rule>
|
|
|
|
> In the above example, notice a *default rule* (196) as well as specific sectors with their own rules.
|
|
|
|
## Purge Logs
|
|
|
|
Data purge events are logged to the file `edex-ingest-purge-<yyyymmdd>.log`, where `<yyyymmdd>` is the date stamp.
|
|
|
|
|
|
tail -f edex-ingest-purge-20120327.log
|
|
|
|
--------START LOG PURGE---------
|
|
INFO 2012-03-27 00:30:00,027 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::Skipped file with invalid fileName: afos-trigger.log
|
|
INFO 2012-03-27 00:30:00,193 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::Removed 1 old files
|
|
INFO 2012-03-27 00:31:23,155 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::Archived 14 files
|
|
INFO 2012-03-27 00:31:23,155 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::Skipped processing 1 files
|
|
INFO 2012-03-27 00:31:23,155 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::---------END LOG PURGE-----------
|
|
|
|
|
|
## All Purge Rules
|
|
|
|
To see all purge rule directories (base, site, region, configured):
|
|
|
|
find /awips2/edex/data/utility -name purge
|
|
|
|
/awips2/edex/data/utility/edex_static/region/CR/purge
|
|
/awips2/edex/data/utility/edex_static/base/purge
|
|
/awips2/edex/data/utility/edex_static/configured/OAX/purge
|
|
/awips2/edex/data/utility/edex_static/site/OAX/purge
|
|
/awips2/edex/data/utility/common_static/region/CR/purge
|
|
/awips2/edex/data/utility/common_static/base/purge
|
|
/awips2/edex/data/utility/common_static/configured/OAX/purge
|
|
/awips2/edex/data/utility/common_static/site/AFC/purge
|
|
/awips2/edex/data/utility/common_static/site/OAX/purge
|
|
|
|
|
|
To see a list of the data plug-ins that have purge rules:
|
|
|
|
find /awips2/edex/data/utility -name "*PurgeRules.xml"
|
|
|
|
/awips2/edex/data/utility/edex_static/base/purge/airmetPurgeRules.xml
|
|
/awips2/edex/data/utility/edex_static/base/purge/convsigmetPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/bufruaPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/obsPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/nctextPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/ncscatPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/vaaPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/bufrmosHPCPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/bufrhdwPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/cwaPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/airmetPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/ffmpPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/lsrPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/profilerPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/poessoundingPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/gpdPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/mcidasPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/acarssoundingPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/ffgPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/sgwhvPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/intlsigmetPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/aggregatePurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/radarPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/idftPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/wcpPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/warningPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/satellitePurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/sshaPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/ldadhydroPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/ldadmesonetPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/ccfpPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/atcfPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/bufrssmiPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/sgwhPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/geomagPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/modelsoundingPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/bufrmosAVNPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/ncuairPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/acarsPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/airepPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/bufrquikscatPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/cwatPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/nonconvsigmetPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/pirepPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/statsPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/bufrmthdwPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/bufrascatPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/gridPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/bufrmosGFSPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/ncpafmPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/bufrsigwxPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/pgenPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/dmwPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/bufrmosMRFPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/bufrmosETAPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/sfcobsPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/modisPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/awwPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/defaultPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/ldadprofilerPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/madisPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/convsigmetPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/redbookPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/bufrncwfPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/tcmPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/bufrmosLAMPPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/goessoundingPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/binlightningPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/stormTrackPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/ghcdPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/ldadmanualPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/ntransPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/mosaicPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/regionalsatPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/nctafPurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/base/purge/solarimagePurgeRules.xml
|
|
/awips2/edex/data/utility/common_static/site/AFC/purge/regionalsatPurgeRules.xml
|