xan/awips2
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
awips2/docs/edex/data-purge.md

8.9 KiB
Raw Blame History

AWIPS uses a plugin-based purge strategy for HDF5 data, allowing the user to change the purge frequency for each plugin individually, and specific products for a particular plugin.

Purge is triggered by a quartz timer event that fires at 30 minutes after each hour.

Purging rules are defined in XML files in the Localization Store, accessible from the CAVE localization perspective. On EDEX, most are located in /awips2/edex/data/utility/common_static/base/purge, and follow the base/site localization pattern (e.g. site purge files are in site/XXX/purge rather than base/purge, where XXX is the site identifier.

Time-base purge

If a plugin has no XML file, the default rule of 1 day (24 hours) is used, from /awips2/edex/data/utility/common_static/base/purge/defaultPurgeRules.xml

<purgeRuleSet>
    <defaultRule>
        <period>01-00:00:00</period>
    </defaultRule>
</purgeRuleSet>

Time-based purging uses the reference time of the data, and dtermination of the reference time is decoder based.

30-day NEXRAD3 Example

Modify /awips2/edex/data/utility/common_static/base/purge/radarPurgeRules.xml to increase the data retention period from 7 to 31 days:

<purgeRuleSet>
        <defaultRule>
                <period>31-00:00:00</period>
        </defaultRule>
</purgeRuleSet>

Note: you do NOT have to restart EDEX when you change a purge rule!

Frame-based purge

Some plugins use frame-base purging, retaining and certain number of prpduct "versions".

/awips2/edex/data/utility/common_static/base/purge/satellitePurgeRules.xml

<purgeRuleSet>
    <key>sectorID</key>
    <key>physicalElement</key>
    <defaultRule>
            <versionsToKeep>196</versionsToKeep>
    </defaultRule>
    <rule>
            <keyValue>NEXRCOMP</keyValue>
            <versionsToKeep>288</versionsToKeep>
    </rule>
    <rule>
            <keyValue>Alaska National</keyValue>
            <versionsToKeep>48</versionsToKeep>
    </rule>

In the above example, notice a default rule (196) as well as specific sectors with their own rules.

Purge Logs

Data purge events are logged to the file edex-ingest-purge-<yyyymmdd>.log, where <yyyymmdd> is the date stamp.

tail -f edex-ingest-purge-20120327.log 

--------START LOG PURGE---------
INFO  2012-03-27 00:30:00,027 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::Skipped file with invalid fileName: afos-trigger.log
INFO  2012-03-27 00:30:00,193 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::Removed 1 old files
INFO  2012-03-27 00:31:23,155 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::Archived 14 files
INFO  2012-03-27 00:31:23,155 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::Skipped processing 1 files
INFO  2012-03-27 00:31:23,155 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::---------END LOG PURGE-----------

All Purge Rules

To see all purge rule directories (base, site, configured):

find /awips2/edex/data/utility -name purge

/awips2/edex/data/utility/edex_static/base/purge
/awips2/edex/data/utility/edex_static/configured/OAX/purge
/awips2/edex/data/utility/edex_static/site/OAX/purge
/awips2/edex/data/utility/common_static/base/purge
/awips2/edex/data/utility/common_static/configured/OAX/purge
/awips2/edex/data/utility/common_static/site/AFC/purge
/awips2/edex/data/utility/common_static/site/OAX/purge

To see a list of the data plug-ins that have purge rules:

find /awips2/edex/data/utility -name "*PurgeRules.xml"

/awips2/edex/data/utility/edex_static/base/purge/airmetPurgeRules.xml
/awips2/edex/data/utility/edex_static/base/purge/convsigmetPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/bufruaPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/obsPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/nctextPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/ncscatPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/vaaPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/bufrmosHPCPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/bufrhdwPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/cwaPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/airmetPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/ffmpPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/lsrPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/profilerPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/poessoundingPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/gpdPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/mcidasPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/acarssoundingPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/ffgPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/sgwhvPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/intlsigmetPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/aggregatePurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/radarPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/idftPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/wcpPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/warningPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/satellitePurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/sshaPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/ccfpPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/atcfPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/bufrssmiPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/sgwhPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/geomagPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/modelsoundingPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/bufrmosAVNPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/ncuairPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/acarsPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/airepPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/bufrquikscatPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/cwatPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/nonconvsigmetPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/pirepPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/statsPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/bufrmthdwPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/bufrascatPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/gridPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/bufrmosGFSPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/ncpafmPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/bufrsigwxPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/pgenPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/dmwPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/bufrmosMRFPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/bufrmosETAPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/sfcobsPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/modisPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/awwPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/defaultPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/convsigmetPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/redbookPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/bufrncwfPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/tcmPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/bufrmosLAMPPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/goessoundingPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/binlightningPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/stormTrackPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/ghcdPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/ntransPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/regionalsatPurgeRules.xml
/awips2/edex/data/utility/common_static/base/purge/solarimagePurgeRules.xml
/awips2/edex/data/utility/common_static/site/AFC/purge/regionalsatPurgeRules.xml