awips2/edex/archiver/index.html
2022-07-27 15:06:02 +00:00

1002 lines
No EOL
33 KiB
HTML

<!DOCTYPE html>
<html lang="en" class="no-js">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta http-equiv="x-ua-compatible" content="ie=edge">
<meta name="description" content="Documentation for all things AWIPS.">
<link rel="canonical" href="http://unidata.github.io/awips2/edex/archiver/">
<meta name="author" content="support-awips@unidata.ucar.edu">
<link rel="shortcut icon" href="../../images/favicon.ico">
<meta name="generator" content="mkdocs-1.3.1, mkdocs-unidata-2.0.3">
<title>Grant Users Permission to Create Case Study Archives - Unidata AWIPS User Manual</title>
<script src="../../assets/javascripts/modernizr-e826f8942a.js"></script>
<link rel="stylesheet" href="../../assets/stylesheets/application-37fa87ea8f.css">
<link rel="stylesheet" href="../../assets/stylesheets/application-624922a329.palette.css">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,400i,700|Roboto+Mono">
<style>body,input{font-family:"Roboto","Helvetica Neue",Helvetica,Arial,sans-serif}code,kbd,pre{font-family:"Roboto Mono","Courier New",Courier,monospace}</style>
<link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons">
<link rel="stylesheet" href="../../css/extra.css">
</head>
<body data-md-color-primary="blue-grey" data-md-color-accent="light-blue">
<svg class="md-svg">
<defs>
<svg xmlns="http://www.w3.org/2000/svg" width="416" height="448" viewBox="0 0 416 448" id="github"><path fill="currentColor" d="M160 304q0 10-3.125 20.5t-10.75 19T128 352t-18.125-8.5-10.75-19T96 304t3.125-20.5 10.75-19T128 256t18.125 8.5 10.75 19T160 304zm160 0q0 10-3.125 20.5t-10.75 19T288 352t-18.125-8.5-10.75-19T256 304t3.125-20.5 10.75-19T288 256t18.125 8.5 10.75 19T320 304zm40 0q0-30-17.25-51T296 232q-10.25 0-48.75 5.25Q229.5 240 208 240t-39.25-2.75Q130.75 232 120 232q-29.5 0-46.75 21T56 304q0 22 8 38.375t20.25 25.75 30.5 15 35 7.375 37.25 1.75h42q20.5 0 37.25-1.75t35-7.375 30.5-15 20.25-25.75T360 304zm56-44q0 51.75-15.25 82.75-9.5 19.25-26.375 33.25t-35.25 21.5-42.5 11.875-42.875 5.5T212 416q-19.5 0-35.5-.75t-36.875-3.125-38.125-7.5-34.25-12.875T37 371.5t-21.5-28.75Q0 312 0 260q0-59.25 34-99-6.75-20.5-6.75-42.5 0-29 12.75-54.5 27 0 47.5 9.875t47.25 30.875Q171.5 96 212 96q37 0 70 8 26.25-20.5 46.75-30.25T376 64q12.75 25.5 12.75 54.5 0 21.75-6.75 42 34 40 34 99.5z"/></svg>
</defs>
</svg>
<input class="md-toggle" data-md-toggle="drawer" type="checkbox" id="drawer">
<input class="md-toggle" data-md-toggle="search" type="checkbox" id="search">
<label class="md-overlay" data-md-component="overlay" for="drawer"></label>
<header class="md-header" data-md-component="header">
<a href="../..">
<div class="background-logo" style="background-image:url('../../images/cave.png');background-repeat:no-repeat;height:107px">
<nav class="md-header-nav md-grid">
<div class="md-flex">
<div class="md-flex__cell md-flex__cell--shrink">
<a href="http://unidata.github.io/awips2/" title="Unidata AWIPS User Manual" class="md-header-nav__button md-logo">
<img src="../../images/home.png" width="24" height="24">
</a>
</div>
<div class="md-flex__cell md-flex__cell--shrink">
<label class="md-icon md-icon--menu md-header-nav__button" for="drawer"></label>
</div>
<div class="md-flex__cell md-flex__cell--stretch">
<span class="md-flex__ellipsis md-header-nav__title">
Grant Users Permission to Create Case Study Archives
</span>
</div>
<div class="md-flex__cell md-flex__cell--shrink">
<label class="md-icon md-icon--search md-header-nav__button" for="search"></label>
<div class="md-search" data-md-component="search" role="dialog">
<label class="md-search__overlay" for="search"></label>
<div class="md-search__inner">
<form class="md-search__form" name="search">
<input type="text" class="md-search__input" name="query" required placeholder="Search" autocapitalize="off" autocorrect="off" autocomplete="off" spellcheck="false" data-md-component="query">
<label class="md-icon md-search__icon" for="search"></label>
<button type="reset" class="md-icon md-search__icon" data-md-component="reset">&#xE5CD;</button>
</form>
<div class="md-search__output">
<div class="md-search__scrollwrap" data-md-scrollfix>
<div class="md-search-result" data-md-component="result" data-md-lang-search="" data-md-lang-tokenizer="[\s\-]+">
<div class="md-search-result__meta" data-md-lang-result-none="No matching documents" data-md-lang-result-one="1 matching document" data-md-lang-result-other="# matching documents">
Type to start searching
</div>
<ol class="md-search-result__list"></ol>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="md-flex__cell md-flex__cell--shrink">
<div class="md-header-nav__source">
<a href="https://github.com/Unidata/awips2" title="Go to repository" class="md-source" data-md-source="github">
<div class="md-source__icon">
<svg viewBox="0 0 24 24" width="24" height="24">
<use xlink:href="#github" width="24" height="24"></use>
</svg>
</div>
<div class="md-source__repository">
GitHub
</div>
</a>
</div>
</div>
</div>
</nav>
</header>
<div class="md-container">
<main class="md-main">
<div class="md-main__inner md-grid" data-md-component="container">
<div class="md-sidebar md-sidebar--primary" data-md-component="navigation">
<div class="md-sidebar__scrollwrap">
<div class="md-sidebar__inner">
<nav class="md-nav md-nav--primary" data-md-level="0">
<label class="md-nav__title md-nav__title--site" for="drawer">
<a href="http://unidata.github.io/awips2/" title="Unidata AWIPS User Manual" class="md-nav__button md-logo">
<img src="../../images/home.png">
</a>
Unidata AWIPS User Manual
</label>
<div class="md-nav__source">
<a href="https://github.com/Unidata/awips2" title="Go to repository" class="md-source" data-md-source="github">
<div class="md-source__icon">
<svg viewBox="0 0 24 24" width="24" height="24">
<use xlink:href="#github" width="24" height="24"></use>
</svg>
</div>
<div class="md-source__repository">
GitHub
</div>
</a>
</div>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../.." title="Home" class="md-nav__link">
Home
</a>
</li>
<li class="md-nav__item">
<a href="../../install/install-cave/" title="Install CAVE" class="md-nav__link">
Install CAVE
</a>
</li>
<li class="md-nav__item">
<a href="../../install/install-edex/" title="Install EDEX" class="md-nav__link">
Install EDEX
</a>
</li>
<li class="md-nav__item">
<a href="../../appendix/common-problems/" title="Common Problems" class="md-nav__link">
Common Problems
</a>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-toggle md-nav__toggle" data-md-toggle="nav-5" type="checkbox" id="nav-5">
<label class="md-nav__link" for="nav-5">
CAVE User Manual
</label>
<nav class="md-nav" data-md-component="collapsible" data-md-level="1">
<label class="md-nav__title" for="nav-5">
CAVE User Manual
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../cave/d2d-perspective/" title="D2D Perspective" class="md-nav__link">
D2D Perspective
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/maps-views-projections/" title="Maps, Views, Projections" class="md-nav__link">
Maps, Views, Projections
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/bundles-and-procedures/" title="Displays and Procedures" class="md-nav__link">
Displays and Procedures
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/localization-perspective/" title="Localization Perspective" class="md-nav__link">
Localization Perspective
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/nsharp/" title="NSHARP" class="md-nav__link">
NSHARP
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/warngen/" title="WarnGen Walkthrough" class="md-nav__link">
WarnGen Walkthrough
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/goes-16-17-satellite/" title="GOES 16/17" class="md-nav__link">
GOES 16/17
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/d2d-gis-shapefiles/" title="GIS and Shapefiles" class="md-nav__link">
GIS and Shapefiles
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/d2d-gridded-models/" title="Volume Browser" class="md-nav__link">
Volume Browser
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/d2d-tools/" title="Display Tools" class="md-nav__link">
Display Tools
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/d2d-radar-tools/" title="Radar Tools" class="md-nav__link">
Radar Tools
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/d2d-edit-menus/" title="Editing Menus" class="md-nav__link">
Editing Menus
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/cave-localization/" title="Change Localization" class="md-nav__link">
Change Localization
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/import-export/" title="Import/Export" class="md-nav__link">
Import/Export
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-toggle md-nav__toggle" data-md-toggle="nav-6" type="checkbox" id="nav-6">
<label class="md-nav__link" for="nav-6">
EDEX User Manual
</label>
<nav class="md-nav" data-md-component="collapsible" data-md-level="1">
<label class="md-nav__title" for="nav-6">
EDEX User Manual
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../install/start-edex/" title="EDEX Basic Commands" class="md-nav__link">
EDEX Basic Commands
</a>
</li>
<li class="md-nav__item">
<a href="../settings/" title="EDEX Settings" class="md-nav__link">
EDEX Settings
</a>
</li>
<li class="md-nav__item">
<a href="../distributed-computing/" title="Distributed EDEX" class="md-nav__link">
Distributed EDEX
</a>
</li>
<li class="md-nav__item">
<a href="../edex-ingest-docker-container/" title="Docker EDEX" class="md-nav__link">
Docker EDEX
</a>
</li>
<li class="md-nav__item">
<a href="../ldm/" title="LDM Feeds" class="md-nav__link">
LDM Feeds
</a>
</li>
<li class="md-nav__item">
<a href="../data-distribution-files/" title="Data Distribution Files" class="md-nav__link">
Data Distribution Files
</a>
</li>
<li class="md-nav__item">
<a href="../new-grid/" title="Ingest a New Grid" class="md-nav__link">
Ingest a New Grid
</a>
</li>
<li class="md-nav__item">
<a href="../data-purge/" title="Purging and Retention" class="md-nav__link">
Purging and Retention
</a>
</li>
<li class="md-nav__item">
<a href="../edex-users/" title="Monitor Users" class="md-nav__link">
Monitor Users
</a>
</li>
<li class="md-nav__item">
<a href="../data-plugins/" title="Data Plugins" class="md-nav__link">
Data Plugins
</a>
</li>
<li class="md-nav__item">
<a href="../case-studies/" title="Archive Case Studies" class="md-nav__link">
Archive Case Studies
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="../../python/overview/" title="Python-AWIPS" class="md-nav__link">
Python-AWIPS
</a>
</li>
<li class="md-nav__item">
<a href="../../appendix/educational-resources/" title="Educational Resources" class="md-nav__link">
Educational Resources
</a>
</li>
<li class="md-nav__item">
<a href="../../dev/awips-development-environment/" title="Development" class="md-nav__link">
Development
</a>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-toggle md-nav__toggle" data-md-toggle="nav-10" type="checkbox" id="nav-10">
<label class="md-nav__link" for="nav-10">
Appendix
</label>
<nav class="md-nav" data-md-component="collapsible" data-md-level="1">
<label class="md-nav__title" for="nav-10">
Appendix
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../appendix/appendix-grid-parameters/" title="AWIPS Grid Parameters" class="md-nav__link">
AWIPS Grid Parameters
</a>
</li>
<li class="md-nav__item">
<a href="../../python/maps-database/" title="Maps Database" class="md-nav__link">
Maps Database
</a>
</li>
<li class="md-nav__item">
<a href="../../appendix/appendix-acronyms/" title="Acronyms and Abbreviations" class="md-nav__link">
Acronyms and Abbreviations
</a>
</li>
<li class="md-nav__item">
<a href="../../appendix/appendix-wsr88d/" title="WSR-88D Product Table" class="md-nav__link">
WSR-88D Product Table
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/cave-keyboard-shortcuts/" title="Keyboard Shortcuts" class="md-nav__link">
Keyboard Shortcuts
</a>
</li>
<li class="md-nav__item">
<a href="../../raytheon/cave_d2d/" title="Raytheon: CAVE User's Manual" class="md-nav__link">
Raytheon: CAVE User's Manual
</a>
</li>
<li class="md-nav__item">
<a href="../../raytheon/smm/" title="Raytheon: AWIPS System Manager's Manual" class="md-nav__link">
Raytheon: AWIPS System Manager's Manual
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div class="md-sidebar md-sidebar--secondary" data-md-component="toc">
<div class="md-sidebar__scrollwrap">
<div class="md-sidebar__inner">
<nav class="md-nav md-nav--secondary">
</nav>
</div>
</div>
</div>
<div class="md-content">
<article class="md-content__inner md-typeset">
<a href="https://github.com/Unidata/awips2/edit/unidata_18.2.1/docs/edex/archiver.md" title="Edit this page" class="md-icon md-content__icon">&#xE3C9;</a>
<h1 id="grant-users-permission-to-create-case-study-archives">Grant Users Permission to Create Case Study Archives<a class="headerlink" href="#grant-users-permission-to-create-case-study-archives" title="Permanent link"></a></h1>
<p>The file <code>/awips2/edex/data/utility/common_static/base/roles/archiveAdminRoles.xml</code> controls which users can run the archiving tools from CAVE.</p>
<pre><code>&lt;nwsRoleData xmlns:ns2="group"&gt;
&lt;application&gt;Data Archiving&lt;/application&gt;
&lt;permission id="archive.retention"&gt;
&lt;description&gt;
This permission allows the user to access Archive Retention.
&lt;/description&gt;
&lt;/permission&gt;
&lt;permission id="archive.casecreation"&gt;
&lt;description&gt;
This permission allows the user to access Archive Case Creation.
&lt;/description&gt;
&lt;/permission&gt;
&lt;user userId="ALL"&gt;
&lt;userPermission&gt;archive.retention&lt;/userPermission&gt;
&lt;userPermission&gt;archive.casecreation&lt;/userPermission&gt;
&lt;/user&gt;
&lt;/nwsRoleData&gt;
</code></pre>
<p><code>&lt;user userId="ALL"&gt;</code> will allow any connected CAVE user to run both the Archive Retention and the Archive Case Creation tools. If you want to control access to individual users, such as the example bwlo, which will allow any user to create case studies, but only the <em>awips</em> user to run the Archive Retention tool.</p>
<pre><code> &lt;user userId="awips"&gt;
&lt;userPermission&gt;archive.retention&lt;/userPermission&gt;
&lt;/user&gt;
&lt;user userId="ALL"&gt;
&lt;userPermission&gt;archive.casecreation&lt;/userPermission&gt;
&lt;/user&gt;
</code></pre>
<h1 id="define-edex-user-administration-roles">Define EDEX User Administration Roles<a class="headerlink" href="#define-edex-user-administration-roles" title="Permanent link"></a></h1>
<p>Admins can use the CAVE User Administration interface to manage user access roles. The file <code>/awips2/edex/data/utility/common_static/base/roles/awipsUserAdminRoles.xml</code> controls access to this tool.</p>
<pre><code>&lt;nwsRoleData xmlns:ns2="group"&gt;
&lt;application&gt;User Administration&lt;/application&gt;
&lt;permission id="awips.user.admin"&gt;
&lt;description&gt;
This permission allows the user to access and edit AWIPS 2 User Administration
&lt;/description&gt;
&lt;/permission&gt;
&lt;user userId="awips"&gt;
&lt;userPermission&gt;awips.user.admin&lt;/userPermission&gt;
&lt;/user&gt;
&lt;/nwsRoleData&gt;
</code></pre>
<h1 id="edex-archiver">EDEX Archiver<a class="headerlink" href="#edex-archiver" title="Permanent link"></a></h1>
<p>/awips2/edex/conf/resources/com.raytheon.uf.edex.archive.cron.properties</p>
<pre><code># enable archive
archive.enable=false
# runs database and hdf5 archive for archive server to pull data from
archive.cron=0+40+*+*+*+?
# path to store processed archive data
archive.path=/awips2/archive
# enable archive purge
archive.purge.enable=true
# when to purge archives
archive.purge.cron=0+5+0/2+*+*+?
# compress database records
archive.compression.enable=false
# To change Default case directory.
#archive.case.directory=/awips2/edex/data/archiver/
# to disable a specific archive, use property archive.disable=pluginName,pluginName...
#archive.disable=grid,text,acars
</code></pre>
<p>The EDEX Archiver plugin can be used to automate data backup or create case study archive files to be retained by EDEX. The file <code>/awips2/edex/data/utility/common_static/base/archiver/purger/PROCESSED_DATA.xml</code> controls which products are ardhived, and how.</p>
<h2 id="archive-log">Archive Log<a class="headerlink" href="#archive-log" title="Permanent link"></a></h2>
<p>The file <code>/awips2/edex/logs/edex-ingest-archive-*.log</code> will report status of the archiver whenever it is run. With regular archiving disabled (by default) will see messages such as </p>
<pre><code>INFO 2016-11-30 09:40:00,010 [Archiver] DataArchiver: EDEX - Archival of plugin data disabled, exiting
INFO 2016-11-30 10:40:00,009 [Archiver] DataArchiver: EDEX - Archival of plugin data disabled, exiting
INFO 2016-11-30 11:40:00,010 [Archiver] DataArchiver: EDEX - Archival of plugin data disabled, exiting
INFO 2016-11-30 12:40:00,010 [Archiver] DataArchiver: EDEX - Archival of plugin data disabled, exiting
</code></pre>
<h2 id="awips2edexdatautilitycommon_staticbasearchiverpurgerprocessed_dataxml">/awips2/edex/data/utility/common_static/base/archiver/purger/PROCESSED_DATA.xml<a class="headerlink" href="#awips2edexdatautilitycommon_staticbasearchiverpurgerprocessed_dataxml" title="Permanent link"></a></h2>
<p><code>&lt;name&gt;</code>, <code>&lt;rootDir&gt;</code>, <code>&lt;defaultRetentionHours&gt;</code>, and <code>&lt;category&gt;</code> are the four tags which configure the EDEX Archiver. </p>
<pre><code>&lt;archive&gt;
&lt;name&gt;Processed&lt;/name&gt;
&lt;rootDir&gt;/awips2/archive/&lt;/rootDir&gt;
&lt;defaultRetentionHours&gt;168&lt;/defaultRetentionHours&gt;
&lt;category&gt;
&lt;name&gt;Model&lt;/name&gt;
&lt;selectedRetentionHours&gt;168&lt;/selectedRetentionHours&gt;
&lt;dataSet&gt;
&lt;dirPattern&gt;(grid)/(.*)/(.*)/.*-(\d{4})-(\d{2})-(\d{2})-(\d{2})-.*&lt;/dirPattern&gt;
&lt;displayLabel&gt;{2}&lt;/displayLabel&gt;
&lt;dateGroupIndices&gt;4,5,6,7&lt;/dateGroupIndices&gt;
&lt;/dataSet&gt;
&lt;dataSet&gt;
&lt;dirPattern&gt;(modelsounding)/(.*)/.*/.*(\d{4})-(\d{2})-(\d{2})-(\d{2}).*&lt;/dirPattern&gt;
&lt;dirPattern&gt;(bufrmos)(.*)/.*(\d{4})-(\d{2})-(\d{2})-(\d{2})&lt;/dirPattern&gt;
&lt;displayLabel&gt;{1} - {2}&lt;/displayLabel&gt;
&lt;dateGroupIndices&gt;3,4,5,6&lt;/dateGroupIndices&gt;
&lt;/dataSet&gt;
&lt;/category&gt;
</code></pre>
<p><code>&lt;category&gt;</code> is used as a logical grouping of the archive sub-directories, and contains the following tags:</p>
<ul>
<li><code>&lt;name&gt;</code> - The id for the category, used in CAVE.</li>
<li><code>&lt;selectedRetentionHours&gt;</code> - Optional. The hours to retain data in directories of selected Data Sets for a category. Default is 1 hour.</li>
<li><code>&lt;selectedDisplayNames&gt;</code> - A directory matching <code>&lt;dirPattern&gt;</code>. These are selected directories from the Retention GUI. The purger will used the category's <code>&lt;selectedRetentionHours&gt;</code> instead of the Arhivie's <code>&lt;defaultRetentionHours&gt;</code>. An Optional. may have more then one. (NOTE these are set internally when a selection configuration file is loaded. They should not appear in the configuration file.)</li>
<li>
<p><code>&lt;dataSet&gt;</code> - Required to have a least one. Each one contains a set of tags explained below.</p>
<p>The <code>&lt;dataSet&gt;</code> tag contains:</p>
<ul>
<li><code>&lt;dirPattern&gt;</code> - A regex pattern for finding directories for this category. Required to have at least one. The pattern is relative to <code>&lt;rootDir&gt;</code>. Wildcard patterns do not cross directory delimiter <code>/</code>. Thus to match 3 levels of directories you would need <code>.*/.*/.*</code> (see patterns and groups section). There may be more then one <code>&lt;dirPattern&gt;</code> in a <code>&lt;dataSet&gt;</code>, but they must all have the same number of groupings and be in the same order to match up with <code>&lt;displayLabel&gt;</code>, and <code>&lt;dateGroupIndicies&gt;</code>.</li>
<li><code>&lt;filePattern&gt;</code> - Optional. A pattern to find files in the directories that match <code>&lt;dirPattern&gt;</code>. Default is everything in the directories that match <code>&lt;dirPattern&gt;</code>. See patterns and groups section.</li>
<li><code>&lt;displayLabel&gt;</code> - The label to display for directories that match <code>&lt;dirPattern&gt;</code>. Any group in the <code>&lt;dirPattern&gt;</code> may be made part of the label by placing the group's index inside parenthesis, <code>{1}</code>. More then one directory may match the <code>&lt;dirPattern&gt;</code>. The archive GUIs may collapse them into a single table entry.</li>
<li>
<p><code>&lt;timeType&gt;</code> - Optional tag to determine what type of time stamp is being used to get files/directories for retention and case creation. The value dictates how many groupings in the <code>&lt;dirPattern&gt;</code>s and/or <code>&lt;filePattern&gt;</code> are used to get the time stamp for a file. </p>
<p>The five values are:</p>
<ul>
<li><strong>Date</strong> - (default) the time stamp is made up of 3 or 4 groups in the patterns: <strong>year</strong>, <strong>month</strong>, <strong>day</strong> and (optional) <strong>hour</strong>.</li>
<li><strong>Julian</strong> - Time stamp is made up of 2 or 3 groups in the patterns: <strong>year</strong>, <strong>day_of_year</strong> and (optional) <strong>hour</strong>.</li>
<li><strong>EpochSec</strong> - Time stamp epoch time in seconds.</li>
<li><strong>EpochMS</strong> - Time stamp epoch time in milliseconds.</li>
<li><strong>File</strong> - Instead use the files date of last modification. No group is used to get the time stamp. </li>
</ul>
</li>
<li>
<p><code>&lt;dateGroupIndicies&gt;</code> - Required tag when <code>&lt;timeType&gt;</code> has any value but <strong>File</strong>. </p>
<ul>
<li><strong>Date</strong> - A comma separated list of 3 or 4 numbers which are in order the index for <strong>year</strong>, <strong>month</strong>, <strong>day</strong> and <strong>hour</strong>. When only 3 numbers the hour is value is 23.</li>
<li><strong>Julian</strong> - A comma separated list of 2 or 3 numbers which are in order the index for <strong>year</strong>, <strong>day of year</strong>, and <strong>hour</strong>. When only two numbers the hour value is 23.</li>
<li><strong>EpochSec</strong> - A number which is the index for the epoch in seconds.</li>
<li><strong>EpochMS</strong> - A number which is the index for the epoch in milliseconds.</li>
<li><strong>File</strong> - Not needed since no group is used to get the time stamp.</li>
</ul>
<p>This is used to determine what files/directories to retain or a range of directories/files to copy
for case creation. Note to get the group's index the <code>&lt;dirPattern&gt;</code> and <code>&lt;filePattern&gt;</code> are combined.
Thus if there are 5 groups in the <code>&lt;dirPattern&gt;</code> then the first group in the <code>&lt;filePattern&gt;</code> is index 6.</p>
</li>
</ul>
</li>
</ul>
<p>## Patterns and groups.</p>
<p><code>&lt;dirPattern&gt;</code> and <code>&lt;filePattern&gt;</code> use <a href="http://docs.oracle.com/javase/tutorial/essential/regex/">Java regex expressions</a>, similar to the ldm's pqact.conf file.</p>
<p>The groupings index start at one. The groups in the <code>&lt;dirPattern&gt;</code> can be used in the <code>&lt;displayLabel&gt;</code>. For example:</p>
<pre><code>&lt;dirPattern&gt;(grib2)/(\d{4})(\d{2})(\d{2})/(\d{2})/(.*)&lt;/dirPattern&gt;
&lt;displayLabel&gt;{1} - {6}&lt;/displayLabel&gt;
&lt;dateGroupIndices&gt;2,3,4,5&lt;/dateGroupIndices&gt;
</code></pre>
<p><code>&lt;dirPattern&gt;</code> contains six groups. The first group is the literal grib2 which matches only a directory named grib2 that is a sub-directory of the <code>&lt;rootDir&gt;</code>. The groups 2, 3 and 4 break apart the next level of sub-directories into a 4 digit and two 2 digit groups. This is the expected <strong>year</strong>, <strong>month</strong>, <strong>day</strong> sub-subdirectory indicated by the first 3 entries in <code>&lt;dateGroupIndices&gt;</code>. The next sub-directory contains the fifth group which is a two digit number representing the hour. Finally the sixth group will match any sub-directory that in the hour directory.</p>
<p>Thus the directory paths <code>&lt;rootPath&gt;/grib2/20130527/18/GFS</code> will generate the display string, <strong>grib2 - GFS</strong>, and from the grouping we can find the year, <strong>2013</strong>; month, <strong>05</strong>; day, <strong>27</strong> and hour, <strong>18</strong>.</p>
<h4 id="example-with-filepattern">Example with <strong>&lt;filePattern&gt;</strong><a class="headerlink" href="#example-with-filepattern" title="Permanent link"></a></h4>
<pre><code>&lt;dirPattern&gt;hdf5/(redbook)&lt;/dirPattern&gt;
&lt;displayLabel&gt;{1}&lt;/displayLabel&gt;
&lt;filePattern&gt;redbook-(\d{4})-(\d{2})-(\d{2})-(\d{2})\..*&lt;/filePattern&gt;
&lt;dateGroupIndices&gt;2,3,4,5&lt;/dateGroupIndices&gt;
</code></pre>
<h4 id="example-with-multiple-dirpattern">Example with multiple <strong>&lt;dirPattern&gt;</strong><a class="headerlink" href="#example-with-multiple-dirpattern" title="Permanent link"></a></h4>
<pre><code>&lt;category&gt;
&lt;name&gt;Observation&lt;/name&gt;
&lt;selectedRetentionHours&gt;168&lt;/selectedRetentionHours&gt;
&lt;dataSet&gt;
&lt;dirPattern&gt;(acars|airep|airmet|taf)&lt;/dirPattern&gt;
&lt;dirPattern&gt;(bufrsigwx|sfcobs)/.*&lt;/dirPattern&gt;
&lt;displayLabel&gt;{1}&lt;/displayLabel&gt;
&lt;timeType&gt;Date&lt;/timeType&gt;
&lt;dateGroupIndices&gt;2,3,4,5&lt;/dateGroupIndices&gt;
&lt;filePattern&gt;.*-(\d{4})-(\d{2})-(\d{2})-(\d{2})\..*&lt;/filePattern&gt;
&lt;/dataSet&gt;
&lt;/category&gt;
</code></pre>
<p>The first <code>&lt;dirPattern&gt;</code> looks for files matching the <code>&lt;filePattern&gt;</code> in the directories <strong>acars</strong>, <strong>airep</strong>, <strong>airmet</strong> or <strong>taf</strong>.</p>
<p>The second <code>&lt;dirPattern&gt;</code> expects to find the files in subdirectories of <strong>bufrsigwx</strong> or <strong>sfcobs</strong> such as <strong>bufrsigwx/SWH</strong>.</p>
<p>Here the display will only show, redbook. The directory looked at will be <code>&lt;rootPath&gt;/redbook/</code>. The <code>&lt;dateGroupIndices&gt;</code> all come from the <code>&lt;filePattern&gt;</code> since there is one group in the <code>&lt;dirPattern&gt;</code> the groups in the <code>&lt;filePattern&gt;</code> start at two. This matches file names <code>redbook-YYYY-MM-DD-HH.&lt;extension&gt;</code>. Thus the file name <code>redbook-2013-05-28-00.hd5</code> would match the <code>&lt;filePattern&gt;</code>.</p>
<blockquote>
<p>NOTE group <code>{0}</code> is a string that matches the whole <code>&lt;dirPattern&gt;</code>. If this is used in the <code>&lt;displayLabel&gt;</code> would see every directory that matches the pattern.</p>
</blockquote>
</article>
</div>
</div>
</main>
<footer class="md-footer">
<div class="md-footer-meta md-typeset">
<div class="md-footer-meta__inner md-grid">
<div class="md-footer-copyright">
<div class="md-footer-copyright__highlight">
Copyright &copy; 2022 Unidata Program Center.
</div>
powered by
<a href="http://www.mkdocs.org" title="MkDocs">MkDocs</a>
and
<a href="http://squidfunk.github.io/mkdocs-material/" title="Material for MkDocs">
Material for MkDocs</a>
</div>
<div class="md-footer-social">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
<a href="https://www.unidata.ucar.edu/software/awips2" class="md-footer-social__link fa fa-globe"></a>
<a href="https://github.com/Unidata/awips2" class="md-footer-social__link fa fa-github-alt"></a>
<a href="https://twitter.com/Unidata" class="md-footer-social__link fa fa-twitter"></a>
<a href="https://facebook.com/Unidata" class="md-footer-social__link fa fa-facebook"></a>
</div>
</div>
</div>
</footer>
</div>
<script src="../../assets/javascripts/application-eaeb7b21e3.js"></script>
<script>app.initialize({url:{base:"../.."}})</script>
</body>
</html>