xan/awips2
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
awips2/devguide/data-flow/index.html
tiffanycmeyer13 672919a556 deploy: d6ccb6daa8
2022-04-08 20:25:20 +00:00

961 lines
No EOL
27 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html class="no-js">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="description" content="Documentation for all things AWIPS.">
<link rel="canonical" href="http://unidata.github.io/awips2/devguide/data-flow/">
<meta name="author" content="support-awips@unidata.ucar.edu">
<link rel="shortcut icon" href="../../images/favicon.ico">
<meta name="generator" content="mkdocs-1.3.0, mkdocs-unidata-1.5.4">
<title>Data flow - Unidata AWIPS User Manual</title>
<script src="../../assets/javascripts/modernizr-56ade86843.js"></script>
<link rel="stylesheet" href="../../assets/stylesheets/application-b1a1975878.css">
<link rel="stylesheet" href="../../assets/stylesheets/application-f78e5cb881.palette.css">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,400i,700|Roboto+Mono">
<style>body,input{font-family:"Roboto","Helvetica Neue",Helvetica,Arial,sans-serif}code,kbd,pre{font-family:"Roboto Mono","Courier New",Courier,monospace}</style>
<link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons">
<link rel="stylesheet" href="../../css/extra.css">
</head>
<body data-md-color-primary="blue-grey" data-md-color-accent="light-blue">
<svg class="md-svg">
<defs>
<svg xmlns="http://www.w3.org/2000/svg" width="416" height="448" viewBox="0 0 416 448" id="github"><path fill="currentColor" d="M160 304q0 10-3.125 20.5t-10.75 19T128 352t-18.125-8.5-10.75-19T96 304t3.125-20.5 10.75-19T128 256t18.125 8.5 10.75 19T160 304zm160 0q0 10-3.125 20.5t-10.75 19T288 352t-18.125-8.5-10.75-19T256 304t3.125-20.5 10.75-19T288 256t18.125 8.5 10.75 19T320 304zm40 0q0-30-17.25-51T296 232q-10.25 0-48.75 5.25Q229.5 240 208 240t-39.25-2.75Q130.75 232 120 232q-29.5 0-46.75 21T56 304q0 22 8 38.375t20.25 25.75 30.5 15 35 7.375 37.25 1.75h42q20.5 0 37.25-1.75t35-7.375 30.5-15 20.25-25.75T360 304zm56-44q0 51.75-15.25 82.75-9.5 19.25-26.375 33.25t-35.25 21.5-42.5 11.875-42.875 5.5T212 416q-19.5 0-35.5-.75t-36.875-3.125-38.125-7.5-34.25-12.875T37 371.5t-21.5-28.75Q0 312 0 260q0-59.25 34-99-6.75-20.5-6.75-42.5 0-29 12.75-54.5 27 0 47.5 9.875t47.25 30.875Q171.5 96 212 96q37 0 70 8 26.25-20.5 46.75-30.25T376 64q12.75 25.5 12.75 54.5 0 21.75-6.75 42 34 40 34 99.5z"/></svg>
</defs>
</svg>
<input class="md-toggle" data-md-toggle="drawer" type="checkbox" id="drawer">
<input class="md-toggle" data-md-toggle="search" type="checkbox" id="search">
<label class="md-overlay" data-md-component="overlay" for="drawer"></label>
<header class="md-header" data-md-component="header">
<nav class="md-header-nav md-grid">
<a href="../.."><div class="background-logo" style="background-image: url(../../images/cave.png);"></div>
<div class="md-flex">
<div class="md-flex__cell md-flex__cell--shrink">
<a href="../.." title="Unidata AWIPS User Manual" class="md-header-nav__button">
</a>
</div>
<div class="md-flex__cell md-flex__cell--shrink">
<label class="md-icon md-icon--menu md-header-nav__button" for="drawer"></label>
</div>
<div class="md-flex__cell md-flex__cell--stretch">
<span class="md-flex__ellipsis md-header-nav__title">
Data flow
</span>
</div>
<div class="md-flex__cell md-flex__cell--shrink">
<label class="md-icon md-icon--search md-header-nav__button" for="search"></label>
<div class="md-search" data-md-component="search" role="dialog">
<label class="md-search__overlay" for="search"></label>
<div class="md-search__inner">
<form class="md-search__form" name="search">
<input type="text" class="md-search__input" name="query" required placeholder="Search" autocapitalize="off" autocorrect="off" autocomplete="off" spellcheck="false" data-md-component="query">
<label class="md-icon md-search__icon" for="search"></label>
<button type="reset" class="md-icon md-search__icon" data-md-component="reset">&#xE5CD;</button>
</form>
<div class="md-search__output">
<div class="md-search__scrollwrap" data-md-scrollfix>
<div class="md-search-result" data-md-component="result" data-md-lang-search="" data-md-lang-tokenizer="[\s\-]+">
<div class="md-search-result__meta" data-md-lang-result-none="No matching documents" data-md-lang-result-one="1 matching document" data-md-lang-result-other="# matching documents">
Type to start searching
</div>
<ol class="md-search-result__list"></ol>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="md-flex__cell md-flex__cell--shrink">
<div class="md-header-nav__source">
<a href="https://github.com/Unidata/awips2" title="Go to repository" class="md-source" data-md-source="github">
<div class="md-source__icon">
<svg viewBox="0 0 24 24" width="24" height="24">
<use xlink:href="#github" width="24" height="24"></use>
</svg>
</div>
<div class="md-source__repository">
GitHub
</div>
</a>
</div>
</div>
</div>
</nav>
</header>
<div class="md-container">
<main class="md-main">
<div class="md-main__inner md-grid" data-md-component="container">
<div class="md-sidebar md-sidebar--primary" data-md-component="navigation">
<div class="md-sidebar__scrollwrap">
<div class="md-sidebar__inner">
<nav class="md-nav md-nav--primary" data-md-level="0">
<label class="md-nav__title md-nav__title--site" for="drawer">
<a href="http://unidata.github.io/awips2/" title="Unidata AWIPS User Manual" class="md-nav__button md-logo">
<i class="md-icon md-icon--home"></i>
</a>
Unidata AWIPS User Manual
</label>
<div class="md-nav__source">
<a href="https://github.com/Unidata/awips2" title="Go to repository" class="md-source" data-md-source="github">
<div class="md-source__icon">
<svg viewBox="0 0 24 24" width="24" height="24">
<use xlink:href="#github" width="24" height="24"></use>
</svg>
</div>
<div class="md-source__repository">
GitHub
</div>
</a>
</div>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../.." title="Home" class="md-nav__link">
Home
</a>
</li>
<li class="md-nav__item">
<a href="../../install/install-cave/" title="Install CAVE" class="md-nav__link">
Install CAVE
</a>
</li>
<li class="md-nav__item">
<a href="../../install/install-edex/" title="Install EDEX" class="md-nav__link">
Install EDEX
</a>
</li>
<li class="md-nav__item">
<a href="../../install/common-problems/" title="Common Problems" class="md-nav__link">
Common Problems
</a>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-toggle md-nav__toggle" data-md-toggle="nav-5" type="checkbox" id="nav-5">
<label class="md-nav__link" for="nav-5">
CAVE User Manual
</label>
<nav class="md-nav" data-md-component="collapsible" data-md-level="1">
<label class="md-nav__title" for="nav-5">
CAVE User Manual
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../cave/d2d-perspective/" title="D2D Perspective" class="md-nav__link">
D2D Perspective
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/maps-views-projections/" title="Maps, Views, Projections" class="md-nav__link">
Maps, Views, Projections
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/bundles-and-procedures/" title="Displays and Procedures" class="md-nav__link">
Displays and Procedures
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/localization-perspective/" title="Localization Perspective" class="md-nav__link">
Localization Perspective
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/nsharp/" title="NSHARP" class="md-nav__link">
NSHARP
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/warngen/" title="WarnGen Walkthrough" class="md-nav__link">
WarnGen Walkthrough
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/goes-16-17-satellite/" title="GOES 16/17" class="md-nav__link">
GOES 16/17
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/d2d-gis-shapefiles/" title="GIS and Shapefiles" class="md-nav__link">
GIS and Shapefiles
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/d2d-gridded-models/" title="Volume Browser" class="md-nav__link">
Volume Browser
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/d2d-tools/" title="Display Tools" class="md-nav__link">
Display Tools
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/d2d-radar-tools/" title="Radar Tools" class="md-nav__link">
Radar Tools
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/d2d-edit-menus/" title="Editing Menus" class="md-nav__link">
Editing Menus
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/cave-localization/" title="Change Localization" class="md-nav__link">
Change Localization
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/import-export/" title="Import/Export" class="md-nav__link">
Import/Export
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-toggle md-nav__toggle" data-md-toggle="nav-6" type="checkbox" id="nav-6">
<label class="md-nav__link" for="nav-6">
EDEX User Manual
</label>
<nav class="md-nav" data-md-component="collapsible" data-md-level="1">
<label class="md-nav__title" for="nav-6">
EDEX User Manual
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../install/start-edex/" title="EDEX Basic Commands" class="md-nav__link">
EDEX Basic Commands
</a>
</li>
<li class="md-nav__item">
<a href="../../edex/settings/" title="EDEX Settings" class="md-nav__link">
EDEX Settings
</a>
</li>
<li class="md-nav__item">
<a href="../../edex/distributed-computing/" title="Distributed EDEX" class="md-nav__link">
Distributed EDEX
</a>
</li>
<li class="md-nav__item">
<a href="../../edex/edex-ingest-docker-container/" title="Docker EDEX" class="md-nav__link">
Docker EDEX
</a>
</li>
<li class="md-nav__item">
<a href="../../edex/ldm/" title="LDM Feeds" class="md-nav__link">
LDM Feeds
</a>
</li>
<li class="md-nav__item">
<a href="../../edex/data-distribution-files/" title="Data Distribution Files" class="md-nav__link">
Data Distribution Files
</a>
</li>
<li class="md-nav__item">
<a href="../../edex/new-grid/" title="Ingest a New Grid" class="md-nav__link">
Ingest a New Grid
</a>
</li>
<li class="md-nav__item">
<a href="../../edex/data-purge/" title="Purging and Retention" class="md-nav__link">
Purging and Retention
</a>
</li>
<li class="md-nav__item">
<a href="../../edex/edex-users/" title="Monitor Users" class="md-nav__link">
Monitor Users
</a>
</li>
<li class="md-nav__item">
<a href="../../edex/data-plugins/" title="Data Plugins" class="md-nav__link">
Data Plugins
</a>
</li>
<li class="md-nav__item">
<a href="../../edex/case-studies/" title="Archive Case Studies" class="md-nav__link">
Archive Case Studies
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="../../python/overview/" title="Python-AWIPS" class="md-nav__link">
Python-AWIPS
</a>
</li>
<li class="md-nav__item">
<a href="../../appendix/educational-resources/" title="Educational Resources" class="md-nav__link">
Educational Resources
</a>
</li>
<li class="md-nav__item">
<a href="../../dev/awips-development-environment/" title="Development" class="md-nav__link">
Development
</a>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-toggle md-nav__toggle" data-md-toggle="nav-10" type="checkbox" id="nav-10">
<label class="md-nav__link" for="nav-10">
Appendix
</label>
<nav class="md-nav" data-md-component="collapsible" data-md-level="1">
<label class="md-nav__title" for="nav-10">
Appendix
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../appendix/appendix-grid-parameters/" title="AWIPS Grid Parameters" class="md-nav__link">
AWIPS Grid Parameters
</a>
</li>
<li class="md-nav__item">
<a href="../../python/maps-database/" title="Maps Database" class="md-nav__link">
Maps Database
</a>
</li>
<li class="md-nav__item">
<a href="../../appendix/appendix-acronyms/" title="Acronyms and Abbreviations" class="md-nav__link">
Acronyms and Abbreviations
</a>
</li>
<li class="md-nav__item">
<a href="../../appendix/appendix-wsr88d/" title="WSR-88D Product Table" class="md-nav__link">
WSR-88D Product Table
</a>
</li>
<li class="md-nav__item">
<a href="../../cave/cave-keyboard-shortcuts/" title="Keyboard Shortcuts" class="md-nav__link">
Keyboard Shortcuts
</a>
</li>
<li class="md-nav__item">
<a href="../../raytheon/cave_d2d/" title="Raytheon: CAVE User's Manual" class="md-nav__link">
Raytheon: CAVE User's Manual
</a>
</li>
<li class="md-nav__item">
<a href="../../raytheon/smm/" title="Raytheon: AWIPS System Manager's Manual" class="md-nav__link">
Raytheon: AWIPS System Manager's Manual
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div class="md-sidebar md-sidebar--secondary" data-md-component="toc">
<div class="md-sidebar__scrollwrap">
<div class="md-sidebar__inner">
<nav class="md-nav md-nav--secondary">
<label class="md-nav__title" for="toc">Table of contents</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#data-receipt" title="Data Receipt" class="md-nav__link">
Data Receipt
</a>
</li>
<li class="md-nav__item">
<a href="#data-decoding" title="Data Decoding" class="md-nav__link">
Data Decoding
</a>
<nav class="md-nav">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#processed-data-storage-architecture" title="Processed Data Storage Architecture" class="md-nav__link">
Processed Data Storage Architecture
</a>
</li>
<li class="md-nav__item">
<a href="#data-retrieval-architecture" title="Data Retrieval Architecture" class="md-nav__link">
Data Retrieval Architecture
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#data-purge-architecture" title="Data Purge Architecture" class="md-nav__link">
Data Purge Architecture
</a>
<nav class="md-nav">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#raw-data-purge" title="Raw Data Purge" class="md-nav__link">
Raw Data Purge
</a>
</li>
<li class="md-nav__item">
<a href="#processed-data-purge" title="Processed Data Purge" class="md-nav__link">
Processed Data Purge
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div class="md-content">
<article class="md-content__inner md-typeset">
<a href="https://github.com/Unidata/awips2/edit/unidata_18.2.1/docs/devguide/data-flow.md" title="Edit this page" class="md-icon md-content__icon">edit</a>
<h1>Data flow</h1>
<h2 id="data-receipt">Data Receipt<a class="headerlink" href="#data-receipt" title="Permanent link"></a></h2>
<ol>
<li>The LDM obtains a data product from an upstream LDM site on the IDD.</li>
<li>The LDM writes the data to file in Raw Data Storage.</li>
<li>The LDM uses edexBridge to post a “data available” message to the Qpid message broker.</li>
<li>The EDEX Ingest process obtains the “data available” message from Qpid and removes the message from the message queue.</li>
<li>The EDEX Ingest process obtains the data files from Raw Data Storage. </li>
</ol>
<p>This architecture provides separation between data sources and ingest processing. Any data source, not just the LDM/IDD, can follow this architecture to provide data for EDEX to process.</p>
<h2 id="data-decoding">Data Decoding<a class="headerlink" href="#data-decoding" title="Permanent link"></a></h2>
<p>Data decoding is defined as the process of converting data from a raw format into a decoded format that is usable by CAVE. In AWIPS, data decoding is performed by the EDEX Ingest proessing (<strong>ingest</strong> and <strong>ingestGrib</strong>).</p>
<ol>
<li>EDEX Ingest obtains the “data available” message from the Qpid message broker, and determines the appropriate data decoder based on the message contents. EDEX Ingest then forwards the message to the chosen decoder. Finally, the message is removed from the message queue.</li>
<li>EDEX Ingest reads the data from Raw Data Storage.</li>
<li>EDEX Ingest decodes the data.</li>
<li>EDEX Ingest forwards the decoded data to Processed Data Storage.</li>
<li>EDEX Ingest sends a message to the CAVE client indicating that newly-decoded data is available.</li>
</ol>
<p>It is important to note that in AWIPS all data types are processed by either the standard <strong>ingest</strong> process, or by the <strong>ingestGrib</strong> process, which handles all grib message ingestion. Once this data decoding process is complete, clients may obtain and perform additional processing on the data, including displaying data in CAVE.</p>
<h3 id="processed-data-storage-architecture">Processed Data Storage Architecture<a class="headerlink" href="#processed-data-storage-architecture" title="Permanent link"></a></h3>
<p>Processed Data Storage refers to the persistence of decoded data and occurs in two separate forms: 1) metadata and some decoded data, which is stored in Postgres database tables; and 2) imagery data, gridded forecast data, and certain point data, which is stored in HDF5 files, and is managed by PyPIES. </p>
<p>Writing to Processed Data Storage involves the following:</p>
<ul>
<li>1) The <strong>EDEX Process</strong> sends serialized data, area data, and certain point data to PyPIES.</li>
<li>2) <strong>PyPIES</strong> writes the data to the HDF5 data store.</li>
<li>3) <strong>EDEX</strong> send the metadata to the Postgres DBMS</li>
<li>4) <strong>Postgres</strong> writes the metadata to the AWIPS database.</li>
</ul>
<p>For data not stored in HDF5, Steps 1 and 2 are skipped.</p>
<p>For processed data retrieval, the process is revered:</p>
<ul>
<li>3) <strong>EDEX</strong> requests the metadata from Postgres.</li>
<li>4) <strong>Postgres</strong> reads the AWIPS database and returns the requested metadata to EDEX.</li>
<li>1) <strong>EDEX</strong> sends a data request to PyPIES.</li>
<li>2) <strong>PyPIES</strong> reads the data from the HDF5 data store and returns it to EDEX.</li>
</ul>
<p>In this case, if the data is not stored in HDF5, then Steps 3 and 4 are skipped.</p>
<h3 id="data-retrieval-architecture">Data Retrieval Architecture<a class="headerlink" href="#data-retrieval-architecture" title="Permanent link"></a></h3>
<p>Data retrieval is the process by which the CAVE client obtains data using the EDEX Request server; the Request server obtains the data from processed data storage (Postgres and HDF5) and returns it to CAVE. </p>
<ol>
<li><strong>CAVE</strong> sends a request via TCP to the EDEX Request server.</li>
<li><strong>EDEX Request</strong> server obtains the requested metadata via Postgres and stored data via PyPIES.</li>
<li><strong>EDEX Request</strong> forwards the requested data directly back to the CAVE client.</li>
</ol>
<p>For clustered EDEX servers using IPVS, this architecture allows CAVE clients to access any available EDEX Request process, providing an improvement in system reliability and speed. Data retrieval from processed data storage follows the same pattern as data storage: retrieval of HDF5 is handled by PyPIES; retrieval of database data is handled by Postgres.</p>
<h2 id="data-purge-architecture">Data Purge Architecture<a class="headerlink" href="#data-purge-architecture" title="Permanent link"></a></h2>
<p>Raw data storage and processed data storage use two different purge mechanisms. For processed data storage, AWIPS implements a plugin based purge strategy, where the user has the option to change the purge frequency for each plugin individually. </p>
<h3 id="raw-data-purge">Raw Data Purge<a class="headerlink" href="#raw-data-purge" title="Permanent link"></a></h3>
<p>Purging of Raw Data Storage is managed by the LDM user account cron, which executes the ldmadmin scour process, removing data files using an age-based strategy. The directories and retention times for raw data storage are controlled by <em>scour.conf</em>, which is located in the ldm user's <em>~/etc/</em> directory. Each entry in scour.conf contains the directory to manage, the retention time and an optional file name pattern for data files. </p>
<ol>
<li>An <strong>ldm</strong> user cron job executes ldmadmin.</li>
<li><strong>ldmadmin</strong> executes the LDM scour program.</li>
<li>The <strong>LDM scour program</strong> deletes outdated data from AWIPS Raw Data Storage.</li>
</ol>
<h3 id="processed-data-purge">Processed Data Purge<a class="headerlink" href="#processed-data-purge" title="Permanent link"></a></h3>
<p>Rules for this version-based purge are contained in XML files located in <strong>/awips2/edex/data/utility/</strong>. The purge is triggered by a quartz timer event that fires at 30 minutes after each hour.</p>
<ol>
<li>A <strong>Quartz</strong> event is triggered in the EDEX Ingest process causing the Purge Service to obtain a purge lock. If the lock is already taken, the Purge Service will exit, ensuring that only a single EDEX Ingest process performs the purge.</li>
<li>The <strong>EDEX Purge Service</strong> sends a delete message to Postgres.</li>
<li><strong>Postgres</strong> deletes the specified data from the database.</li>
<li>If HDF5 data is to be purged, the <strong>Purge Service</strong> messages PyPIES.</li>
<li><strong>PyPIES</strong> deletes the specified HDF5 files.</li>
</ol>
</article>
</div>
</div>
</main>
<footer class="md-footer">
<div class="md-footer-meta md-typeset">
<div class="md-footer-meta__inner md-grid">
<div class="md-footer-copyright">
<div class="md-footer-copyright__highlight">
Copyright &copy; 2020 Unidata Program Center.
</div>
powered by
<a href="http://www.mkdocs.org" title="MkDocs">MkDocs</a>
and
<a href="http://squidfunk.github.io/mkdocs-material/" title="Material for MkDocs">
Material for MkDocs</a>
</div>
<div class="md-footer-social">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
<a href="https://www.unidata.ucar.edu/software/awips2" class="md-footer-social__link fa fa-globe"></a>
<a href="https://github.com/Unidata/awips2" class="md-footer-social__link fa fa-github-alt"></a>
<a href="https://twitter.com/Unidata" class="md-footer-social__link fa fa-twitter"></a>
<a href="https://facebook.com/Unidata" class="md-footer-social__link fa fa-facebook"></a>
</div>
</div>
</div>
</footer>
</div>
<script src="../../assets/javascripts/application-0b7df094bf.js"></script>
<script>app.initialize({url:{base:"../.."}})</script>
<script src="../../assets/javascripts/application-0b7df094bf.js"></script>
</body>
</html>
Reference in a new issue View git blame Copy permalink