For this example, this server will be referred to by the IP address 10.0.0.9.
groupadd fxalpha && useradd -G fxalpha awips
-mkdir /awips2
-wget -O /etc/yum.repos.d/awips2.repo https://downloads.unidata.ucar.edu/awips2/current/linux/awips2.repo
-yum clean all
-yum groupinstall awips2-database
+wget https://downloads.unidata.ucar.edu/awips2/current/linux/awips_install.sh
+chmod 755 awips_install.sh
+sudo ./awips_install.sh --database
2. IPtables Config
It is required that ports 5432 and 5672 be open for the specific IP addresses of outside EDEX ingest servers. It is not recommended that you leave port 5432 open to all connections (since the default awips database password is known, and is not meant as a security measure). Further, it is recommended that you change the default postgres awips user password (which then requires a reconfiguration of every remote EDEX ingest server in order to connect to this database/request server).
@@ -1076,14 +1102,19 @@ COMMIT
The two ports left open to all connections (9581,9582) in addition to default port 22 are for outside CAVE client connections
3. Database Config
-In the file /awips2/database/data/pg_hba.conf you define remote connections for all postgres tables with as <IP address>/32, after the block of IPv4 local connections:
+In the file /awips2/database/data/pg_hba.conf you define remote connections for all postgres tables with as <IP address>/32, after the block of IPv4 local connections and generic <IP address/24> for hostnossl:
vi /awips2/database/data/pg_hba.conf
# "local" is for Unix domain socket connections only
-local all all trust
-hostssl all all 10.0.0.7/32 cert clientcert=1
-hostssl all all 162.0.0.0/8 cert clientcert=1
-hostssl all all 127.0.0.1/32 cert clientcert=1
+local all all trust
+hostssl all all 162.0.0.0/8 cert clientcert=1
+hostssl all all 127.0.0.1/32 cert clientcert=1
+
+hostssl all all 10.0.0.7/32 cert clientcert=1
+hostnossl postgres all 10.0.0.0/24 md5
+hostnossl fxatext all 10.0.0.0/24 md5
+hostnossl metadata all 10.0.0.0/24 md5
+
# IPv6 local connections:
hostssl all all ::1/128 cert clientcert=1
hostnossl all all ::1/128 md5
@@ -1123,18 +1154,18 @@ Found 499 db classes in 720 ms
If the log stops at the Found db classes... line, that means EDEX is not connecting to PostgreSQL - double-check DB_ADDR in /awips2/edex/bin/setup.env
-Ingest/Decode Server
-For this example, this server will be referred to by the IP address 10.0.0.7.
+Ancillary EDEX Server (Ingest/Decode EDEX Server)
+For this example, this server will be referred to by the IP address 10.0.0.7.
+The Main EDEX server will be referred to by the IP address 10.0.0.9.
1. Install
-groupadd fxalpha && useradd -G fxalpha awips
-wget -O /etc/yum.repos.d/awips2.repo https://downloads.unidata.ucar.edu/awips2/current/linux/awips2.repo
-yum clean all
-yum groupinstall awips2-ingest
+wget https://downloads.unidata.ucar.edu/awips2/current/linux/awips_install.sh
+chmod 755 awips_install.sh
+sudo ./awips_install.sh --ingest
2. EDEX Config
vi /awips2/edex/bin/setup.env
-Here you should redefine DB_ADDR and PYPIES_SERVER to point to the Database/Request server (10.0.0.9)
-export EDEX_SERVER=10.0.0.7
+Here you should redefine DB_ADDR and PYPIES_SERVER to point to the Main or Database/Request server (10.0.0.9) and the EXT_ADDR to point to the current Ingest server (10.0.0.7)
+export EXT_ADDR=10.0.0.7
# postgres connection
export DB_ADDR=10.0.0.9
@@ -1144,13 +1175,46 @@ export DB_PORT=5432
export PYPIES_SERVER=http://10.0.0.9:9582
# qpid connection
-export BROKER_ADDR=${EDEX_SERVER}
+export BROKER_ADDR=${EXT_ADDR}
-Notice that EDEX_SERVER and BROKER_ADDR (qpid) should remain defined as the localhost IP address (10.0.0.7)
-3. Start EDEX
-edex start ingest
+Notice that EXT_ADDR and BROKER_ADDR (qpid) should remain defined as the localhost IP address (10.0.0.7)
+3. Modify the edexServiceList
+Most likely if you are running a distributed EDEX setup, you are only processing a subset of data. You can change your edexServiceList to only run the processes you need. You will need to update the /etc/init.d/edexServiceList file. For example replace the services with the associated right column based on the data you're processing:
+export SERVICES=('')
-This will start Qpid and the EDEX Ingest and IngestGrib JVMs (and not start PostgreSQL, httpd-pypies, or the EDEX Request JVM)
+
+
+
+Data Processing:
+edexServiceList
+
+
+
+
+radar
+ingestRadar
+
+
+satellite
+ingestGoesR
+
+
+model
+ingestGrids, ingestGrib
+
+
+
+4. Configure your LDM
+You'll want to modify your pqact.conf file to store only the data you want processed. There are example files in /awips2/ldm/etc that you can copy over to the main pqact.conf file. For example if you are wanting to process goesr data only, you can do the following steps:
+cd /awips2/ldm/etc
+mv pqact.conf pqact.conf.orig
+cp pqact.goesr pqact.conf
+
+You will also want to edit the pqact.conf file on your Main EDEX and comment out any entries you're processing on this EDEX server.
+5. Start EDEX
+edex start
+
+This will start LDM, Qpid and the specified EDEX Ingest JVMs (and not start PostgreSQL, httpd-pypies, or the EDEX Request JVM)
4. Monitor Services
Watch the edex JVM log with the command
edex log
@@ -1185,6 +1249,9 @@ psql -U awips -h <remote IP address> -p 5432 metadata
Every EDEX Ingest IP address must be allowed in both iptables and pg_hba.conf as shown above.
+
+Data processed on
+
diff --git a/search/search_index.json b/search/search_index.json
index 0d1816a2b1..da7fa425fb 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1422,7 +1422,7 @@
},
{
"location": "/edex/distributed-computing/",
- "text": "Distributed EDEX\n\uf0c1\n\n\nAWIPS makes use of service-oriented architecture to request, process, and serve real-time meteorological data. While originally developed for use on internal NWS forecast office networks, where operational installations of AWIPS could consist of a dozen servers or more, the early Unidata releases were stripped of operations-specific configurations and plugins, and released as a standalone server. This worked, since (at the time) a single EDEX instance with an attached SSD could handle most of NOAAport. However, with GOES-R(16) coming online in 2017, and more gridded forecast models being created at finer temporal and spatial resolutions, there is now a need to distribute the data decoding across multiple machines to handle this firehose of data.\n\n\n\n\nUnidata's Current EDEX Server\n\uf0c1\n\n\nCurrently, with our specific EDEX server we use a Database/Request instance that also decodes and ingests a good portion of the data. It handles all data requests from CAVE users, as well as the majority of the decoding and ingesting for data feeds coming down on the LDM. The \nradar\n data has been specifically exluded (from the decoding and ingest) and it has its own \nIngest/Decode Server\n which is explained in more detail below.\n\n\nFor our EDEX we have designated an instance of the ingest/decoding server to be dedicated to handling the radar data. Our \nRadar-EDEX\n recieves and decodes all radar down from the LDM and then stores it back on our main \nDatabase/Request EDEX\n in the form of HDF5 data files and PostgreSQL metadata.\n\n\n\n\nExample Installation\n\uf0c1\n\n\nThis walkthrough will install different EDEX components on two machines in the XSEDE Jetstream Cloud, the first is used to \nstore and serve\n while the second is used to \ningest and decode\n data.\n\n\n\n\n\n\nDatabase/Request Server\n\uf0c1\n\n\nFor this example, this server will be referred to by the IP address \n10.0.0.9\n.\n\n\n1. Install\n\uf0c1\n\n\ngroupadd fxalpha && useradd -G fxalpha awips\nmkdir /awips2\nwget -O /etc/yum.repos.d/awips2.repo https://downloads.unidata.ucar.edu/awips2/current/linux/awips2.repo\nyum clean all\nyum groupinstall awips2-database\n\n\n\n2. IPtables Config\n\uf0c1\n\n\nIt is required that ports 5432 and 5672 be open for the specific IP addresses of outside EDEX ingest servers. It is \nnot recommended\n that you leave port 5432 open to all connections (since the default awips database password is known, and is not meant as a security measure). Further, it \nis recommended\n that you change the default postgres awips user password (which then requires a reconfiguration of every remote EDEX ingest server in order to connect to this database/request server).\n\n\nvi /etc/sysconfig/iptables\n\n*filter\n:INPUT DROP [0:0]\n:FORWARD DROP [0:0]\n:OUTPUT ACCEPT [0:0]\n:EXTERNAL - [0:0]\n:EDEX - [0:0]\n-A INPUT -i lo -j ACCEPT\n-A INPUT -p icmp --icmp-type any -j ACCEPT\n-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT\n-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT\n-A INPUT -m state --state NEW -m tcp -p tcp --dport 9581 -j ACCEPT\n-A INPUT -m state --state NEW -m tcp -p tcp --dport 9582 -j ACCEPT\n-A INPUT -s 10.0.0.7 -j EDEX\n-A INPUT -j EXTERNAL\n-A EXTERNAL -j REJECT\n-A EDEX -m state --state NEW -p tcp --dport 5432 -j ACCEPT\n-A EDEX -m state --state NEW -p tcp --dport 5672 -j ACCEPT\n-A EDEX -j REJECT\nCOMMIT\n\n\n\nNote the line \n-A INPUT -s 10.0.0.7 -j EDEX\n as well as the following \n-A EDEX ...\n rules for ports 5432 (PostgreSQL) and 5672 (PyPIES/HDF5). \n\n\n\n\nThe two ports left open to all connections (9581,9582) in addition to default port 22 are for outside CAVE client connections\n\n\n\n\n3. Database Config\n\uf0c1\n\n\nIn the file \n/awips2/database/data/pg_hba.conf\n you define remote connections for all postgres tables with as \n/32\n, after the block of IPv4 local connections:\n\n\nvi /awips2/database/data/pg_hba.conf\n\n# \"local\" is for Unix domain socket connections only\nlocal all all trust\nhostssl all all 10.0.0.7/32 cert clientcert=1\nhostssl all all 162.0.0.0/8 cert clientcert=1\nhostssl all all 127.0.0.1/32 cert clientcert=1\n# IPv6 local connections:\nhostssl all all ::1/128 cert clientcert=1\nhostnossl all all ::1/128 md5\n\n\n\n4. Start EDEX\n\uf0c1\n\n\nedex start database\n\n\n\nThis will start PostgreSQL, httpd-pypies, Qpid, and the EDEX Request JVM (and will not start the LDM or the EDEX Ingest and IngestGrib JVMs)\n\n\n5. Monitor Services\n\uf0c1\n\n\nThe command \nedex\n will show which services are running, and for a Database/Request server, will not include the LDM, EDEXingest, or EDEXgrib:\n\n\nedex\n\n[edex status]\npostgres :: running :: pid 571\npypies :: running :: pid 639\nqpid :: running :: pid 674\nEDEXingest :: not running\nEDEXgrib :: not running\nEDEXrequest :: running :: pid 987 1029 23792\n\n\n\nSince this Database/Request server is not running the main \nedexIngest\n JVM, we won't see anything from \nedex log\n, instead watch the Request Server with the command\n\n\nedex log request\n\n\n\n\n\nConfirm that EDEX Request connects to PostgreSQL!\n\n\nWith the above \nedex log request\n, ensure that the log progresses \npast this point\n:\n\n\nSpring-enabled Plugins:\n-----------------------\nacars-common, acars-common-dataaccess, acarssounding-common, activetable-common,\nactivetable-request, airep-common, airep-common-dataaccess, airmet-common, \natcf-common, atcf-request, auth-request, awipstools-request, aww-common...\n\nJAXB context for PersistencePathKeySet inited in: 5ms\nINFO 20:21:09,134 5584 [EDEXMain] Reflections: Reflections took 436 ms to scan 258 urls, producing 31 keys and 3637 values\nFound 499 db classes in 720 ms\n\n\n\nIf the log stops at the \nFound db classes...\n line, that means EDEX is not connecting to PostgreSQL - double-check \nDB_ADDR\n in \n/awips2/edex/bin/setup.env\n\n\n\n\n\n\nIngest/Decode Server\n\uf0c1\n\n\nFor this example, this server will be referred to by the IP address \n10.0.0.7\n.\n\n\n1. Install\n\uf0c1\n\n\ngroupadd fxalpha && useradd -G fxalpha awips\nwget -O /etc/yum.repos.d/awips2.repo https://downloads.unidata.ucar.edu/awips2/current/linux/awips2.repo\nyum clean all\nyum groupinstall awips2-ingest\n\n\n\n2. EDEX Config\n\uf0c1\n\n\nvi /awips2/edex/bin/setup.env\n\n\nHere you should redefine \nDB_ADDR\n and \nPYPIES_SERVER\n to point to the \nDatabase/Request\n server (10.0.0.9)\n\n\nexport EDEX_SERVER=10.0.0.7\n\n# postgres connection\nexport DB_ADDR=10.0.0.9\nexport DB_PORT=5432\n\n# pypies hdf5 connection\nexport PYPIES_SERVER=http://10.0.0.9:9582\n\n# qpid connection\nexport BROKER_ADDR=${EDEX_SERVER}\n\n\n\nNotice that \nEDEX_SERVER\n and \nBROKER_ADDR\n (qpid) should remain defined as the \nlocalhost\n IP address (10.0.0.7)\n\n\n3. Start EDEX\n\uf0c1\n\n\nedex start ingest\n\n\n\nThis will start Qpid and the EDEX Ingest and IngestGrib JVMs (and not start PostgreSQL, httpd-pypies, or the EDEX Request JVM)\n\n\n4. Monitor Services\n\uf0c1\n\n\nWatch the edex JVM log with the command\n\n\nedex log\n\n\n\n\n\nConfirm that EDEX connects to PostgreSQL!\n\n\nWith the above \nedex log\n, ensure that the log progresses \npast this point\n:\n\n\nSpring-enabled Plugins:\n-----------------------\nacars-common, acars-common-dataaccess, acarssounding-common, activetable-common,\nactivetable-ingest, airep-common, airep-common-dataaccess, airmet-common, \natcf-common, atcf-ingest, aww-common...\n\nJAXB context for PersistencePathKeySet inited in: 5ms\nINFO 20:21:09,134 5584 [EDEXMain] Reflections: Reflections took 436 ms to scan 258 urls, producing 31 keys and 3637 values\nFound 499 db classes in 720 ms\n\n\n\nIf the log stops at the \nFound db classes...\n line, that means EDEX is not connecting to the \nremote PostgreSQL instance\n - double-check \nDB_ADDR\n in \n/awips2/edex/bin/setup.env\n\n\nYou can \nmanually check remote PostgreSQL connectivity\n on any EDEX Ingest server from the command line:\n\n\nsu - awips\npsql -U awips -h -p 5432 metadata\n\n\n\nWhere the default passwd is \nawips\n and is defined in files in \n/awips2/edex/conf/db/hibernateConfig/\n\n\n\n\n\n\nAdditional Notes\n\uf0c1\n\n\n\n\nBe mindful of what IP address and hostnames are used in \n/awips2/edex/bin/setup.env\n and \n/awips2/database/data/pg_hba.conf\n, and that they are resolvable from the command line. Consult or edit \n/etc/hosts\n as needed.\n\n\n\n\nYou can install multiple \nawips2-ingest\n servers, each decoding a different dataset or feed, all pointing to the same Database/Request server (\nDB_ADDR\n and \nPYPIES_SERVER\n in \n/awips2/edex/bin/setup.env\n):\n\n\n\n\n\n\nEvery EDEX Ingest IP address must be allowed in both \niptables\n and \npg_hba.conf\n as \nshown above\n.",
+ "text": "Distributed EDEX\n\uf0c1\n\n\nAWIPS makes use of service-oriented architecture to request, process, and serve real-time meteorological data. While originally developed for use on internal NWS forecast office networks, where operational installations of AWIPS could consist of a dozen servers or more, the early Unidata releases were stripped of operations-specific configurations and plugins, and released as a standalone server. This worked, since (at the time) a single EDEX instance with an attached SSD could handle most of NOAAport. However, with GOES-R(16) coming online in 2017, and more gridded forecast models being created at finer temporal and spatial resolutions, there is now a need to distribute the data decoding across multiple machines to handle this firehose of data.\n\n\n\n\nUnidata's Current EDEX Server\n\uf0c1\n\n\nCurrently, with our specific EDEX server we use a Database/Request instance that also decodes and ingests a good portion of the data. It handles all data requests from CAVE users, as well as the majority of the decoding and ingesting for data feeds coming down on the LDM. The \nradar\n data has been specifically exluded (from the decoding and ingest) and it has its own \nIngest/Decode Server\n which is explained in more detail below.\n\n\nFor our EDEX we have designated an instance of the ingest/decoding server to be dedicated to handling the radar data. Our \nRadar-EDEX\n recieves and decodes all radar down from the LDM and then stores it back on our main \nDatabase/Request EDEX\n in the form of HDF5 data files and PostgreSQL metadata.\n\n\n\n\nExample Installation\n\uf0c1\n\n\nThis walkthrough will install different EDEX components on two machines in the XSEDE Jetstream Cloud, the first is used to \nstore and serve\n while the second is used to \ningest and decode\n data.\n\n\n\n\n\n\nDatabase/Request Server\n\uf0c1\n\n\nFor this example, this server will be referred to by the IP address \n10.0.0.9\n.\n\n\n1. Install\n\uf0c1\n\n\nwget https://downloads.unidata.ucar.edu/awips2/current/linux/awips_install.sh\nchmod 755 awips_install.sh\nsudo ./awips_install.sh --database\n\n\n\n2. IPtables Config\n\uf0c1\n\n\nIt is required that ports 5432 and 5672 be open for the specific IP addresses of outside EDEX ingest servers. It is \nnot recommended\n that you leave port 5432 open to all connections (since the default awips database password is known, and is not meant as a security measure). Further, it \nis recommended\n that you change the default postgres awips user password (which then requires a reconfiguration of every remote EDEX ingest server in order to connect to this database/request server).\n\n\nvi /etc/sysconfig/iptables\n\n*filter\n:INPUT DROP [0:0]\n:FORWARD DROP [0:0]\n:OUTPUT ACCEPT [0:0]\n:EXTERNAL - [0:0]\n:EDEX - [0:0]\n-A INPUT -i lo -j ACCEPT\n-A INPUT -p icmp --icmp-type any -j ACCEPT\n-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT\n-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT\n-A INPUT -m state --state NEW -m tcp -p tcp --dport 9581 -j ACCEPT\n-A INPUT -m state --state NEW -m tcp -p tcp --dport 9582 -j ACCEPT\n-A INPUT -s 10.0.0.7 -j EDEX\n-A INPUT -j EXTERNAL\n-A EXTERNAL -j REJECT\n-A EDEX -m state --state NEW -p tcp --dport 5432 -j ACCEPT\n-A EDEX -m state --state NEW -p tcp --dport 5672 -j ACCEPT\n-A EDEX -j REJECT\nCOMMIT\n\n\n\nNote the line \n-A INPUT -s 10.0.0.7 -j EDEX\n as well as the following \n-A EDEX ...\n rules for ports 5432 (PostgreSQL) and 5672 (PyPIES/HDF5). \n\n\n\n\nThe two ports left open to all connections (9581,9582) in addition to default port 22 are for outside CAVE client connections\n\n\n\n\n3. Database Config\n\uf0c1\n\n\nIn the file \n/awips2/database/data/pg_hba.conf\n you define remote connections for all postgres tables with as \n/32\n, after the block of IPv4 local connections and generic \n\n for hostnossl:\n\n\nvi /awips2/database/data/pg_hba.conf\n\n# \"local\" is for Unix domain socket connections only\nlocal all all trust\nhostssl all all 162.0.0.0/8 cert clientcert=1\nhostssl all all 127.0.0.1/32 cert clientcert=1\n\nhostssl all all 10.0.0.7/32 cert clientcert=1\nhostnossl postgres all 10.0.0.0/24 md5\nhostnossl fxatext all 10.0.0.0/24 md5\nhostnossl metadata all 10.0.0.0/24 md5\n\n# IPv6 local connections:\nhostssl all all ::1/128 cert clientcert=1\nhostnossl all all ::1/128 md5\n\n\n\n4. Start EDEX\n\uf0c1\n\n\nedex start database\n\n\n\nThis will start PostgreSQL, httpd-pypies, Qpid, and the EDEX Request JVM (and will not start the LDM or the EDEX Ingest and IngestGrib JVMs)\n\n\n5. Monitor Services\n\uf0c1\n\n\nThe command \nedex\n will show which services are running, and for a Database/Request server, will not include the LDM, EDEXingest, or EDEXgrib:\n\n\nedex\n\n[edex status]\npostgres :: running :: pid 571\npypies :: running :: pid 639\nqpid :: running :: pid 674\nEDEXingest :: not running\nEDEXgrib :: not running\nEDEXrequest :: running :: pid 987 1029 23792\n\n\n\nSince this Database/Request server is not running the main \nedexIngest\n JVM, we won't see anything from \nedex log\n, instead watch the Request Server with the command\n\n\nedex log request\n\n\n\n\n\nConfirm that EDEX Request connects to PostgreSQL!\n\n\nWith the above \nedex log request\n, ensure that the log progresses \npast this point\n:\n\n\nSpring-enabled Plugins:\n-----------------------\nacars-common, acars-common-dataaccess, acarssounding-common, activetable-common,\nactivetable-request, airep-common, airep-common-dataaccess, airmet-common, \natcf-common, atcf-request, auth-request, awipstools-request, aww-common...\n\nJAXB context for PersistencePathKeySet inited in: 5ms\nINFO 20:21:09,134 5584 [EDEXMain] Reflections: Reflections took 436 ms to scan 258 urls, producing 31 keys and 3637 values\nFound 499 db classes in 720 ms\n\n\n\nIf the log stops at the \nFound db classes...\n line, that means EDEX is not connecting to PostgreSQL - double-check \nDB_ADDR\n in \n/awips2/edex/bin/setup.env\n\n\n\n\n\n\nAncillary EDEX Server (Ingest/Decode EDEX Server)\n\uf0c1\n\n\nFor this example, this server will be referred to by the IP address \n10.0.0.7\n.\nThe \nMain EDEX\n server will be referred to by the IP address \n10.0.0.9\n.\n\n\n1. Install\n\uf0c1\n\n\nwget https://downloads.unidata.ucar.edu/awips2/current/linux/awips_install.sh\nchmod 755 awips_install.sh\nsudo ./awips_install.sh --ingest\n\n\n\n2. EDEX Config\n\uf0c1\n\n\nvi /awips2/edex/bin/setup.env\n\n\nHere you should redefine \nDB_ADDR\n and \nPYPIES_SERVER\n to point to the \nMain\n or \nDatabase/Request\n server (10.0.0.9) and the \nEXT_ADDR\n to point to the current \nIngest\n server (10.0.0.7)\n\n\nexport EXT_ADDR=10.0.0.7\n\n# postgres connection\nexport DB_ADDR=10.0.0.9\nexport DB_PORT=5432\n\n# pypies hdf5 connection\nexport PYPIES_SERVER=http://10.0.0.9:9582\n\n# qpid connection\nexport BROKER_ADDR=${EXT_ADDR}\n\n\n\nNotice that \nEXT_ADDR\n and \nBROKER_ADDR\n (qpid) should remain defined as the \nlocalhost\n IP address (10.0.0.7)\n\n\n3. Modify the edexServiceList\n\uf0c1\n\n\nMost likely if you are running a distributed EDEX setup, you are only processing a subset of data. You can change your edexServiceList to only run the processes you need. You will need to update the \n/etc/init.d/edexServiceList\n file. For example replace the services with the associated right column based on the data you're processing:\n\n\nexport SERVICES=('')\n\n\n\n\n\n\n\n\n\nData Processing:\n\n\nedexServiceList\n\n\n\n\n\n\n\n\n\n\nradar\n\n\ningestRadar\n\n\n\n\n\n\nsatellite\n\n\ningestGoesR\n\n\n\n\n\n\nmodel\n\n\ningestGrids, ingestGrib\n\n\n\n\n\n\n\n\n4. Configure your LDM\n\uf0c1\n\n\nYou'll want to modify your pqact.conf file to store only the data you want processed. There are example files in \n/awips2/ldm/etc\n that you can copy over to the main pqact.conf file. For example if you are wanting to process goesr data only, you can do the following steps:\n\n\ncd /awips2/ldm/etc\nmv pqact.conf pqact.conf.orig\ncp pqact.goesr pqact.conf\n\n\n\nYou will also want to edit the \npqact.conf\n file on your \nMain EDEX\n and comment out any entries you're processing on this EDEX server. \n\n\n5. Start EDEX\n\uf0c1\n\n\nedex start\n\n\n\nThis will start LDM, Qpid and the specified EDEX Ingest JVMs (and not start PostgreSQL, httpd-pypies, or the EDEX Request JVM)\n\n\n4. Monitor Services\n\uf0c1\n\n\nWatch the edex JVM log with the command\n\n\nedex log\n\n\n\n\n\nConfirm that EDEX connects to PostgreSQL!\n\n\nWith the above \nedex log\n, ensure that the log progresses \npast this point\n:\n\n\nSpring-enabled Plugins:\n-----------------------\nacars-common, acars-common-dataaccess, acarssounding-common, activetable-common,\nactivetable-ingest, airep-common, airep-common-dataaccess, airmet-common, \natcf-common, atcf-ingest, aww-common...\n\nJAXB context for PersistencePathKeySet inited in: 5ms\nINFO 20:21:09,134 5584 [EDEXMain] Reflections: Reflections took 436 ms to scan 258 urls, producing 31 keys and 3637 values\nFound 499 db classes in 720 ms\n\n\n\nIf the log stops at the \nFound db classes...\n line, that means EDEX is not connecting to the \nremote PostgreSQL instance\n - double-check \nDB_ADDR\n in \n/awips2/edex/bin/setup.env\n\n\nYou can \nmanually check remote PostgreSQL connectivity\n on any EDEX Ingest server from the command line:\n\n\nsu - awips\npsql -U awips -h -p 5432 metadata\n\n\n\nWhere the default passwd is \nawips\n and is defined in files in \n/awips2/edex/conf/db/hibernateConfig/\n\n\n\n\n\n\nAdditional Notes\n\uf0c1\n\n\n\n\nBe mindful of what IP address and hostnames are used in \n/awips2/edex/bin/setup.env\n and \n/awips2/database/data/pg_hba.conf\n, and that they are resolvable from the command line. Consult or edit \n/etc/hosts\n as needed.\n\n\n\n\nYou can install multiple \nawips2-ingest\n servers, each decoding a different dataset or feed, all pointing to the same Database/Request server (\nDB_ADDR\n and \nPYPIES_SERVER\n in \n/awips2/edex/bin/setup.env\n):\n\n\n\n\n\n\nEvery EDEX Ingest IP address must be allowed in both \niptables\n and \npg_hba.conf\n as \nshown above\n.\n\n\n\n\n\n\nData processed on",
"title": "Distributed EDEX"
},
{
@@ -1447,7 +1447,7 @@
},
{
"location": "/edex/distributed-computing/#1-install",
- "text": "groupadd fxalpha && useradd -G fxalpha awips\nmkdir /awips2\nwget -O /etc/yum.repos.d/awips2.repo https://downloads.unidata.ucar.edu/awips2/current/linux/awips2.repo\nyum clean all\nyum groupinstall awips2-database",
+ "text": "wget https://downloads.unidata.ucar.edu/awips2/current/linux/awips_install.sh\nchmod 755 awips_install.sh\nsudo ./awips_install.sh --database",
"title": "1. Install"
},
{
@@ -1457,7 +1457,7 @@
},
{
"location": "/edex/distributed-computing/#3-database-config",
- "text": "In the file /awips2/database/data/pg_hba.conf you define remote connections for all postgres tables with as /32 , after the block of IPv4 local connections: vi /awips2/database/data/pg_hba.conf\n\n# \"local\" is for Unix domain socket connections only\nlocal all all trust\nhostssl all all 10.0.0.7/32 cert clientcert=1\nhostssl all all 162.0.0.0/8 cert clientcert=1\nhostssl all all 127.0.0.1/32 cert clientcert=1\n# IPv6 local connections:\nhostssl all all ::1/128 cert clientcert=1\nhostnossl all all ::1/128 md5",
+ "text": "In the file /awips2/database/data/pg_hba.conf you define remote connections for all postgres tables with as /32 , after the block of IPv4 local connections and generic for hostnossl: vi /awips2/database/data/pg_hba.conf\n\n# \"local\" is for Unix domain socket connections only\nlocal all all trust\nhostssl all all 162.0.0.0/8 cert clientcert=1\nhostssl all all 127.0.0.1/32 cert clientcert=1\n\nhostssl all all 10.0.0.7/32 cert clientcert=1\nhostnossl postgres all 10.0.0.0/24 md5\nhostnossl fxatext all 10.0.0.0/24 md5\nhostnossl metadata all 10.0.0.0/24 md5\n\n# IPv6 local connections:\nhostssl all all ::1/128 cert clientcert=1\nhostnossl all all ::1/128 md5",
"title": "3. Database Config"
},
{
@@ -1471,24 +1471,34 @@
"title": "5. Monitor Services"
},
{
- "location": "/edex/distributed-computing/#ingestdecode-server",
- "text": "For this example, this server will be referred to by the IP address 10.0.0.7 .",
- "title": "Ingest/Decode Server"
+ "location": "/edex/distributed-computing/#ancillary-edex-server-ingestdecode-edex-server",
+ "text": "For this example, this server will be referred to by the IP address 10.0.0.7 .\nThe Main EDEX server will be referred to by the IP address 10.0.0.9 .",
+ "title": "Ancillary EDEX Server (Ingest/Decode EDEX Server)"
},
{
"location": "/edex/distributed-computing/#1-install_1",
- "text": "groupadd fxalpha && useradd -G fxalpha awips\nwget -O /etc/yum.repos.d/awips2.repo https://downloads.unidata.ucar.edu/awips2/current/linux/awips2.repo\nyum clean all\nyum groupinstall awips2-ingest",
+ "text": "wget https://downloads.unidata.ucar.edu/awips2/current/linux/awips_install.sh\nchmod 755 awips_install.sh\nsudo ./awips_install.sh --ingest",
"title": "1. Install"
},
{
"location": "/edex/distributed-computing/#2-edex-config",
- "text": "vi /awips2/edex/bin/setup.env Here you should redefine DB_ADDR and PYPIES_SERVER to point to the Database/Request server (10.0.0.9) export EDEX_SERVER=10.0.0.7\n\n# postgres connection\nexport DB_ADDR=10.0.0.9\nexport DB_PORT=5432\n\n# pypies hdf5 connection\nexport PYPIES_SERVER=http://10.0.0.9:9582\n\n# qpid connection\nexport BROKER_ADDR=${EDEX_SERVER} Notice that EDEX_SERVER and BROKER_ADDR (qpid) should remain defined as the localhost IP address (10.0.0.7)",
+ "text": "vi /awips2/edex/bin/setup.env Here you should redefine DB_ADDR and PYPIES_SERVER to point to the Main or Database/Request server (10.0.0.9) and the EXT_ADDR to point to the current Ingest server (10.0.0.7) export EXT_ADDR=10.0.0.7\n\n# postgres connection\nexport DB_ADDR=10.0.0.9\nexport DB_PORT=5432\n\n# pypies hdf5 connection\nexport PYPIES_SERVER=http://10.0.0.9:9582\n\n# qpid connection\nexport BROKER_ADDR=${EXT_ADDR} Notice that EXT_ADDR and BROKER_ADDR (qpid) should remain defined as the localhost IP address (10.0.0.7)",
"title": "2. EDEX Config"
},
{
- "location": "/edex/distributed-computing/#3-start-edex",
- "text": "edex start ingest This will start Qpid and the EDEX Ingest and IngestGrib JVMs (and not start PostgreSQL, httpd-pypies, or the EDEX Request JVM)",
- "title": "3. Start EDEX"
+ "location": "/edex/distributed-computing/#3-modify-the-edexservicelist",
+ "text": "Most likely if you are running a distributed EDEX setup, you are only processing a subset of data. You can change your edexServiceList to only run the processes you need. You will need to update the /etc/init.d/edexServiceList file. For example replace the services with the associated right column based on the data you're processing: export SERVICES=('') Data Processing: edexServiceList radar ingestRadar satellite ingestGoesR model ingestGrids, ingestGrib",
+ "title": "3. Modify the edexServiceList"
+ },
+ {
+ "location": "/edex/distributed-computing/#4-configure-your-ldm",
+ "text": "You'll want to modify your pqact.conf file to store only the data you want processed. There are example files in /awips2/ldm/etc that you can copy over to the main pqact.conf file. For example if you are wanting to process goesr data only, you can do the following steps: cd /awips2/ldm/etc\nmv pqact.conf pqact.conf.orig\ncp pqact.goesr pqact.conf You will also want to edit the pqact.conf file on your Main EDEX and comment out any entries you're processing on this EDEX server.",
+ "title": "4. Configure your LDM"
+ },
+ {
+ "location": "/edex/distributed-computing/#5-start-edex",
+ "text": "edex start This will start LDM, Qpid and the specified EDEX Ingest JVMs (and not start PostgreSQL, httpd-pypies, or the EDEX Request JVM)",
+ "title": "5. Start EDEX"
},
{
"location": "/edex/distributed-computing/#4-monitor-services",
@@ -1497,7 +1507,7 @@
},
{
"location": "/edex/distributed-computing/#additional-notes",
- "text": "Be mindful of what IP address and hostnames are used in /awips2/edex/bin/setup.env and /awips2/database/data/pg_hba.conf , and that they are resolvable from the command line. Consult or edit /etc/hosts as needed. You can install multiple awips2-ingest servers, each decoding a different dataset or feed, all pointing to the same Database/Request server ( DB_ADDR and PYPIES_SERVER in /awips2/edex/bin/setup.env ): Every EDEX Ingest IP address must be allowed in both iptables and pg_hba.conf as shown above .",
+ "text": "Be mindful of what IP address and hostnames are used in /awips2/edex/bin/setup.env and /awips2/database/data/pg_hba.conf , and that they are resolvable from the command line. Consult or edit /etc/hosts as needed. You can install multiple awips2-ingest servers, each decoding a different dataset or feed, all pointing to the same Database/Request server ( DB_ADDR and PYPIES_SERVER in /awips2/edex/bin/setup.env ): Every EDEX Ingest IP address must be allowed in both iptables and pg_hba.conf as shown above . Data processed on",
"title": "Additional Notes"
},
{
diff --git a/sitemap.xml b/sitemap.xml
index a4738cf0c2..d7d18ee261 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -2,192 +2,192 @@
http://unidata.github.io/awips2/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/install/install-cave/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/install/install-edex/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/install/common-problems/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/d2d-perspective/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/maps-views-projections/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/bundles-and-procedures/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/localization-perspective/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/nsharp/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/warngen/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/goes-16-17-satellite/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/d2d-gis-shapefiles/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/d2d-gridded-models/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/d2d-tools/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/d2d-radar-tools/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/d2d-edit-menus/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/cave-localization/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/import-export/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/install/start-edex/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/edex/settings/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/edex/distributed-computing/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/edex/edex-ingest-docker-container/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/edex/ldm/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/edex/data-distribution-files/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/edex/new-grid/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/edex/data-purge/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/edex/edex-users/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/edex/data-plugins/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/edex/case-studies/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/python/overview/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/dev/awips-development-environment/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/appendix/appendix-grid-parameters/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/python/maps-database/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/appendix/appendix-acronyms/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/appendix/appendix-wsr88d/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/cave/cave-keyboard-shortcuts/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/raytheon/cave_d2d/
- 2021-08-26
+ 2021-09-17
daily
http://unidata.github.io/awips2/raytheon/smm/
- 2021-08-26
+ 2021-09-17
daily
\ No newline at end of file