python-awips/examples/notebooks/GOES_CIRA_Product_Writer.ipynb

673 lines
23 KiB
Text
Raw Permalink Normal View History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a name=\"top\"></a>\n",
"<div style=\"width:1000 px\">\n",
"\n",
"<div style=\"float:right; width:98 px; height:98px;\">\n",
"<img src=\"https://docs.unidata.ucar.edu/images/logos/unidata_logo_vertical_150x150.png\" alt=\"Unidata Logo\" style=\"height: 98px;\">\n",
"</div>\n",
"\n",
"# GOES CIRA Product Writer\n",
"**Python-AWIPS Tutorial Notebook**\n",
"\n",
"<div style=\"clear:both\"></div>\n",
"</div>\n",
"\n",
"---\n",
"\n",
"<div style=\"float:right; width:250 px\"><img src=\"../images/GOES_CIRA_preview.png\" alt=\"GOES East Geocolor composite image\" style=\"height: 300px;\"></div>\n",
"\n",
"\n",
"# Objectives\n",
"\n",
"* Use python-awips to connect to an EDEX server\n",
"* Define and filter the data request specifically for new [CIRA GOES16 data products](#Additional-Documentation)\n",
"* Resize the products to their native resolution\n",
"* Write the individual bands (channels) locally\n",
"* Combine and write the RGB product locally\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"toc": true
},
"source": [
"<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
"<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Imports\" data-toc-modified-id=\"Imports-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Imports</a></span></li><li><span><a href=\"#Initial-Setup\" data-toc-modified-id=\"Initial-Setup-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Initial Setup</a></span><ul class=\"toc-item\"><li><span><a href=\"#EDEX-Connection\" data-toc-modified-id=\"EDEX-Connection-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>EDEX Connection</a></span></li><li><span><a href=\"#Parameter-Definition\" data-toc-modified-id=\"Parameter-Definition-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Parameter Definition</a></span></li></ul></li><li><span><a href=\"#Function:-set_size()\" data-toc-modified-id=\"Function:-set_size()-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Function: set_size()</a></span></li><li><span><a href=\"#Function:-write_img()\" data-toc-modified-id=\"Function:-write_img()-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Function: write_img()</a></span></li><li><span><a href=\"#Get-the-Data-and-Write-it-Out!\" data-toc-modified-id=\"Get-the-Data-and-Write-it-Out!-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Get the Data and Write it Out!</a></span><ul class=\"toc-item\"><li><span><a href=\"#Filter-the-Data\" data-toc-modified-id=\"Filter-the-Data-5.1\"><span class=\"toc-item-num\">5.1&nbsp;&nbsp;</span>Filter the Data</a></span></li><li><span><a href=\"#Define-Output-Location\" data-toc-modified-id=\"Define-Output-Location-5.2\"><span class=\"toc-item-num\">5.2&nbsp;&nbsp;</span>Define Output Location</a></span></li><li><span><a href=\"#Write-Out-GOES-Images\" data-toc-modified-id=\"Write-Out-GOES-Images-5.3\"><span class=\"toc-item-num\">5.3&nbsp;&nbsp;</span>Write Out GOES Images</a></span></li></ul></li><li><span><a href=\"#See-Also\" data-toc-modified-id=\"See-Also-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>See Also</a></span><ul class=\"toc-item\"><li><span><a href=\"#Related-Notebooks\" data-toc-modified-id=\"Related-Notebooks-6.1\"><span class=\"toc-item-num\">6.1&nbsp;&nbsp;</span>Related Notebooks</a></span></li><li><span><a href=\"#Additional-Documentation\" data-toc-modified-id=\"Additional-Documentation-6.2\"><span class=\"toc-item-num\">6.2&nbsp;&nbsp;</span>Additional Documentation</a></span></li></ul></li></ul></div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Imports\n",
"\n",
"The imports below are used throughout the notebook. Note the first import is coming directly from python-awips and allows us to connect to an EDEX server. The subsequent imports are for data manipulation and visualization."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"from awips.dataaccess import DataAccessLayer\n",
"import cartopy.crs as ccrs\n",
"import cartopy.feature as cfeat\n",
"import matplotlib.pyplot as plt\n",
"from datetime import datetime\n",
"import numpy as np\n",
"import os"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"#top\">Top</a>\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Initial Setup\n",
"\n",
"### EDEX Connection\n",
"\n",
"First we establish a connection to Unidata's public EDEX server. With that connection made, we can create a [new data request object](http://unidata.github.io/python-awips/api/IDataRequest.html) and set the data type to ***satellite***."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"# Create an EDEX data request\n",
"DataAccessLayer.changeEDEXHost(\"edex-cloud.unidata.ucar.edu\")\n",
"request = DataAccessLayer.newDataRequest()\n",
"request.setDatatype(\"satellite\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Parameter Definition\n",
"\n",
"After establishing the python-awips specific objects, we create a few other parameters that will be used for the data query based off of known values: projection, and extent."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"# Create a projection for ECONUS and WCONUS\n",
"# Set up the projection using known parameters (from the netcdf of GOES products)\n",
"globe = ccrs.Globe(semimajor_axis=6378137.0, semiminor_axis=6356752.5, ellipse=None)\n",
"sat_h = 35785830.0\n",
"proj = ccrs.Geostationary(globe=globe, central_longitude=-75.0, satellite_height=sat_h, sweep_axis='x')\n",
"\n",
"# Define the extents for ECONUS and WCONUS in goes native coords\n",
"# (originally taken from netcdf GOES data)\n",
"extent = (-3626751., 1382263.5, 1583666.1, 4588674.)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"#top\">Top</a>\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Function: set_size()\n",
"\n",
"Here we're defining a function that will allow us to pass in the dimensions of the output file we desire in pixels. Default Python methods require the size to be set in inches, which is confusing in our case, since we know what the size of GOES images are in pixels. Also, default Python functions add a padding when creating figures, and we don't want that. \n",
"\n",
"This function allows the exact final image to be specified based in pixels, with no padding or buffers."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"def set_size(w,h, plt):\n",
" \"\"\" w, h: width, height in pixels \"\"\"\n",
" \n",
" # Convert from pixels to inches\n",
" DPI = plt.figure().get_dpi()\n",
" w = w/float(DPI)\n",
" h = h/float(DPI)\n",
" \n",
" # Get the axes\n",
" ax=plt.gca()\n",
"\n",
" # Remove the padding\n",
" l = ax.figure.subplotpars.left\n",
" r = ax.figure.subplotpars.right\n",
" t = ax.figure.subplotpars.top\n",
" b = ax.figure.subplotpars.bottom\n",
" figw = float(w)/(r-l)\n",
" figh = float(h)/(t-b)\n",
" \n",
" # Set the final size\n",
" ax.figure.set_size_inches(figw, figh)\n",
" \n",
" # Return the DPI, this is used when in the \n",
" # write_image() function\n",
" return DPI"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"#top\">Top</a>\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Function: write_img()\n",
"\n",
"Next, we're defining another function which takes the image data, file name, projection, extent, reference time, and whether or not to print out a footnote.\n",
"\n",
"This method specifies the size of the output image and creates a plot object to draw all our data into. Then it draws the GOES data, coastlines, state boundaries, and lat/lon lines onto the image. Additionally, if we want, it writes out a short footnote describing what product we're looking at. Finally, it writes out the figure to disk. \n",
"\n",
"By default we're specifying the output dimensions to be 5000x4000 pixels, because that is the native GOES image size, but feel free to modify these values if you wish to print out an image of another size (you may want to keep the w:h ratio the same though)."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"def write_img(data, name, proj, extent, reftime, footnote):\n",
" \n",
" # Specify the desired size, in pixels\n",
" px_width = 5000.0\n",
" px_height = 3000.0\n",
"\n",
" # Create the plot with proper projection, and set the figure size\n",
" fig = plt.figure()\n",
" DPI = set_size(px_width, px_height, plt)\n",
" ax = plt.axes(projection=proj)\n",
" \n",
" # Draw GOES data\n",
" ax.imshow(data, cmap='gray', transform=proj, extent=extent)\n",
"\n",
" # Add Coastlines and States\n",
" ax.coastlines(resolution='50m', color='magenta', linewidth=1.0)\n",
" ax.add_feature(cfeat.STATES, edgecolor='magenta', linewidth=1.0)\n",
" ax.gridlines(color='cyan', linewidth=2.0, xlocs=np.arange(-180, 180, 10), linestyle=(0,(5,10)))\n",
"\n",
" # Create and draw the footnote if needed\n",
" if footnote: \n",
" footnoteStr = ' CIRA-'+name[7:-4]+'-'+str(reftime)\n",
" plt.annotate(str(footnoteStr), (0,0), (0, 0), xycoords='axes fraction', textcoords='offset points', va='top')\n",
" \n",
" # Write out the figure\n",
" plt.savefig(name, dpi=DPI, bbox_inches='tight', pad_inches=0)\n",
" #plt.show()\n",
" plt.close()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"#top\">Top</a>\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Get the Data and Write it Out!\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Filter the Data\n",
"\n",
"Define exactly what data we want to be printing out. This notebook is designed to loop through and print out multiple images, so here we can pick which images we're wanting to print out. We're specifying ***ECONUS*** (for East CONUS), ***CLDSNOW***, ***DBRDUST***, and ***GEOCOLR*** (for the new CIRA products) and the ***three channels*** for the RBG composites."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert-info\">\n",
"<b>Tip:</b> \n",
"More information could be gathered by looking at all the available location names (sectors), identifiers (entities), and parameters (channels). To see those run the following lines of code after the dataType has been set to satellite on the request object:\n",
"</div>\n",
"\n",
"```\n",
"## Print Available Location Names\n",
"print((DataAccessLayer.getAvailableLocationNames(request))\n",
" \n",
"## Print Available Identifiers and Values\n",
"ids = DataAccessLayer.getOptionalIdentifiers(request)\n",
"print(ids)\n",
"for id in ids:\n",
" print(id, DataAccessLayer.getIdentifierValues(request, id))\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"# Define Location names\n",
"sectors = [\"ECONUS\"]\n",
"\n",
"# Define creatingEntity Identifiers\n",
"entities = [\"CLDSNOW\", \"DBRDUST\", \"GEOCOLR\"]\n",
"\n",
"# Define parameters\n",
"ch1 = \"CH-01-0.47um\"\n",
"ch2 = \"CH-02-0.64um\"\n",
"ch3 = \"CH-03-0.87um\"\n",
"channels = [ch1, ch2, ch3]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Define Output Location\n",
"\n",
"Here we define a folder for where the satellite images will be written to. The default directory is a new folder called 'output' that lives whereever this notebook lives."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert-info\">\n",
"<b>Tip:</b> \n",
"If you specify the fully qualified path, it will no longer depend on where this notebook is located. For example (for a Mac):\n",
"</div>\n",
"\n",
"```\n",
"outputDir = '/Users/awips/test_dir/output/'\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Output directory exists!\n"
]
}
],
"source": [
"# Define name of the desired end directory\n",
"outputDir = 'output/'\n",
"\n",
"# Check to see if this folder exists\n",
"if not os.path.exists(outputDir):\n",
" # If not, create the directory\n",
" print('Creating new output directory: ',outputDir)\n",
" os.makedirs(outputDir)\n",
"else:\n",
" # If so, let the user know\n",
" print('Output directory exists!')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Write Out GOES Images\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"selected time: [<DataTime instance: 2025-02-11 23:06:17 >]\n",
"writing output/CLDSNOW-ECONUS-CH-01-0.47um.png\n",
"writing output/CLDSNOW-ECONUS-CH-02-0.64um.png\n",
"writing output/CLDSNOW-ECONUS-CH-03-0.87um.png\n",
"writing output/CLDSNOW-ECONUS-RGB.png\n",
"selected time: [<DataTime instance: 2025-02-11 23:21:17 >]\n",
"writing output/DBRDUST-ECONUS-CH-01-0.47um.png\n",
"writing output/DBRDUST-ECONUS-CH-02-0.64um.png\n",
"writing output/DBRDUST-ECONUS-CH-03-0.87um.png\n",
"writing output/DBRDUST-ECONUS-RGB.png\n",
"selected time: [<DataTime instance: 2025-02-11 23:41:17 >]\n",
"writing output/GEOCOLR-ECONUS-CH-01-0.47um.png\n",
"writing output/GEOCOLR-ECONUS-CH-02-0.64um.png\n",
"writing output/GEOCOLR-ECONUS-CH-03-0.87um.png\n",
"writing output/GEOCOLR-ECONUS-RGB.png\n",
"Done!\n"
]
},
{
"data": {
"text/plain": [
"<Figure size 640x480 with 0 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<Figure size 640x480 with 0 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<Figure size 640x480 with 0 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<Figure size 640x480 with 0 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<Figure size 640x480 with 0 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<Figure size 640x480 with 0 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<Figure size 640x480 with 0 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<Figure size 640x480 with 0 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<Figure size 640x480 with 0 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<Figure size 640x480 with 0 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<Figure size 640x480 with 0 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"<Figure size 640x480 with 0 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# First loop through the sectors (location names)\n",
"for sector in sectors:\n",
" \n",
" # Set the location on the request\n",
" request.setLocationNames(sector)\n",
" \n",
" # Next loop through the Products (entities)\n",
" for entity in entities:\n",
" # Reset the time and channel variables since we're on a new product\n",
" time = None\n",
" R = None\n",
" G = None\n",
" B = None\n",
" \n",
" # Set the product\n",
" request.addIdentifier(\"creatingEntity\", entity)\n",
" \n",
" # Cycle through the channels (parameters)\n",
" for channel in channels:\n",
" request.setParameters(channel)\n",
" \n",
" # Set the time for this product if it hasn't been set\n",
" # If it has been set, then we proceed with that value\n",
" # so that all bands in for the one product are pulled\n",
" # from the same time\n",
" if(time is None):\n",
" times = DataAccessLayer.getAvailableTimes(request)\n",
" time = [times[-1]]\n",
" print(\"selected time:\", time)\n",
"\n",
" # Request the data from EDEX\n",
" response = DataAccessLayer.getGridData(request, time)\n",
" # Grab the actual data from the response\n",
" grid = response[0]\n",
" # Get the raw data from the response\n",
" data = grid.getRawData()\n",
" reftime = grid.getDataTime().getRefTime()\n",
" \n",
" # Set the R,G,B channel\n",
" if(channel == ch1):\n",
" B = data\n",
" elif (channel == ch2):\n",
" R = data\n",
" elif (channel == ch3):\n",
" G = data\n",
" \n",
" # Create the single channel name\n",
" name = outputDir+entity+'-'+sector+'-'+channel+'.png'\n",
"\n",
" # Write out the single channel\n",
" print('writing',name)\n",
2021-05-28 01:16:33 -06:00
" write_img(data, name, proj, extent, reftime, False)\n",
" \n",
" # --- End of channel loop \n",
" \n",
" # Create the RGB product\n",
" # Apply range limits for each channel. RGB values must be between 0 and 1\n",
" R = np.clip(R, 0, 1)\n",
" G = np.clip(G, 0, 1)\n",
" B = np.clip(B, 0, 1)\n",
" RGB = np.dstack([R, G, B])\n",
" \n",
" # Create RGB name\n",
" rgbName = outputDir+entity+'-'+sector+'-RGB.png'\n",
" # Write out the RGB image\n",
" print('writing', rgbName)\n",
" write_img(RGB, rgbName, proj, extent, time, False)\n",
" \n",
" # --- End of entity loop\n",
"#--- End of sector loop\n",
" \n",
"print('Done!')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"#top\">Top</a>\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## See Also\n",
"\n",
"### Related Notebooks\n",
"\n",
"- [Satellite Imagery](http://unidata.github.io/python-awips/examples/generated/Satellite_Imagery.html)\n",
"\n",
"### Additional Documentation\n",
"\n",
"**CIRA Quick Guides**\n",
"\n",
"- [DEBRA-Dust](https://rammb.cira.colostate.edu/training/visit/quick_guides/QuickGuide_DEBRA-Dust_20210217.pdf)\n",
"- [Cloud-Snow](https://rammb.cira.colostate.edu/training/visit/quick_guides/GOES_Cloud_Snow_Discriminator_Quick_Guide_20190814.pdf)\n",
"- [GEOCOLOR](https://rammb.cira.colostate.edu/training/visit/quick_guides/QuickGuide_CIRA_Geocolor_20171019.pdf)\n",
"\n",
"\n",
"**python-awips**\n",
"\n",
"- [DataAccessLayer.changeEDEXHost()](http://unidata.github.io/python-awips/api/DataAccessLayer.html#awips.dataaccess.DataAccessLayer.changeEDEXHost)\n",
"- [DataAccessLayer.newDataRequest()](http://unidata.github.io/python-awips/api/DataAccessLayer.html#awips.dataaccess.DataAccessLayer.newDataRequest)\n",
"- [DataAccessLayer.getAvailableLocationNames()](http://unidata.github.io/python-awips/api/DataAccessLayer.html#awips.dataaccess.DataAccessLayer.getAvailableLocationNames)\n",
"- [DataAccessLayer.getOptionalIdentifiers()](http://unidata.github.io/python-awips/api/DataAccessLayer.html#awips.dataaccess.DataAccessLayer.getOptionalIdentifiers)\n",
"- [DataAccessLayer.getIdentifierValues()](http://unidata.github.io/python-awips/api/DataAccessLayer.html#awips.dataaccess.DataAccessLayer.getIdentifierValues)\n",
"- [DataAccessLayer.getAvailableTimes()](http://unidata.github.io/python-awips/api/DataAccessLayer.html#awips.dataaccess.DataAccessLayer.getAvailableTimes)\n",
"- [IDataRequest](http://unidata.github.io/python-awips/api/IDataRequest.html)\n",
"\n",
"**matplotlib**\n",
"\n",
"- [matplotlib.pyplot()](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.html)\n",
"- [matplotlib.pyplot.axes()](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.axes.html)\n",
"- [matplotlib.pyplot.figure()](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.figure.html)\n",
"\n",
"**numpy**\n",
"\n",
"- [numpy.clip()](https://numpy.org/doc/stable/reference/generated/numpy.clip.html)\n",
"- [numpy.dstack()](https://numpy.org/doc/stable/reference/generated/numpy.dstack.html)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"#top\">Top</a>\n",
"\n",
"---"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.1"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": true,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": true,
"toc_position": {
"height": "calc(100% - 180px)",
"left": "10px",
"top": "150px",
"width": "230px"
},
"toc_section_display": true,
"toc_window_display": true
}
},
"nbformat": 4,
"nbformat_minor": 4
}