Arguments in the Standard File
   Naming Conventions
   Sample Analysis
        Statistics By Time Range
        Arguments to SampleAnalysis methods
       Checking for Edit Areas
        Global Thresholds and Variables
        Scalar Analysis methods
        Vector Analysis methods
        DiscreteAnalysis methods
        Weather Analysis methods
           Algorithm for rankedWx and dominantWx
   Text Rules Libraries for Narrative Products
        Setting Day and Night Times
        Analysis-Driven Phrases
        Arguments to Narrative methods
        Checking for Edit Areas
        Global Thresholds and Variables
        Narrative Phrases
           Scalar Weather Element Phrases
           Vector-Related Phrases (Wind and WindGust)
           Sea Breeze Identification
           Weather Phrases
           Weather Subkey Attributes "Primary" and "Mention"
           Customizing Weather Phrases
           Pop/Wx Consistency
           Combined Element Phrases
           Marine Phrases
         Algorithm for WaveHeight and WindWaveHgt Reporting
           Fire Weather Phrases
           Discrete Phrases (Headlines)
        Narrative Strategies
           Non-Linear Thresholds
           DataValue Extraction from the Grids
            Unit Conversion
            Standard and Non-Standard Rounding
            Range  Adjustment
           Null Values
           Local Effects
            Local Effects and Compound Phrases
            Consolidation Rules for Local Effects
            Local Effect Examples
              Null Values and Local Effects
             "Except" vs. "Otherwise" wording
             More Local Effect Area Features
             Local Effects for Snow Accumulation and Total Snow Accumulation -- Using a Method for Checking the Local Effect Threshold
             Local Effects for the Combined SkyPopWx Phrase
           Period Combining
           Phrase Consolidation
           Phrase Ordering
           "Until" Phrasing
           Visibility
           Weather Phrasing
   Appendix A:  Local Effects Design

Arguments in the Standard Product File

Naming conventions

Sample Analysis

Statistics By Time Range

       "analysisList" : [
                                   ("MaxT", self.avg),
                                    ]

The result of the analysis will be a simple average value for the entire time range being examined.   Using the optional "divide" argument, the analysisList might look like this:

       "analysisList" : [
                                   ("MaxT", self.avg, [6]),
                                    ]

In this instance, the time range will be divided into 6-hour sub-ranges and an average value for MaxT will be reported for each sub-range. This information will be reported as a list of tuples. For example, a 12-hour time range would produce:

                [ (avg1, subRange1), (avg2, subRange2)]

The time range is divided from the endtime backward so that "odd" sub-ranges occur at the beginning of the time range.  For example, suppose we are dividing an 8 hour time range into 6-hour sub-ranges.  This will result in a 2-hour sub-range followed by a 6-hour sub-range.

The Text Rules methods are built to handle varying types of Sample Analyis return values.

If the time-divide argument is zero,  the statistics will be reported "byTimeRange" returning a list of (statistics, subRange) tuples,  one for each grid overlapping timeRange being examined.

In summary, the following table shows analysisList methods and their corresponding results:
 
 

NOTE that if there is no data for ANY ONE  of the subRanges,  None is returned for the entire time range.

Arguments to Sample Analysis Methods

Checking for Edit Areas

GLOBAL THRESHOLDS AND VARIABLES

There are four categories of analysis methods: absolute, cumulative, standard deviation (stdDev) and moderated (percentage).  The absolute and cumulative categories do not filter the sampled data. All statistics are based on the entire  sampled data set.  The standard deviation (stdDev) and moderated methods do filter the sampled data set, however.  The stdDev methods ignore values that lie outside a configurable number of standard deviations from the mean.  Low and high thresholds are defined independently. The moderated methods work exactly the same way as stdDev methods, except that they filter data based on a ranked percentage.  Values are ranked, low to high, and outliers are excluded from the data set (e.g., the upper 10%).

Most analysis methods screen grids for temporal coverage of the given time range before including their data values in the analysis.    The exceptions to this are the methods that work with accumulative elements. The following table describes the temporal coverage thresholds you can override for this screening process.  To override a threshold, simply copy the method from the SampleAnalysis utility into your _Overrides file and change desired values.  Most methods weight grids temporally as well. Each grid will contribute proportional to its time duration for the sampled period.

Here is how the standard deviation SampleAnalysis methods work:
1) compute the standard deviation of all grid values
2) the stdDev_dict contains a threshold, i.e., number of sigmas for each the min and the max
3) the routine calculates the min and max from the std deviation and the sigmas
4) in some cases, the result may fall outside of the actual data range.  In this case, the returned values will be the min or max of the actual data points.

Here is how the moderated SampleAnalysis methods work:
1) compute the "histogram" of the data values
2) the moderated_dict gives a percentage for min, and a percentage for max
3) we return values that are % percent from the min and % percent from the max, not numerically, but from a sorted list.  Thus if the histogram is
4,4,5,6,7,8,9,10,15,19, and your percent is 20% for the min and 10% for the max, you will pick out the 20% of the number and the 90% of the number in the
sequence.  Since there are exactly 10 samples in my exmaple, the 20% point will be 5, and the 90% point will be 15.  It isn't numerical.

Here is some further explanation (from Bill Schneider) on the moderated methods:

The moderated methods look at all the points in the grid, then they throw away highest (for max) or lowest (for min) percentage you have specified in the moderated dictionary. Then it uses the max or min of the remaining gridpoints.

So for example if you use moderatedMaximum for temperature and have defined the moderated percentages fore temperature as (10,15) in the dictionary, the method will take the grid points that have the highest values and throw away the top 15 percent of them. The remaining maximum value will be reported. For moderatedMinimum, the method will take the grid points that have the lowest values and throw away 15 % of the lowest values before reporting the minimum.
 

SCALAR WEATHER ELEMENTS

VECTOR WEATHER ELEMENTS

Note: all vector directions are returned as numeric.   You can set the vectorDirection_algorithm to be "Average" or "MostFrequent".  It will then apply to all vector analysis methods. 

DISCRETE ELEMENTS

WX  ELEMENT  

Algorithm for rankedWx and dominantWx

• Throw out any grids that do not cover the temporalCoverage_percentage of the given time range.
• For each weather type in the grids, determine:
• an aggregate subkey
• coverage -- aggregate coverage algorithm can be chosen by overriding "aggregateCov_algorithm" (SampleAnalysis). There are 2 choices:
• "getAggregateCov" chooses the highest ranked coverage (default)
  
• "getWeightedAggregateCov" favors stronger coverage terms.
  
• intensity -- highest ranking OR if ranks are close, dominant
• visibility -- lowest visibility
• attributes -- aggregate of all occurrences
• a rank i.e. the percentage of areal coverage over the time period.  This raw rank is then weighted by the coverage term.
  
• Throw out any weather type that does not meet the wxkey_coverage_percentage except:
• If a weather subkey is designated "Primary" it will be the aggregate subkey and will automatically be the highest ranked.
• If a weather subkey is designated as "Mention", it will be the aggregate subkey and will not be eliminated.
  
• Return the dominantKeys threshold number of weather subkeys sorted by rank, wxType, and coverage
  
• For rankedWx, the computed rank is returned with each subkey         
  

Text Rules Libraries for Simple Table Products

Text Rules Libraries for Smart Tabular Products

Text Rules Libraries for Narrative Products

Narrative Tree and Phrase Builder Infrastructure

The Text Rules and all of it's inherited libraries can be found in the Utilities window of the Define Text Products Dialog.   This section describes the threshold, variable and phrase methods in the TextRules libraries to support Narrative Products.  This information will help you customize your narrative product by looking up the phrases that are being generated in the "phraseList" of your product components.  For each weather element (or combinations of weather elements), we list Example Phrases, corresponding SampleAnalysis methods, and Thresholds and variables you can set to customize the phrase.
 

Setting DAY and NIGHT times

            def DAY(self):
                return 6
            def NIGHT(self):
                return 18
 

Analysis-Driven Phrases

If a threshold or variable appears in bold, then it MUST be set in order for the phrase to appear.

Arguments to Narrative Methods

Checking for Edit Areas

                                inlandWatersAreas = ["TampaBayWaters", "InlandWaters"]
                                if self.currentAreaContains(tree, inlandWatersAreas):
                                         # Return a value for the inland waters areas
                                else:
                                        # Return a value for the  other areas

GLOBAL THRESHOLDS AND VARIABLES

NARRATIVE PHRASES

SCALAR WEATHER ELEMENT PHRASES -- ScalarPhrases module

WIND AND WINDGUST PHRASES -- VectorRelatedPhrases module

Sea Breeze Identification

         NW winds 15 to 25 knots becoming onshore 10 to 15 knots.

To set up sea breeze indentification, you need to:

• Create the following named edit areas from the GFE GUI:
• OffShoreArea -- offshore area
• OnShoreArea -- onshore area
 
•  Override the product component definition to include the "intersectAreas" for Wind.   For example:

        return {
            "type": "component",
            "methodList": [
                          self.assemblePhrases,
                          self.wordWrap,
                          ],
            "analysisList": [
                          # Split time range in quarters for Wind and WindGust
                          ("Wind", self.vectorModeratedMinMax, [3]),
                          ("Wind", self.vectorMinMax, [12]),
                          ("WindGust", self.maximum, [3]),
                          ("WaveHeight", self.minMax, [6]),
                          ("WindWaveHgt", self.minMax, [6]),
                          # Split time range in half for Wx and Swell
                          ("Wx", self.rankedWx, [6]),
                          ("Swell", self.vectorMinMax, [6]),
                          ("Swell2", self.vectorMinMax, [6]),
                          ("Period", self.avg, [6]),
                          ("Period2", self.avg, [6]),
                        ],

             "phraseList":[
                           # WINDS
                           self.marine_wind_withGusts_phrase,
                           # Alternative:
                           #self.marine_wind_phrase,
                           #self.gust_phrase,
                           # WAVES
                           self.wave_withPeriods_phrase,
                           # Alternative:
                           #self.wave_phrase,
                           # Optional:
                           self.chop_phrase,
                           # WEATHER
                           self.weather_phrase,
                           # SWELLS AND PERIODS
                           self.swell_withPeriods_phrase,
                           # Alternative:
                           #self.swell_phrase,
                           #self.period_phrase,
                           ],

            "intersectAreas" :[
                  ("Wind", ["OffShoreArea", "OnShoreArea"]),
                  ],

            }
 

• Override the seaBreeze_thresholds  (found in ConfigVariables) appropriately for your areas to identify the offshore and onshore wind directions that indicate a sea breeze.  If the magnitude is constant throughout the period AND if the offshore winds are between offshoreDirection1 and offshoreDirection2 during the first part of the period AND the onshore winds are between onshoreDirection1 and onshoreDirection2 during the next part of the period, the becoming onshore wording is used. For example, if you set OffShoreArea for Hours 1-6 to 1020 and OnShoreArea for Hours 7-12 to 2420, you will get the desired wording.
  

WEATHER PHRASES -- WxPhrases module

Weather Subkey Attributes "Primary" and "Mention"

• "Primary":  If a subkey is marked as "Primary," it will appear as the dominant weather subkey in the weather wording.
• "Mention":  If a subkey is marked as "Mention," it will appear in the weather wording regardless of its temporal/areal coverage.  Otherwise, subkeys that do not meet the coverage requirements (as defined by the "wxkey_coverage_percent") are eliminated from the wording.  So, for example, the wxkey_coverage_percent for IP is 15% so that if we have just 1 hour of IP in a 12-hour period, it will not be mentioned in the wording.  However, if this subkey is given an attribute of "Mention," it will not be eliminated and will appear in the wording. This gives the forecaster the ability to supercede the coverage thresholds.

Customizing Weather Phrases

    def wxTypeDescriptors(self):
        # This is the list of coverages, wxTypes, intensities, attributes for which special
        # weather type wording is desired.  Wildcards (*) can be used to match any value.
        # If a weather subkey is not found in this list, default wording
        # will be used from the Weather Definition in the server.
        # The format of each tuple is:
        #    (coverage, wxType, intensity, attribute, weatherType descriptor)
        return [
                ("*", "RW", "--", "*", "sprinkles"),
                ("*", "SW", "--", "*", "flurries"),
            ]

In many cases, more than one of these methods will have to be overridden to accomplish the desired result.  We might have to "turn on" one kind of wording and "turn off" other default wording. In the above example, we have to "turn off" the default descriptor for "very light" using the wxIntensityDescriptors:

    def wxIntensityDescriptors(self):
        # This is the list of coverages, wxTypes, intensities, attribute for which special
        # weather intensity wording is desired.  Wildcards (*) can be used to match any value.
        # If a weather subkey is not found in this list, default wording
        # will be used from the Weather Definition in the server.
        # The format of each tuple is:
        #    (coverage, wxType, intensity, attribute, intensity descriptor)
        return [
                ("*", "RW", "--", "*", ""),
                ("*", "RW", "-", "*", ""),
                ("*", "R", "--", "*", "very light"),
                ("*", "R", "-", "*", "light"),
                ("*", "R", "+", "*", "heavy"),
                ("*", "RW", "+", "*", "heavy"),
                ("*", "SW", "--", "*", ""),
                ("*", "SW", "-", "*", ""),
                ("*", "SW", "+", "*", "heavy"),
                ("*", "S", "--", "*", "very light"),
                ("*", "S", "-", "*", "light"),
                ("*", "S", "+", "*", "heavy"),
                ("*", "ZR", "--", "*", "light"),
                ("*", "ZR", "+", "*", "heavy"),
                ("*", "L", "*", "*", ""),
                ("*", "F", "+", "*", "dense"),
            ]

Suppose we want to report "heavy snow" as "snow...heavy at times", we would include the following in our Overrides file:

    def wxTypeDescriptors(self):
         list = TextRules.TextRules.wxTypeDescriptors(self)
         list.append( ("*", "S", "+", "*", "snow...heavy at times") )
         return list

    def wxIntensityDescriptors(self):
         list = TextRules.TextRules.wxIntensityDescriptors(self)
         list.append( ("*", "S", "+", "")
         return list

As another example, suppose we want to describe thunderstorms with an attribute of "dry" as "dry thunderstorms."  Including the following in our Overrides file could accomplish this:

    def wxTypeDescriptors(self):
         list = TextRules.TextRules.wxTypeDescriptors(self)
         list.append( ("*", "T", "*", "Dry", "dry thunderstorms") )
         return list

    def wxAttributeDescriptors(self):
         list = TextRules.TextRules.wxAttributeDescriptors(self)
         list.append( ("*", "T", "*", "Dry", "") )
         return list

It is possible to specify a method instead of text string for the descriptor.  For example, the default wxTypeDescriptor for rain showers uses a method as follows:

    def wxTypeDescriptors(self):
        return [
                ("*", "SW", "--", "*", "flurries"),
                ("*", "RW", "*", "*", self.rainShowersDescriptor),
            ]

    def rainShowersDescriptor(self, tree, node, subkey):
        if subkey.intensity() == "--":
            return "sprinkles"
        if tree is None:
            return "showers"
        t = tree.stats.get(
              "T", node.getTimeRange(),
              node.getAreaLabel(), statLabel="minMax",
              mergeMethod="Min")
        if t is None:
            return "showers"
        if t < 40:
            return "rain showers"
        else:
            return "showers"

These methods are  used in Table products as well as Narratives.  NOTE:  IF you are working with a narrative product, you can include the "tree, node" arguments and use them to qualify your customizations.  For example, we could check the node time range and qualify fog as "morning fog" as in the example below:

    def wxTypeDescriptors(self, tree, node):
         list = TextRules.TextRules.wxTypeDescriptors(self)
         # Find out what time range we are working with
         timeRange = node.getTimeRange()
         # See if this time range is for the Morning
         # First, shift to Local time from GMT
         timeRange = self.shiftedTimeRange(timeRange)
         startHour = timeRange.startTime().hour()
         endHour = timeRange.endTime().hour()
         if startHour >= 6 and endHour <= 12:
                list.append( ("*", "F", "*", "*", "morning fog") )
         return list

For more information on customizing Weather Phrases, see the sections on Weather Phrasing  and Phrase Consolidation.

Pop/Wx Consistency

COMBINED ELEMENT PHRASES -- CombinedPhrases module

MARINE PHRASES -- MarinePhrases module

Algorithm for WaveHeight and/or WindWaveHgt Reporting

• If the edit area is in the inlandWatersAreas list, report from the  inlandWatersWave_element(default "WaveHeight".  If not available, report on second choice (default WindWaveHgt).  For this phrase, use the phrase_descriptor_dict  entry for "inland waters".
• If the edit area is not inland waters:
• If maximum Wind > waveHeight_wind_threshold  OR if maximum WindWaveHgt and maximum Swell > combinedSeas_threshold:
• report from seasWaveHeight_element(default WaveHeight).
• if "_withPeriod" report average of Period and Period2  embedded in phrase.
• For this phrase, use the phrase_descriptor entry for "seas".
• Otherwise, report seasWindWave_element(default WindWaveHgt).  For this phrase, use the phrase_descriptor_dictfor "waves".
• If no data available, return the noWaveHeight_phrase.

FIRE WEATHER PHRASES

Ridge versus Valley Winds

    def ridgeValleyAreas(self, tree, node):
        # List of edit area names for which we want
        # ridge/valley winds reported:
        #
        # 20-FOOT WINDS...
        #     VALLEYS/LWR SLOPES...
        #     RIDGES/UPR SLOPES....
        #
        # e.g.
        # return ["Area1"]
        return []

Next, make sure you have edit areas defined for "Ridges" and "Valleys" to intersect with the current area.  If you want to use different edit area names, you can include the following override (from the FirePhrases module) in your Overrides file:

    def valleyRidgeAreaNames(self, tree, node):
        # These are the areas for valleys and ridges, respectively,
        # to be intersected with the current edit area for
        # reporting valley winds and ridge winds, respectively.
        # NOTE: If you change these area names, you will also
        # need to change the names in the FirePeriod "intersectAreas"
        # section.
        return "Valleys", "Ridges"
 

 DISCRETE PHRASES (DiscretePhrases module)

NARRATIVE STRATEGIES

NON-LINEAR THRESHOLDS

     threshold = self.nlValue(threshold, value)

Any variable that is named with the suffix _nlValue can operate in this way returning EITHER a single value or a dictionary of values.  If a dictionary is returned, it is of the form:

      {
      'default': <default value>,
      (lowVal, highVal) :  <value>,
      }
where lowVal and highVal consitute a range for applying the given value.  The lowVal is inclusive while the highVal is exclusive.  For example:

    def scalar_difference_nlValue_dict(self, tree, node):
        # Scalar difference.  If the difference between scalar values
        # for 2 sub-periods is greater than this value,
        # the different values will be noted in the phrase.
        return {
            "WindGust": {
                        'default': 10,
                        (0, 30):   10,
                        (30, 50):  15,
                        (50, 200): 20,
                        },
             }

In this case, if the WindGust is greater or equal to 0 and less than 30, the scalar_difference will be 10; if WindGust is greater or equal to 30  and less that 50, the scalar_difference will be 15; etc.

You always have the additional option of basing the return value on the current edit area.  For example:

    def scalar_difference_nlValue_dict(self, tree, node):
        # Scalar difference.  If the difference between scalar values
        # for 2 sub-periods is greater than this value,
        # the different values will be noted in the phrase.
        return {
            "WindGust": self.windGust_scalar_nlValue_difference,
            }

     def windGust_scalar_nlValue_difference(self, tree, node):
         mountains = ["Area1", "Area2"]
         if self.currentAreaContains(tree, mountains):
            return 15
         else:
            return {
                        'default': 10,
                        (0, 30):   10,
                        (31, 50):  15,
                        (51, 200): 20,
                        }

The method applying the threshold must do the following before using it:

   threshold = self.scalar_difference_threshold(
                    tree, node, element, element)
   for each value to check:
        curThreshold = self.nlValue(threshold, value)
        # Use curThreshold for the given value
 

DATA VALUE EXTRACTION FROM THE GRIDS

• Sampling -- simple histogram of raw data
• Sample Analysis -- statistical methods specified in the product component definition (e.g. avg, mininum, maximum, vectorModeratedMinMax)
• Unit Conversion
• Standard and Non-Standard Rounding
• Range Adjustment

Unit Conversion

    def element_inUnits_dict(self, tree, node):
        # Dictionary of descriptors for various units

    def element_outUnits_dict(self, tree, node):
        # Dictionary of descriptors for various units

 Standard and Non-Standard rounding

Example of standard rounding increment (from ZFP_<site>_Overrides file, overriding increment_nlValue_dict in ConfigVariables module):

    def increment_nlValue_dict(self, tree, node):
        # Increment for rounding values
        # Units depend on the product
        dict = TextRules.TextRules.increment_nlValue_dict(self, tree, node)
        dict["Wind"] = 5
        return dict

Example of non-standard rounding method (found in ConfigVariables module):

    def rounding_method_dict(self, tree, node):
        # Special rounding methods
        #
        return {
            "Wind": self.marineRounding,
            }

    def marineRounding(self, value, mode, increment, maxFlag):
        # Rounding for marine winds
        mode = "Nearest"
        if maxFlag:
            if value > 30 and value < 34:
                mode = "RoundDown"
            elif value > 45 and value < 48:
                mode = "RoundDown"
            else:
                mode = "Nearest"
        return self.round(value, mode, increment)

Range Adjustment

The basic premise of range adjustment is that you have to commit to the ranges you choose (and rounding increments as well) from the beginning of processing.  In other words, we can't just slap on a range at the very end when we are building the phrase; ranges need to be applied from the moment the grids are analyzed so that phrase combining and period combining are comparing pre-adjusted values.  Then, if we do combine two sub-phrases or two periods, we have to combine their statistics and, then re-apply the range adjustments to the new values. The system is set up to recognize these cases when you set the range values listed below.

The following variables are used to adjust the range reported for the data value:

    def range_nlValue_dict(self, tree, node):
        # If the range of values less than this amount, return as a single value

    def minimum_range_nlValue_dict(self, tree, node):
        # This threshold is the "smallest" min/max difference allowed between values reported.
        # For example, if threshold is set to 5 for "MaxT", and the min value is 45
        # and the max value is 46, the range will be adjusted to at least a 5 degree
        # range e.g. 43-48.  These are the values that are then submitted for phrasing
        # such as:
        #   HIGHS IN THE MID 40S

    def maximum_range_nlValue_dict(self, tree, node):
        # Maximum range to be reported within a phrase
        #   e.g. 5 to 10 mph
        # Units depend on the product

    def range_bias_nlValue_dict(self, tree, node):
        # "Min", "Average", "Max"
        #  Should the range be taken from the "min" "average" or "max" value of the current range?
        return {'default': "Average"}

    def maximum_range_bias_nlValue_dict(self, tree, node):
        # "Min", "Average", "Max"
        #  Should the maximum_range be taken from the "min" "average" or "max" value of the current range?
        return {
                      "default": "Average",
                      "Wind": "Max"
                     }

     def minimum_range_bias_nlValue_dict(self, tree, node):
        # "Min", "Average", "Max"
        #  Should the minimum_range be taken from the "min" "average" or "max" value of the current range?
        return {
                      "default": "Average",
                      "Wind": "Max"
                     }

NOTE: The range_nlValue is  applied first and thus supercedes range adjustments (minimum/maximum ranges).  So if your minimum or maximum ranges are not taking effect, check to make sure there is not a range_nlValue set for the weather element in question.

Ranges are adjusted at several points in the processing of phrases:

• When statistics are initially calculated and entered into the StatisticsDictionary by the ForecastNarrative module.
• When statistics are gathered over multiple time ranges from the StatisticsDictionary by the "tree.stats.get" command.
• When statistics are combined (e.g. combining sub-phrases)

Example: Using Maximum Range and Bias

To accomplish this, include the following in the  Overrides file:

    def maximum_range_nlValue_dict(self, tree, node):
        # Maximum range to be reported within a vector phrase
        #   e.g. 5 to 10 mph
        # Units depend on the product
        dict = TextRules.TextRules.maximum_range_nlValue_dict(self, tree, node)
        mountainZones = ["area3"]
        if self.currentAreaContains(tree, mountainZones):
            range = 10
        else:
            range = 5
        dict["MaxT"] = range
        dict["MinT"] = range
        dict["MaxRH"] = range
        dict["MinRH"] = range
        return dict

    def maximum_range_bias_nlValue_dict(self, tree, node):
        # "Min", "Average", "Max"
        #  Should the maximum_range be taken from the "min" "average" or "max"
        #  value of the current range?
        dict = TextRules.TextRules.maximum_range_bias_nlValue_dict(self, tree, node)
        dict["MaxT"] = "Max"
        dict["MinT"] = "Min"
        dict["MaxRH"] = "Max"
        dict["MinRH"] = "Min"
        return dict

    def increment_nlValue_dict(self, tree, node):
        # Increment for rounding values
        # Units depend on the product
        dict = TextRules.TextRules.increment_nlValue_dict(self, tree, node)
        dict["MaxT"] = 5
        dict["MinT"] = 5
        dict["MaxRH"] = 5
        dict["MinRH"] = 5
        return dict

Data Value Extraction and Combining

    def scalar_difference_nlValue_dict(self, tree, node):
        # Scalar difference.  If the difference between scalar values
        # for 2 sub-periods is greater than this value,
        # the different values will be noted in the phrase.

    def vector_mag_difference_nlValue_dict(self, tree, node):
        # Replaces WIND_THRESHOLD
        # Magnitude difference.  If the difference between magnitudes
        # for sub-ranges is greater than this value,
        # the different magnitudes will be noted in the phrase.
        # Units can vary depending on the element and product

Data Value Extraction Consistency

• the minimum_range and maximum_range must be multiples of the increment
• minimum_range must be less or equal to the maximum_range

NULL VALUES

    def null_nlValue_dict(self, tree, node):
        # Magnitude threshold below which values are considered "null" and  not reported.
        return {
            "Wind": 5,  # mph
            "TransWind": 0,  # mph
            "FreeWind": 0,  # mph
            "Swell": 5,  # ft
            "Swell2": 5,  # ft
            "WaveHeight": 3,  # ft
            }

You may specify the phrases that will be used to report Null values.  These phrases may be the empty phrase, "".   You may want to specify a different phrase if the the null value occurs for the entire time period or the for the first sub-phrase versus in subsequent sub-phrases.  Here are the default null phrase dictionaries from the ConfigVariables module:

    def first_null_phrase_dict(self, tree, node):
        # Phrase to use if values THROUGHOUT the period or
        # in the first period are Null (i.e. below threshold OR NoWx)
        # E.g.  LIGHT WINDS.    or    LIGHT WINDS BECOMING N 5 MPH.
        return {
            "Wind": "light winds",
            "TransWind": "",
            "FreeWind": "",
            "Swell": "light swells",
            "Swell2": "",
           "Wx": "",
            "WindGust": "",
            "Wave": "2 feet or less",
            }

    def null_phrase_dict(self, tree, node):
        # Phrase to use for null values in subPhrases other than the first
        # Can be an empty string
        #  E.g.  "NORTH WINDS 20 to 25 KNOTS BECOMING LIGHT"
        return {
            "Wind": "light",
            "TransWind": "light",
            "FreeWind": "light",
            "Swell": "light",
            "Swell2": "",
            "Wx":"",
            "WindGust": "",
            "Wave": "2 feet or less",
            }

LOCAL EFFECTS

• Define contrasting edit areas. Determine what areas might exhibit results that are significantly different.  Create edit areas for them through the GFESuite GUI.  For example, we might expect that elevations above 10,000 feet will have temperatures significantly different from those below 10,000 feet so we create an edit area through the GFE for each.
• Specify the areas that need to be intersected with the current area to examine for local effects.  List these by weather element in the "intersectAreas"  of the Component Product definition.
• Specify the areas that do not need to be intersected with the current area to examine for local effects.  You might want to list areas for local effect comparison "as is" i.e. without intersecting them with the current area.  If so, list these in the "additionalAreas" section of the Product Component definition.
• Set up a "localEffectsList" for the phrase(s) for which you want local effects to be generated.  The local effects list consists of 1 or more LocalEffects.  If  you have more than one LocalEffect, each is examined in order. The first local effect to fire will be reported.  NOTE: The List of Local Effects may be specified as a Python method which will be given arguments (tree, node) and should return a list of Local Effects.A Local Effect consists of the following information:
• List of Local Effect Areas.  You may list 2 or more areas for comparison.  When there are more than 2 areas, the system determines "groupings" of areas with similar statistics and reports them together.  NOTE: The List of Local Effects may be specified as a Python method which will be given arguments (tree, node) and should return a list of Local Effect Areas.  Each Local Effect Area consists of the following:
• Edit Area Name.   This can be a special string "__Current__" to denote the current edit area.
• Description String: This is the string to be used in the resulting phrase e.g. "in the mountains", "on the coast".  NOTE: the Description String can be specified as a Python method which will be given arguments (tree, node, leArea) and should return a description string.  In this way, you can determine the description string on-the-fly perhaps dependent on the current edit area.
• Difference Threshold: This is a scalar difference value to be used in comparing the local effect areas.   NOTE: the Difference Threshold can be specified as a Python method which will be given arguments (tree, node, localEffect, leArea1Label, leArea2Label) and should return 1 if there is a local effect difference detected between the two areas.   This will allow more flexibility in drawing comparisons between local effect areas.
• Local Effect Connecting Words:  This is a word string to connect local effects that are reported within one phrase.  E.g.  "Highs 45-55...except35-45 above timberline."

   def Period_1(self):
        return {
            "type": "component",
            "methodList": [
                          self.orderPhrases,
                          self.assemblePhrases,
                          self.wordWrap,
                          ],
            "analysisList": [
                       ("MinT", self.stdDevMinMax),
                       ("MaxT", self.stdDevMinMax),
                       ("T", self.hourlyTemp),
                       ("Wind", self.vectorMedianRange, [6]),
                       ("Wind", self.vectorMinMax, [6]),
                       ("WindGust", self.avg, [6]),
                       ],
            "phraseList":[
                   self.wind_summary,
                   self.reportTrends,
                   #self.highs_phrase,
                   #self.lows_phrase,
                   (self.highs_phrase, self._tempLocalEffects_list()),
                   (self.lows_phrase, self._tempLocalEffects_list()),
                   #self.highs_range_phrase,
                   #self.lows_range_phrase,
                   #self.wind_withGusts_phrase,
                   (self.wind_withGusts_phrase, self._windLocalEffects_list()),
                   ],
            "intersectAreas": [
                   # Areas listed by weather element that will be
                   # intersected with the current area then
                   # sampled and analysed.
                   # E.g. used in local effects methods.
                   ("MaxT", ["Mountains", "Valleys"]),
                   ("MinT", ["Mountains", "Valleys"]),
                   ("Wind", ["Coast", "Inland"]),
                   ("WindGust", ["Coast", "Inland"]),
 ]

    def _windLocalEffects_list(self):
        leArea1 = self.LocalEffectArea("Coast", "near the coast")
        leArea2 = self.LocalEffectArea("Inland", "inland")
       return [self.LocalEffect([leArea1, leArea2], 10, " and ")]

    def _tempLocalEffects_list(self):
        leArea1 = self.LocalEffectArea("Valleys", "")
        leArea2 = self.LocalEffectArea("Mountains", "in the mountains")
        return [self.LocalEffect([leArea1, leArea2], 8, "...except ")]

As a result,   the system will automatically calculate the intersection of the Mountains, Valleys, (for MaxT, MinT) and Coast, Inland (for Wind and WindGust) areas with each edit area as it processes it.   Sampling and Analysis statistics will be calculated for both intersection areas for the appropriate weather elements. Then, the local effects phrase can examine the statistics for differences and report according to the methods and words given in the local effects definition. For example, the following phrases could be generated depending on the data:

    "Highs 45-55."
    "Highs 45-55...except 35-45 above timberline."
    "North winds 20 mph."
    "North winds 20 mph near the coast and  west winds 10 mph inland."
    "Near the coast...North winds 20 mph becoming 10 mph in the afternoon. Inland...North winds 10 mph becoming light in the afternoon."

Local Effects and Compound Phrases

    def skyWeather_byTimeRange_compoundPhrase(self):
        return {
            "phraseList": [
                self.fireSky_phrase
                #self.weather_phrase,
                (self.weather_phrase, self._generalLocalEffects_list()),
                ],
            "phraseMethods": [
                self.assembleSentences,
                self.skyWeather_finishUp,
            ],
            }

     def _generalLocalEffects_list(self):
        leArea1 = self.LocalEffectArea("Coast",  "on the coast")
        leArea2 = self.LocalEffectArea("Inland", "inland")
        leArea3 = self.LocalEffectArea("City", "in the city")
        return [self.LocalEffect([leArea1, leArea2, leArea3], 5, " ")]

Remember, as always, to include the local effect areas in the "intersectAreas" list within the component definition:

     def getFirePeriod_intersectAreas(self):
        return [
            ("MinT", ["BelowElev", "AboveElev"]),
            ("MaxT", ["BelowElev", "AboveElev"]),
            ("MinRH", ["BelowElev", "AboveElev"]),
            ("MaxRH", ["BelowElev", "AboveElev"]),
            ("RH", ["BelowElev", "AboveElev"]),
            ("Wx", ["BelowElev", "AboveElev"]),
            ]
 

Consolidation Rules for Local Effects

• Embedded Local Effects for the simple case of no sub-phrases simply use the words supplied in the Local Effect definition:
• "Highs 45 except 25 above timberline."
• "North winds 20 to 25 mph near the coast and west winds 10 mph inland."

• Conjunctive Local Effects for multiple sub-phrases: If any sub-phrase has a local effect, separate into two phrases, one for each local effect area.  Qualify each phrase with the descriptive words for the local effect area.
• Instead of: "North winds 20 to 25 mph near the coast and west winds 10 mph inland becoming 10 mph near the coast and light inland in the afternoon."

Produce: "Near the coast...North winds 20 to 25 mph becoming 10 mph in the afternoon.  Inland...North winds 20 to 25 mph becoming light in the
• afternoon."
• Instead of: "North winds 20 to 25 mph except west winds 10 mph in the valleys becoming 10 mph except light in the valleys in the afternoon."

Produce: "North winds 20 to 25 mph becoming 10 mph. In the valleys...west winds 10 mph becoming light in the afternoon."

Local Effect Examples

Null Values and Local Effects

Set Up:

    def _windLocalEffects_list(self):
        srnInland = self.LocalEffectArea("Inland","")
        srnBeaches = self.LocalEffectArea("Beaches","at the beaches")
        return [self.LocalEffect([srnInland, srnBeaches], 5, "...except ")]

The local effect descriptor will appear at the end of the sentence IF there is only one subphrase.

SOUTHWEST WINDS 25 TO 35 MPH AT THE BEACHES.

Otherwise, the sentence would be  ambiguous:

SOUTHWEST WINDS 25 TO 35 MPH INCREASING TO 40 TO 50
MPH IN THE AFTERNOON AT THE BEACHES.

So the system  produces:

AT THE BEACHES...SOUTHWEST WINDS 25 TO 35 MPH INCREASING TO 40 TO 50
MPH IN THE AFTERNOON.

Null values and Wx: For periods in which you have Local Effects for Wx, you might want to specify afirst_null_nlValue and null_nlValue of "dry" so that the local effect can be correctly reported.  For example:

def null_nlValue_dict(self, tree, node):
        # Descriptors for phrases
        dict = TextRules.TextRules.null_nlValue_dict(self, tree, node)
        componentName = node.getComponentName()
        if componentName == "Period_1":
             dict[Wx"] = "dry"
        return dict

Then you will get phrases like:

SCATTERED SHOWERS...EXCEPT DRY IN THE BLUE MOUNTAINS.

instead of just:

SCATTERED SHOWERS.

"Except" versus "Otherwise" Wording

    def _windLocalEffects_list(self):
        srnInland = self.LocalEffectArea("BelowElev","")
        srnBeaches = self.LocalEffectArea("AboveElev","at the beaches")
        return [self.LocalEffect([srnInland, srnBeaches], 5, "...except ")]

SOUTHWEST WINDS 5 TO 15 MPH...EXCEPT SOUTHWEST 25 TO 35 MPH AT THE BEACHES.

SOUTHWEST WINDS 5 TO 15 MPH. AT THE BEACHES...SOUTHWEST WINDS 25 TO
35 MPH INCREASING TO 40 TO 50 MPH IN THE AFTERNOON.

Set-up:

    def _windLocalEffects_list(self):
        srnInland = self.LocalEffectArea("BelowElev","", "inland" )
        srnBeaches = self.LocalEffectArea("AboveElev","at the beaches")
        return [self.LocalEffect([srnBeaches, srnInland,], 5, "...otherwise ")]

SOUTHWEST WINDS 25 TO 35 MPH AT THE BEACHES...OTHERWISE SOUTHWEST 5 TO
15 MPH.

AT THE BEACHES...SOUTHWEST WINDS 25 TO 35 MPH INCREASING TO 40 TO 50
MPH IN THE AFTERNOON. INLAND...SOUTHWEST WINDS 5 TO 15 MPH.

More Local Effect Features

• using "additional" (not "intersect") areas for Local Effects
• using a method for defining the LocalEffect list based on the current edit area
• using the __Current__ area for comparison
• checking for multiple Local Effects in order of precedence

Set-up:

In the product component definition, we specify the local effects list as a method rather returning a list.  This is necessary since we want to define the list "on-the-fly" based on the current edit area:

            (self.highs_phrase, self._tempLocalEffects_list),
                (self.lows_phrase, self._tempLocalEffects_list),

Then we add in the additonal areas to be sampled and analyzed:

            "additionalAreas": [
                   # Areas listed by weather element that will be
                   # intersected with the current area then
                   # sampled and analyzed.
                   # E.g. used in local effects methods.
                   ("MaxT", ["Benches", "Rush_Valley"]),
                   ("MinT", ["Benches", "Rush_Valley"]),
             ],

Finally, we provide the method for defining the local effects list "on-the-fly."

    def _tempLocalEffects_list(self, tree, node):
        if self.currentAreaContains(tree, ["area1"]):
           leArea1 = self.LocalEffectArea(
                    "__Current__","",intersectFlag=0)
           leArea2 = self.LocalEffectArea("Rush_Valley",
                    "in the rush valley", intersectFlag=0)
           leArea3 = self.LocalEffectArea(
                    "Benches", "in the benches",intersectFlag=0)
           return [
               self.LocalEffect([leArea1,leArea2],5,"...except "),
               self.LocalEffect([leArea1,leArea3],5,"...except "),
               ]
        else:
           return []

Note that the Rush Valley will be checked first.  If a local effect is found, it will be reported:

HIGHS IN THE UPPER 50S...EXCEPT IN THE UPPER 20S TO UPPER 40S IN THE RUSH VALLEY.

If not, the Benches will be checked and if a local effect is found, it will be reported:

HIGHS IN THE UPPER 50S...EXCEPT IN THE UPPER 20S TO UPPER 40S IN THE BENCHES.

Finally, if neither area triggers a local effect, a simple phrase will be reported:

HIGHS IN THE UPPER 50S.
 

Local Effects for Snow Accumulation and Total Snow Accumulation -- Using a Method for the Checking the Local Effect Threshold

SNOW ACCUMULATION 4 INCHES...EXCEPT 7 INCHES ABOVE
TIMBERLINE. TOTAL SNOW ACCUMULATION 9 INCHES...EXCEPT 18 INCHES
ABOVE TIMBERLINE.

In your overrides file, you must override the product component(s) for which you want to report local effects for the snow_phrase and/or total_snow_phrase. In these component definitions, specify the local effect as follows:

                   (self.snow_phrase,self._snowAmtLocalEffects_list()),
                   (self.total_snow_phrase,self._totalSnowAmtLocalEffects_list()),

Next, add the "intersectAreas" to these components for SnowAmt and IceAmt (note that IceAmt is handled by the SnowAmt phrase):

            "intersectAreas": [
                   # Areas listed by weather element that will be
                   # intersected with the current area then
                   # sampled and analysed.
                   # E.g. used in local effects methods.
                   ("SnowAmt", ["BelowElev", "AboveElev"]),
                   ("IceAmt", ["BelowElev", "AboveElev"]),
                   ],

IMPORTANT NOTE:  The total_snow_phrase looks AHEAD one period for information about snow ending (SnowAmt = 0). Therefore, you must include the "intersectAreas" in the component FOLLOWING the ones in which you are specifying a local effect for the total_snow_phrase. For example, in the ZFP_<site>_Overrides file using the 10-503 directive, if I set up a total_snow_phrase local effect for Period_1 and Period_2_3, I must also include the "intersectAreas" for Period_4_5.

Finally,  include the following local effect lists in your Overrides file:

    def _snowAmtLocalEffects_list(self):
        leArea1 = self.LocalEffectArea("BelowElev", "")
        leArea2 = self.LocalEffectArea("AboveElev", "above timberline")
        return [self.LocalEffect([leArea1, leArea2], 2, "...except ")]

    def _totalSnowAmtLocalEffects_list(self):
        leArea1 = self.LocalEffectArea("BelowElev", "")
        leArea2 = self.LocalEffectArea("AboveElev", "above timberline")
        return [self.LocalEffect(
            [leArea1, leArea2], self._checkTotalSnow, "...except ")]

    def _checkTotalSnow(self, tree, node, localEffect, leArea1Label, leArea2Label):
        totalSnow1 = self.getTotalSnow(tree, node, leArea1Label)
        totalSnow2 = self.getTotalSnow(tree, node, leArea2Label)
        if totalSnow1 is None or totalSnow2 is None:
            return 0
        if abs(totalSnow1 - totalSnow2) > 3:
            return 1
        return 0

Local Effects for the Combined SkyPopWx Phrase

     "phraseList":[
                   (self.sky_phrase, self._skyLocalEffects_list()),
                   (self.skyPopWx_phrase, self._skyPopWxLocalEffects_list()),
                   (self.weather_phrase,self._wxLocalEffects_list()),
                   (self.popMax_phrase, self._popLocalEffects_list()),
                  ...
                   ],                                 

And also within the product component definition:

            "intersectAreas": [
                   ("Sky", ["AboveElev", "BelowElev"]),
                   ("Wx",  ["AboveElev", "BelowElev"]),
                   ("PoP", ["AboveElev", "BelowElev"]),
                   ], 

 PERIOD COMBINING

    Definition["periodCombining"] = 1     # If 1, do period combining
 

Period combining is set up by methods found in the PhraseBuilder module:

1) If the periodCombining flag is set to 1 in the product, we add a method to the product narrative tree top level node called  "combineComponents" (in PhraseBuilder module).
2) The "combineComponents" method is called at the top of the tree and compares all consecutive components for similarity.  Also, we are only combining components that begin 36 hours after the issuance time of the product.
3) If two components are deemed similar (according to the "periodCombining_elementList" (in ConfigVariables)), then the component nodes are collapsed into one node with the time range spanning the original nodes.
4) From here all phrases and processing pretty much the same as it would have been for the original nodes.  To simplify combined period phrasing, there is a threshold "collapsedSubPhrase_hours" (in ConfigVariables) which is set to 12 hours as the default. If the period is longer than these hours, subphrases will automatically be collapsed.  

Periods can be combined providing the differences between the adjacent periods are small.  There is one method per weather element that determines whether two periods should be combined or not.  These methods are found in the PhraseBuilder module.  Each of them begin with "similar" followed by the weather element name.  The table below shows the methods that have been implemented thus far and their thresholds.
 

Element Name as listed in the periodCombining_elementList	Method Name (in PhraseBuilder module)	Thresholds
Sky	similarSky	Sky words are similar
Wind	similarWind	mag within 10 knots    dir within 45 degrees
Wx	similarWx	Identical wx keys per each analyzed sub-range
PoP	similarPoP	PoPs are equal or both below pop_lower_threshold or both above pop_upper_threshold
MaxT	similarMaxT	MaxT within 5 degrees
MinT	similarMinT	MinT within 5 degrees
WaveHeight	similarWaveHeight	WaveHeights within 4 feet
DiurnalSkyWx	similarDiurnalSkyWx	DiurnalSkyWx will combine if there is a diurnal pattern of sky and wx. For example,  "CLOUDY IN THE NIGHT AND MORNING...OTHERWISE CLEAR." or "LOW CLOUDS AND FOG IN THE NIGHT AND MORNING...OTHERWISE CLEAR." Note that it will also combine if the sky and wx are similar without a diurnal pattern.

Each of these methods can be overridden in your Overrides file.  In general, the threshold values are defined near the top of the method, so you won't need to hunt for them or change them in several places.  If you desire to change the thresholds, copy the method into your Overrides file and modify the threshold value.  You can also modify the algorithm, if you like.  The default method retrieves the statistics and examines them to see if the values between each component are close enough to combine.  You will likely want to leave the code that retrieves the statistics unmodified.  But you can choose to change the logic that decides if the values are "similar".

When components are collapsed, we do not want phrases to have detailed sub-phrases as we would in a normal 12-hour period.  Sub-phrases are then automatically collapsed if the combined period exceeds the collapsedSubPhrase_hours  found in the ConfigVariables module.   When sub-phrases are collapsed, the data is summarized according to the mergeMethod for the weather element being collapsed.  These settings can be overridden in your Overrides file.

    def collapseSubPhrase_hours_dict(self, tree, node):
        # If the period is longer than these hours, subphrases will automatically
        # be collapsed.
        return {
            "otherwise": 24,
            #"Wx": 24,
            }

    def mergeMethod_dict(self, tree, node):
        # Designates the mergeMethod to use when sub-phrases are automatically collapsed.
        return {
            "otherwise": "Average",
            "MinT": "Min",
            "MaxT": "Max",
            "PoP": "Max",
            "Wx": "List",  # Still want Wx to be broken out
            }

 PHRASE CONSOLIDATION

Definitions: Phrases are composed of sub-phrases depending on the temporal resolution of the statistics.  For example, the Wind phrase in the ZFP is composed of 2 6-hour sub-phrases, e.g. "North wind 10-15 mph increasing to 20 mph in the afternoon." A phrase can report on multiple elements (e.g. Wind and WindGust).  "North wind 10-15 mph with gusts up to 35 mph."  Weather elements in these phrases are considered "primary" and "secondary".

• More than one element can be designated as "primary" in a phrase. For example, for the wind phrase "Wind" and "WindGust" would both be primary.
• Though there can be more than one primary element for consolidation purposes, the sub-phrase time ranges will be based upon the first element's statistics.
• Consolidation Rule: Separate any "primary elements" that are similar throughout the time period into a separate phrase.
• Instead of: "North wind 10 mph with gusts up to 35 mph increasing to 20 mph with gusts up to 35 mph in the afternoon."
• Produce: "North wind 10 mph increasing to 20 mph in the afternoon. Gusts up to 35 mph."
• Elements can still be designated as secondary so that when combined, an average, min, max, or sum of their values can be used. Suppose that Swell and Swell2 are primary and Period and Period2 are secondary.
• Instead of: "North swell 10 ft at 5 seconds and northeast swell 12 ft at 5 seconds becoming north swell 10 ft at 15 seconds and north swell 15 ft at 5 seconds"
• Produce: "North swell 10 ft at 10 seconds and northeast swell 12-15 ft at 5 seconds."
• Consolidation Rule For Vector Direction: If there are no secondary elements and the direction is a progression with similar magnitudes, report only the first and last directions.
• Instead of: "North wind 10 mph becoming northeast 15 mph in the late morning becoming east 10 mph."
• Produce: "North wind 10-15 mph becoming east 10 mph in the afternoon."
• Consolidation Rule For Trends: If there are no secondary elements and the magnitude is a progression over the time period, report only the first and last magnitudes with no time descriptors.
• Instead of: "Waves 2 to 4 feet building to 3 to 6 ft in the late morning and early afternoon building to 4 to 7 feet."
• Produce: "Waves 2 to 4 feet building to 4 to 7 feet."

PHRASE ORDERING

.TODAY...STRONG WINDS. THUNDERSTORMS WITH HEAVY RAINFALL.  HIGHS
AROUND 80. SOUTHEAST WINDS AROUND 70 MPH BECOMING SOUTH AROUND 105
MPH IN THE AFTERNOON.

The winds are the most dominant feature and thus you might want to move the "wind_withGusts_phrase" closer to the beginning of the forecast.  To re-order phrases on-the-fly, follow steps similar to this example:

--Make sure the Product Component Definition for any Periods in which you'd like to order the phrases has the method, "orderPhrases", prior to "assembleSubPhrases":

    def Period_1(self):
        component =  {
            "type": "component",
            "methodList": [
                          self.orderPhrases,
                          self.assemblePhrases,
                          self.wordWrap,
                          ],
            ....

 --Modify the "orderPhrases"(PhraseBuilder) method, for example:

    def orderPhrases(self, tree, component):
        reorderList = []
        timeRange = component.getTimeRange()
        areaLabel = component.getAreaLabel()

        # Put in some logic to see if you want to re-arrange the order
        # of the phrases.  For example:
        # Check for high winds
        windMax, dir = tree.stats.get("Wind", timeRange, areaLabel, mergeMethod="Max")
        if windMax > 50:
           # Put wind phrase first
           reorderList.append(("wind_withGusts_phrase", "weather_phrase"))

        # Apply any reorderings that were identified
        for name1, name2 in reorderList:
           self.moveAbove(tree, component, name1, name2)
        return self.DONE()

"UNTIL" PHRASING

        "North winds 20 mph until 10 AM then 35 mph."

To accomplish this, override the flags below from the ConfigVariables module.  Be sure that if you want higher temporal resolution in your phrase, to set the temporal resolution in the analysisList accordingly.  For example:

                       ("Wind", self.vectorMedianRange, [0]),

    def untilPhrasing_flag_dict(self, tree, node):
        # If set to 1, "until" time descriptor phrasing will be used.
        # E.g. "NORTH WINDS 20 MPH UNTIL 10 AM...THEN 35 MPH"
        return {
            "otherwise": 0,
            #"Wind" : 1,
            }

    def onTheFly_untilPhrasing_flag_dict(self, tree, node):
        # If set to 1, "until" time descriptor phrasing will be used ONLY if
        # the time range for a sub-phrase does not end on a 3-hour boundary.
        return {
            "otherwise": 1,
            #"Wind" : 1,
            }

    def untilPhrasing_format_dict(self, tree, node):
        # Format for "until" time descriptors.
        # If "military": UNTIL 1000
        # If "standard": UNTIL 10 AM
        return {
            "otherwise": "military",
            #"Wind": "standard",
            }

The untilPhrasing may apply to only certain components of your product.  You can make this distinction as in this  example from the FWF:

    def untilPhrasing_flag_dict(self, tree, node):
        # If set to 1, "until" time descriptor phrasing will be used.
        # E.g. "NORTH WINDS 20 MPH UNTIL 10 AM...THEN 35 MPH"
        # Be sure to increase the temporal resolution if desired:
        # E.g.  ("MixHgt", self.minMax, [0]),
        dict = TextRules.TextRules.untilPhrasing_flag_dict(self, tree, node)
        dict["LAL"] = 1
        componentName = node.getComponent().get("name")
        if componentName == "FirePeriod":
            dict["Sky"] = 1
            dict["Wx"]  = 1
        return dict

VISIBILITY

Example 1:
  Grids:
      Hour 1-6      Patchy F 1/4SM Chc R 1/2SM
      Hour 7-12   Sct  RW

  Phrase:
CHANCE OF LIGHT RAIN AND PATCHY FOG WITH VISIBILITY LESS THAN 1 NM THEN SCATTERED RAIN
SHOWERS IN THE AFTERNOON.

Example 2:
  Grids:
      Hour 1-6      Patchy F 1/4SM Chc R 1/2SM
      Hour 7-12   Sct  RW 11/2SM

  Phrase:  CHANCE OF LIGHT RAIN AND PATCHY FOG WITH VISIBILITY LESS THAN 1 NM THEN SCATTERED RAIN SHOWERS WITH 2 NM VISIBILITY IN THE AFTERNOON.

The wording for visibility within the weather phrase can be configured by overriding the visibility_weather_phrase_nlValue.

You can set the null_nlValue  for "Visibility" so that only visibilities below the threshold are reported.  The value should be in nautical miles. Suppose we have set this value to 3 nautical miles, Then:

Example 3:
  Grids:
      Hour 1-6      Patchy F 4SM Chc R
      Hour 7-12   Sct  RW

  Phrase:
CHANCE OF LIGHT RAIN AND PATCHY FOG THEN SCATTERED RAIN SHOWERS IN THE AFTERNOON.

You can set the  visibility_wx_threshold  so that weather is only reported if visibility is below the threshold.   Also, in this case you might want to set up a list of significant_wx_visibility_subkeys.   If any of these subkeys appear in the sub-phrase, the weather will be reported regardless of the visibility.  Suppose we have:

    def visibility_wx_threshold(self, tree, node):
        # Weather will be reported if the visibility is below
        # this threshold (in NM) OR if it includes a
        # significant_wx_visibility_subkey (see below)
        return 3

    def significant_wx_visibility_subkeys(self, tree, node):
        # Weather values that constitute significant weather to
        # be reported regardless of visibility.
        # If your visibility_wx_threshold is None, you do not need
        # to set up these subkeys since weather will always be
        # reported.
        # Set of tuples of weather key search tuples in the form:
        #  (cov type inten)
        # Wildcards are permitted.
        return [("* T"), ("* ZY")]

then:

Example 4:
  Grids:
      Hour 1-6      Patchy F 1/4SM Chc R 4SM
      Hour 7-12   Sct  RW

  Phrase:  CHANCE OF LIGHT RAIN AND PATCHY FOG WITH VISIBILITY LESS THAN 1 NM IN THE MORNING.

Example 5:
   Hour 1-6      F 1/4SM  RW 1/2SM
   Hour 7-12   TRW

  Phrase:   CHANCE OF LIGHT RAIN AND PATCHY FOG WITH VISIBILITY LESS THAN 1 NM THEN SCATTERED SHOWERS AND
THUNDERSTORMS IN THE AFTERNOON.

Finally, there is a consolidation method (consolidateVisibility in WxPhrases).  If the low visibility is constant across sub-phrases, visibility is separated out into its own phrase:

Example 6:
   Hour 1-6      F 1/4SM  RW 1/2SM
   Hour 7-12   TRW  1/4SM

  Phrase:   CHANCE OF LIGHT RAIN AND PATCHY FOG THEN SCATTERED SHOWERS AND THUNDERSTORMS IN THE AFTERNOON. VISIBILITY LESS THAN 1 NM.

The wording for visibility within the visibility_phrase can be configured by overriding the visibility_phrase_nlValue.

If you always want visibility reported in its own phrase, you can include the visibility_phrase in your phrase list ANDmake sure to set the  embedded_visibility_flag to zero so that it does not appear redundantly within the weather phrase.
 

WEATHER PHRASING

Weather phrasing in the formatters involves many modules and logic threads as outlined in the following diagram:



Weather phrasing is complex, but centers around two primary methods:  "makeAggregateSubkey" (CommonUtils) and "checkWeatherSimilarity" (PhraseBuilder).  These methods are ultimately employed throughout the many phases of text generation: Analysis, Merging of grid statistics, Sub-phrase Combining, Local Effects, Period Combining and Weather Wording.  This ensures consistency throughout the process and makes customizing easier.

The makeAggregateSubkey method consolidates two subkeys according to the following:

• coverage and intensity:  choose highest ranked, or if similar ranks,  strongest
• visibility: choose lowest visibility
• attributes:  make aggregate list
  

• Have the same number of subkeys,
• If there is just one subkey per rankList, the subkeys must have the same weather type,  similar intensities AND similar coverages,
• The rankLists differ only in intensity (this test is skipped if "noIntensityCombining" is turned on for the node e.g. severeWeather_phrase),
  
• If there is more than one subkey per rankList, they must pass the "checkSubkeysSimilarity" test:
  
• The set of wxTypes in rankList1 is equal to the set of wxTypes in rankList2 AND
  
• Each wxType individually can be combined i.e. they have similar coverages.   
  
• In the case that they can be combined, the "checkSubkeysSimilarity" method returns an aggregate "rankList" which it creates using "makeAggregateSubkey".
  

• If the filter_subkeys_flag is set to 0, no filtering is done.  Default is 1.
• Subkeys with the same weather type are combined using the "makeAggregateSubkey" method.
  
• Subkeys with differing weather types are combined according to the wxCombinations list:

     def combine_T_RW(self, subkey1, subkey2):
        # Combine T and RW only if the coverage of T
        # is dominant over the coverage of RW
        wxType1 = subkey1.wxType()
        wxType2 = subkey2.wxType()
        if wxType1 == "T" and wxType2 == "RW":
            order = self.dominantCoverageOrder(subkey1, subkey2)
            if order == -1:
                return 1, subkey1
        return 0, None
 

• "weather_phrase", "severeWeather_phrase", "heavyPrecip_phrase" rely on the "preProcessWx" method to combine the weather types found in the time period of interest.  This method uses the "checkWeatherSimilarity" method to determine if subkeys can be combined and the "makeAggregateSubkey" method to create new subkeys for the resulting sub-phrases.  Subkeys are put into sub-phrases with others of the same time span within the time period.  This method looks at all the subkeys for all sub-phrases at one time.  For example, if you had
  
• Hours 1-3: Wide R, SChc S
• Hours 4-6: SChc S
• Hours 7-12: SChc S Lkly ZR

• SubPhrase 1: Hours 1-3:  Wide R
• SubPhrase 2: Hours 1-12: SChc S
• SubPhrase 3: Hours 7-12: Lkly ZR

• "skyPopWx_phrase": This phrase is used only for the simpler case of 1 or 2 resulting sub-phrases.  Thus, we use the simpler sub-phrase by sub-phrase combining method, "combineStats" which also employs the "checkWeatherSimilarity" and "makeAggregateSubkey" methods.   The sub-phrase by sub-phrase combining allows us to have phrases such as: "Rain and snow in the morning then snow in the afternoon."  instead of "Rain in the morning.  Snow through the day."

• If the PoP is below the pop_wx_lower_threshold,   pop_related_wxTypes are removed.
• Check for weather wording dependent on the visibility threshold.  See section: Visibility
• If useSimple  (default is 2 or fewer subkeys),  use old-style simpler wording i.e. do not use "mixed with", "pockets of", etc.  Examples of Simple wording: "Rain and snow likely in the evening." "Chance of rain and snow and slight chance of sleet in the evening." "Chance of showers." "Mostly cloudy with rain likely and a slight chance of thunderstorms."   Example of  Complex wording: "Rain mixed with snow likely in the evening." "Chance of rain mixed with snow and sleet in the evening."
• Otherwise, use ranking of subkeys to form wording:
• For subkeys with similar rank and coverages, use "mixed with" wording .
• For subkeys with lesser coverage than the highest ranked, use "possibly mixed with".
• For subkeys with greater coverage than the highest ranked, use "with pockets of"

• The "weather_value" method constructs the words for each subkey.  This wording can be adjusted and is described in the section: Customizing Weather Phrases.
• Add embedded visibility wording if needed.

Appendix A

Local Effects Re-design/Design to Handle Combined Phrases

Phrase Processing Steps including Local Effects

• CONSOLIDATE --Phrase Level: e.g. separateNonPrecip (separate out non-precip events such as Fog from RW), consolidateWx (if RW appears throughout the period and S is just in the afternoon, split them into separate phrases), consolidatePhrase (if Gusts are the same throughout the period and the Wind changes, split them into separate phrases).
• CHECK FOR LOCAL EFFECTS -- Phrase Level:  If local effects are triggered, create separate Conjunctive phrases for each local effect area (or group of areas).
  
• COMBINE -- Phrase Level:  Combine sub-phrases based on similar data.
• CREATE WORDS -- Sub-phrase Level: Create words for each sub-phrase.
• CONSOLIDATE SUB-PHRASES -- Component Level: Consolidate so that we do not have duplicate phrasing within local effect areas. 
  
• ASSEMBLE SUB-PHRASES -- Phrase Level.
  
• ASSEMBLE PHRASES -- Component Level:
  
• Order Wx Phrases
  
• Consolidate Local Effect Phrases
• Incorporate Non Local Effect phrases (not implemented) OR
• Convert to Embedded phrases if possible
• Order Local Effect phrases that remain Conjunctive
• Assemble Component Words with appropriate qualifiers
  

Examples of Phrase Processing Steps

• Area1 is the entire area while WW (Windward)  and LW (Leeward) are local effect areas.
• SPW = skyPopWx_phrase
• Wx = weather_phrase
• PoP = popMax_phrase
• Sky = sky_phrase
• >> Means "executes"
  
• --> Means "results in"

Test Case 3_3 F20

• Sky Phrase >> includeSky == 0 since same throughout, so kept phrase >> empty words since PoP > 60
  
• SPW Area1  >> Consolidation -->
  
•  SPW Area1 for Chc R Lkly S  >> CheckLocalEffects -->
• SPW WW for Lkly R Lkly S --> RAIN AND SNOW LIKELY WINDWARD
• SPW LW for Chc R --> A 40 PERCENT CHANCE OF RAIN LEEWARD
• SPW  Area1 for Iso T >> CheckLocalEffects --> ISOLATED THUNDERSTORMS IN THE MORNING.
  
• Wx Area1 >> Consolidation (preProcessWx rearranges sub-phrases) >> CheckLocalEffects -->
• Wx WW >> removed by SPW Windward Lkly R Lkly S
• Wx LW >> removed by SPW Leeward Chc R
• Pop Area1 >> Removed by SPW Windward and Leeward
  

Test Case 3_3 F6 

• Sky Area1 >> CheckLocalEffects -->
• Sky WW --> Removed by SPW for WW
  
• Sky LW --> SUNNY
  
• SPW Area1  >> Consolidation  >> CheckLocalEffects -->
  
• SPW WW --> MOSTLY CLOUDY WITH SCATTERED SHOWERS
  
• SPW LW   -- Removed because there is NoWx
  
• Wx Area1 >> CheckLocalEffects -->
• Wx WW --> removed by SPW WW -->
  
• Wx LW --> empty
  
• Pop Area1 >> CheckLocalEffects -->
• Pop WW --> CHANCE OF SHOWERS 50 PERCENT
• Pop LW --> empty
  

Test Case 3_3 F30  Sub-phrase consolidation Case 1

• Sky Area1 --> MOSTLY SUNNY.
  
• SPW Area1  >> Consolidation -->
  
• Spawned Wx Area1 (F) --> Consolidation --> CheckLocalEffects -->
• Wx WW  Patchy F
• Wx Leeward Widespread F
  
• Wx Area1 >> CheckLocalEffects -->
• Wx WW --> Patchy F
  
• Wx LW --> Widespread F
  
• Pop Area1 >> No PoP reported.
  

• >> All these subPhrases are processed by ConsolidateSubPhrases which finds Case 1:
• Wx WW   Patchy F
• Wx WW   Patchy F
• Wx LW  Widespread F
• Wx LW  Widespread F
• After Consolidating Sub-Phrases: -->
• Wx WW   Patchy F
• Wx LW    Widespread F

Test Case 3_3 F22  Sub-phrase consolidation Case 2

• Sky Area1 --> CLOUDY.
  
• SPW Area1  >> Consolidation (Chc T is not the same throughout period) >> CheckLocalEffects -->
  
• SPW WW >> Consolidation -->
  
• SPW WW for Chc T --> CHANCE OF THUNDERSTORMS IN THE MORNING
• SPW WW for Lkly RW --> SHOWERS LIKELY
• SPW LW >> Consolidation -->
  
• SPW LW for Chc T --> CHANCE OF THUNDERSTORMS IN THE MORNING
• SPW LW for Chc RW --> CHANCE OF SHOWERS IN THE AFTERNOON
• Wx Area1 >> CheckLocalEffects -->
• Wx WW --> removed by SPW WW
  
• Wx LW --> removed by SPW LW
  
• Pop Area1 >> CheckLocalEffects -->
• Pop WW --> CHANCE OF SHOWERS AND THUNDERSTORMS 70 PERCENT
• Pop LW --> CHANCE OF SHOWERS AND THUNDERSTORMS 40 PERCENT

• >> All these subPhrases are processed by ConsolidateSubPhrases which finds Case 2:
• SPW WW   CHANCE OF THUNDERSTORMS
• SPW LW    CHANCE OF THUNDERSTORMS
• After Consolidating Sub-Phrases: -->
• SPW Area1 Chc T --> CHANCE OF THUNDERSTORMS IN THE MORNING
• 
  
• SPW WW for Lkly RW --> SHOWERS LIKELY
• SPW LW for Chc RW --> CHANCE OF SHOWERS IN THE AFTERNOON

Test Case 3_3 F45 Need checkSkyPopWx to remove pop phrases for component as well as local effect area.

• Sky >> CheckLocalEffects -->
• Sky Windward >> removed by SPW Windward
  
• Sky Leeward >> removed by SPW Leeward
  
• SPW Area1  >> CheckLocalEffects -->
• SPW Windward  >> CLOUDY WITH A 20 PERCENT CHANCE OF SNOW WINDWARD...
• SPW Leeward      >> PARTLY SUNNY WITH A 30 PERCENT CHANCE OF RAIN LEEWARD
• Wx Area1 >> CheckLocalEffects -->
• Wx Windward >> removed by SPW Windward
  
• Wx Leeward >> removed by SPW Leeward
  
• PoP Area1 >> removed by SPW Windward and/or Leeward (which removes both the LE area and Area1).  This works because there is never a case where you have a local effect PoP reported AND you also want an Area1 PoP reported.
  

Design Changes Made from Original Local Effect Strategy to New

• "checkLocalEffects" method:  Change: "checkLocalEffects" must run after spawning new phrases.  We cannot assume that "checkLocalEffects" runs as the first phrase method. With phrases that might split into multiple phrases, we need to check for local effects AFTER phrases have been split, say, into separate sky and weather phrases or separate wind and wind gust phrases.
  
• "checkLocalEffects" was automatically added as the first phraseList method by ForecastNarrative.
• Now Phrase Definitions must  list "checkLocalEffects" explicitly. 
  
• "standard_phraseMethods"  (PhraseBuilder)
  
• "standard_vector_phraseMethods" (VectorRelatedPhrases)
  
• "standard_weather_phraseMethods" (WxPhrases)
  
• "skyPopWx_phrase" (CombinedPhrases)
  
• "sky_phrase" (ScalarPhrases)
• "fireSky_phrase", "trend_DayOrNight_phrase", "dayOrNight_phrase", "haines_phrase", "lal_phrase" (FirePhrases)
  
• Must carry over "localEffects" information when spawning new phrases in "splitPhrase" and "splitWxPhrases" (PhraseBuilder)
• Must apply "disabledSubkeys" when checking local effects in "checkThreshold", "checkSkyWxDifference" (PhraseBuilder)
  
• All local effects start out as conjunctive. Change: Make all local effects conjunctive when created then after all processing is done, convert to embedded if appropriate. ("checkLocalEffects" in PhraseBuilder and "makeLocalEffectNodes") We used to try and determine if a phrase could be embedded at the beginning of processing when we first discovered there was a local effect.  Later, we would find that some of the conjunctive phrases degenerated and could be turned into embedded. 
  
• "localEffect_hook" used to try and decide which phrases could be eliminated. When SPW became more complex, this job was not possible and caused duplicate and missing phrases.  Now we generate them all and remove duplicate information:
• In the "checkSkyPopWx" method
• Through consolidation in the Consolidate Sub Phrases step.
• Create subPhrase words.  Proceed as normal to create subPhrase words for local effect and other subPhrases.
• Consolidate SubPhrases:   At product Component Level:
  

• All Product components must list "consolidateSubPhrases" before "assemblePhrases" or "assembleIndentedPhrases"
• In all Narrative Standard products
• In FirePhrases compound phrases:  "skyWeather_byTimeRange_compoundPhrase", "fireWind_compoundPhrase", "fireValleyWind_compoundPhrase", "fireRidgeWind_compoundPhrase"
• In FWS_Overrides:  "fireEyeWind_compoundPhrase"
  

• Because consolidateSubPhrases waits until all subPhrases are finished AND executes before assembleSubPhrases (and assemblePhrases),  phrases can no longer wait for other phrases to completely finish.  They can, however, wait for all subphrases to finish.  Thus, "findWords" (PhraseBuilder) now just returns concatenated subPhrases instead of concatenated completed phrases.  All methods should use "findWords" rather than waiting for phrases to complete.  
  
• "extremeTemps_words" must be changed to use "findWords" OR the system will give "21 PASSES" message since it is waiting for the reportTrends_phrase to finish.
• Enhanced "checkRepeatingString" (PhraseBuilder) to handle local effect situations.
  
• Assemble Sub Phrases:  "assembleSubPhrases" (PhraseBuilder)  now needs consolidateSubPhrases_trigger to make sure that subPhrases have been consolidated before we assemble them.
• Assemble Phrases: "assemblePhrases", "assembleIndentedPhrases", "assembleSentences":
  
• Consolidate Local Effects:  Replace "combineConjunctiveLocalEffects" with "consolidateLocalEffects".      
  

• Calls "incorporateNonLocalEffectPhrases":
          #  Try to incorporate non-qualified phrases
          #    If there is exactly one leArea group in the set of phrases
          #      AND this group is composed of intersect areas
          #      AND there is more than one local effect phrase
          #      AND the number of non-local effect phrases
          #       < repeatingPhrase_localEffect_threshold:
          #         Convert them to conjunctive local effect phrases
          #         (one for each intersect local effect area)
          #        return 1
          #    Else:
          #        return 0
     
          #   EXAMPLE:
          #   Instead of:
          #     Chance of thunderstorms in the morning.
          #     Windward...Cloudy...Rain likely...Chance of precipitation 70 percent.
          #     Leeward...Partly cloudy...Scattered showers...Chance of precipitation 30
          #     percent. Highs in the 40s.  Winds 20 mph.
          #
          #   We will produce:
          #     Windward...Cloudy....Rain likely...Chance of thunderstorms in the morning...
          #     Chance of precipitation 70 percent.
          #     Leeward...Partly cloudy...Scattered showers...Chance of thunderstorms in
          #     the morning...Chance of precipitation 30 percent. Highs in the 40s.
          #     Winds 20 mph.


• If cannot incorporate, then calls  "convertToEmbedded":  (updated "makeEmbeddedFromConjunctive")
          #
          #   Converts conjunctive local effects to embedded if possible.
          #   For each leGroup:
          #      If number of possible embedded phrases
          #              < repeatingEmbedded_localEffect_threshold
          #         AND there are NO mandatory conjunctives:
          #             Replace conjunctive phrases with embedded phrase(s).

• Finally, calls "orderLocalEffectPhrases":  See Test Cases F8, F14, F15, F16, F22, F37, F38, F46
          #
          #   Group all conjunctive local effect phrases
          #      for each local effect area together
          #      (at the location of the first occurrence).
          #
          #   EXAMPLE:
          #     LEEWARD...SUNNY IN THE MORNING THEN BECOMING PARTLY SUNNY...SCATTERED SHOWERS.
          #     WINDWARD...MOSTLY CLOUDY WITH SCATTERED SHOWERS
          #  
          #   instead of:
          #     LEEWARD...SUNNY IN THE MORNING THEN BECOMING PARTLY SUNNY.
          #     WINDWARD...MOSTLY CLOUDY WITH SCATTERED SHOWERS.
          #     LEEWARD...SCATTERED SHOWERS.         #  

• Qualify conjunctive local effect phrases:  Modified "qualifyWords" to handle local effect qualifiers in this new design and to remove some bugs from previous design.  Added "addPeriod" argument to "sentence" (StringUtils) and removal of extra space from "combineSentences" (StringUtils).

•    New ConfigVariables: