Model Details
As an avid hiker, undercast conditions (where the clouds are below you) are quite a treat. They’re also often unexpected. While the surprise can be fun, I also enjoy knowing whether I can expect nice views, or should go tree skiing instead. As such, I set out to predict undercast conditions in New Hampshire’s White Mountains.
I began this project assuming I could grab cloud base height information from existing weather reports (such as Windy), and use that to forecast undercast conditions. When that failed to have the desired predictive power, I queried weather forecast data from a variety of sources, specifically looking for inversions. I quickly discovered that undercast conditions are far more complex than a simple temperature inversion. As a result, I arrived at the models described below: an ensemble of three machine-learning algorithms, trained on forecasts from six weather models (HRRR, RAP, NAM, GFS, ECMWF, and NBM) along with undercast conditions labeled from the Mt. Washington webcam. While it works better than the previous methods, more data is needed for training.
About Undercasts
While academic literature about undercast skies is sparse (it’s almost like climate scientists have something more important to study), there is a general consensus on what conditions are needed to produce undercast skies.
First and foremost, a temperature inversion is required, where temperature increases with height (rather than decreasing), trapping a cold layer of air at lower temperatures.
The following are good references on undercast conditions:
- ScienceDirect: Atmospheric study (S016980951930941X)
- Boundary-layer meteorology article (Springer)
- What is a cloud inversion? — EatSleepWild
- Mount Washington Observatory — Undercast
- Green Mountain Club — Temperature inversion & undercast
Determining if it’s undercast
To validate any model (ML or otherwise), we need to know if it was undercast. While hiking every day in the White Mountains is the optimal way to empirically determine undercast conditions; I also need to finish my PhD at some point. As such, we use the Mt. Washington Observatory webcam timelapse to determine whether each day had undercast conditions. Labeling timelapse photos was done manually, and not fun.
Note that the webcam timelapse has data from the past 2 years, with one image per day (more on this later).
The Model
You can play around with a couple different examples of undercast, clear, and cloudy days in the region below (link), but no obvious relation jumps out. Notably, when conditions are undercast, forecast models do not predict inversions. That’s not to say that inversions don’t exist; just that global weather models fail to predict them, or that they are small enough (in vertical extent) to not be visible.
With no obvious correlation between undercast conditions and model predictions, I figured this would be a good tool for machine learning.
Input parameters (click to expand)
Weather models such as HRRR, RAP, NAM, GFS, ECMWF, and NBM expose hundreds of parameters. Not all of these are relevant to undercast predictions. The following are used (each is pulled from every model that provides it). They were selected based on what a dumb physicist thought seemed relevant. Parameter importance (predictive power) is shown under model results
| Key | Description |
|---|---|
| cloud_top | Cloud top height (AGL). The altitude of the highest extent of a cloud layer, useful for identifying deep convection or storm tops. |
| boundary_layer_cloud_layer | Boundary layer cloud cover (%). Total cloud fraction within the planetary boundary layer, the lowest ~1–2 km of the atmosphere. |
| low_cloud_layer_percent | Low cloud cover (%). Fractional cloud coverage below ~2 km (~6,500 ft), typically stratus or stratocumulus. |
| middle_cloud_layer_percent | Mid-level cloud cover (%). Fractional cloud coverage between ~2–6 km (~6,500–20,000 ft), typically altocumulus or altostratus. |
| high_cloud_layer_percent | High cloud cover (%). Fractional cloud coverage above ~6 km (~20,000 ft), typically cirrus or cirrostratus. |
| cloud_ceiling_m | Cloud ceiling height (meters AGL). The height of the lowest broken or overcast cloud layer, critical for aviation and visibility assessment. |
| cloud_base_m | Cloud base height (meters AGL). The altitude of the bottom of the lowest cloud layer, closely related to the Lifted Condensation Level (LCL). |
| cloud_top_pres | Cloud top pressure (Pa). The atmospheric pressure at the top of the cloud layer — lower pressure = higher altitude cloud tops. |
| cloud_base_pres | Cloud base pressure (Pa). The atmospheric pressure at the base of the cloud layer, an alternative vertical coordinate to height. |
| cloud_top_hgt | Cloud top geopotential height (meters). Similar to cloud_top but expressed as geopotential height rather than AGL, more consistent across terrain. |
| wind_10m_day_max | Daily maximum 10m wind speed (m/s). The peak wind speed at 10 meters above ground over the forecast day, useful for fire weather and wind advisories. |
| tmp_500mb | Temperature at 500 mb (~5,500 m). Mid-troposphere temperature, used to assess atmospheric instability and identify cold upper-level troughs. |
| tmp_700mb | Temperature at 700 mb (~3,000 m). Lower-mid troposphere temperature, important for identifying warm/cold advection and orographic effects. |
| tmp_850mb | Temperature at 850 mb (~1,500 m). Low-level temperature commonly used as a proxy for surface airmass character and snow level estimation. |
| tmp_925mb | Temperature at 925 mb (~750 m). Near-surface temperature just above the boundary layer, useful for detecting inversions and cold air damming. |
| tmp_1000mb | Temperature at 1000 mb (~100 m). Near sea-level temperature, representing conditions very close to the surface in low-elevation areas. |
| tmp_2m | 2-meter air temperature (°C). Standard screen-level temperature used in most surface weather forecasts and human comfort assessments. |
| rh_2m | 2-meter relative humidity (%). Moisture content of the air at screen level, directly relevant to fire weather, fog formation, and human comfort. |
| hpbl_surface | Planetary boundary layer height (meters). The depth of the turbulent mixing layer above the surface — higher in the afternoon, lower at night and during inversions. |
| hgt_0C_iso | Height of the 0°C isotherm (meters). The freezing level altitude, critical for determining rain/snow transitions and icing conditions. |
| vis_surface | Surface visibility (meters). Forecast horizontal visibility at the surface, reduced by fog, precipitation, smoke, or dust. |
| prate_surface | Instantaneous precipitation rate (kg/m²/s). The rate of precipitation at a given forecast hour, useful for identifying intensity of ongoing precip. |
| apcp_surface | Accumulated total precipitation (kg/m²). Cumulative precipitation over the forecast period, equivalent to mm of liquid-equivalent rainfall. |
The forecast models
Undercast is predicted from six independent weather models. Each one gets its own dedicated ensemble, and a seventh combined model draws on the parameters of all six at once — so the sources can be compared head-to-head, and the combined model can lean on whichever happens to be most informative.
| Model | What it is | Scale / resolution | Cloud fields |
|---|---|---|---|
| HRRR | High-Resolution Rapid Refresh | 3 km · CONUS · short-range | Yes |
| RAP | Rapid Refresh (HRRR's parent model) | 13 km · regional | Yes |
| NAM | North American Mesoscale | 12 km · regional | Yes |
| GFS | Global Forecast System | ~0.25° · global medium-range | Yes |
| ECMWF | Integrated Forecasting System (open data) | ~0.25° · global | Dynamics / thermodynamics only |
| NBM | National Blend of Models (statistical blend) | CONUS blend | Yes — incl. probabilistic ceiling & visibility |
| Combined | Every model's parameters together | — | — |
Model selection & training
We don't have a lot of data to train on, so I only consider relatively simple ML models. To begin, I trained four different classification models—Logistic Regression, Random Forest, XGBoost, and Gradient Boosting. Logistic Regression performed significantly worse, so the other three are kept and studied further. This trio is trained independently for each of the seven models above.
Undercast days are rare, so the classes are heavily imbalanced. Rather than synthesizing samples, each model handles this with class weights (Random Forest's balanced class weights, XGBoost's scale_pos_weight, and balanced sample weights for Gradient Boosting), which suits tree ensembles and keeps the pipeline clean.
Because each date contributes many forecast-hour rows, every cross-validation fold is grouped by date, so no single day's rows land in both training and scoring — otherwise the model would "see the answer" for a day it is being tested on. (An earlier version split rows randomly and looked far more accurate than it really was.) Performance is measured with five-fold grouped cross-validation, and each model's decision threshold is chosen on the pooled out-of-fold predictions rather than a single split: undercast is so rare that any one validation fold holds only a day or two of it. The threshold targets the most recall it can while holding precision above a floor, so the deployed model leans toward staying quiet unless the signal is reasonably strong.
How well does it work? The three algorithms all rank undercast days well — ROC-AUC sits around 0.80–0.87 across the seven data sources (NBM and the combined all-parameters model are strongest). Precision is the hard part: because undercast is only about 4% of days, even good ranking runs into a precision ceiling near 0.25, and the deployed majority vote (below) runs roughly 0.2 precision. In plain terms, when it does call an undercast it's right maybe one time in five, and it catches only a minority of real undercast days.
What the row-leakage bug was hiding (click to expand)
An earlier writeup here implied ~0.8 precision. That number came from the row-leakage bug mentioned above: folds were split by row rather than by date, so when a given forecast-hour row was scored, the other ~22 rows from the same day were sitting in the training set. The model didn't have to forecast an undercast — it only had to recognize a day it had already memorized.
Below is the same all-parameters model evaluated both ways, each panel at its own F1-maximizing threshold. The top row is the leaky split, the bottom row is the honest date-grouped split. Reproducing the old split here looks even better than the original writeup claimed — 0.9+ precision with nearly all undercast days caught — while the same model, same data, same algorithms scored honestly lands near 0.2 precision and misses most events. That collapse is the whole reason the deployed thresholds now aim for ~0.2: the old numbers were never forecast skill, just memorized days.
Top features
This starts on the combined all-parameters model; tick additional weather models to see which of their own parameters carry the signal, or to line several up side by side.
Most important features for each selected XGBoost model. For the all-parameters model, physically sensible signals dominate — 925 mb temperature, boundary-layer mixing height, cloud-top height, and the freezing level. See plot_undercast_models.py for generation details.
Results
No single model is used on its own. The deployed prediction is a majority vote — a day is called undercast only when at least two of the three algorithms agree at their chosen thresholds. Requiring agreement trims false alarms at the cost of some recall, which is the right trade for a page you'd check before driving to a trailhead.
Out-of-fold confusion matrices for each selected model at its deployed thresholds, plus the 2-of-3 consensus. (The site itself runs the all-parameters consensus; the individual weather models are shown for comparison.) Undercast is the rare bottom row: the vote keeps false positives low but still misses a fair share of real events — an honest reflection of a hard, data-limited problem.
Per-source numbers
The figures above show one source at a time. To see the exact grouped, out-of-fold metrics for any individual data source — or to line several up side by side — tick the sources below. Precision, recall, F1 and the false-positive / false-negative counts are at each algorithm's deployed threshold; ROC-AUC and PR-AUC are threshold-free.
Next Steps
The model performance is adequate at best. While there are a number of fancy ML techniques that can be used to squeeze a few percent of accuracy, the main limitation for the model is the quality and quantity of the available data. From the Mt. Washington webcam, there are only a handful of days with clear-cut undercast conditions, and with a single image per day spanning just two years, the labeled dataset is both small and coarse. As such, the highest-leverage next step isn't a fancier algorithm — it's better data: more years of observations, several images per day to pin down timing, and a few additional summit webcams to widen the net. While (some of) these exist, I haven’t taken the time to manually analyze each image, and thus the model remains sub-par for the time being. If you want to know if it’s undercast, you’ll have to hike up yourself.