Model Details

As an avid hiker, undercast conditions (where the clouds are below you) are quite a treat. They’re also often unexpected. While the surprise can be fun, hiking in a white-out isn't as fun when you could have gone tree skiing instead. As such, and in an attempt to avoid doing actual work, I set out to predict undercast conditions in New Hampshire’s White Mountains.

I began this project assuming I could grab cloud base height information from existing weather reports (such as Windy), and use that to forecast undercast conditions. 10's of research papers, 1000's of lines of code, and more dead ends and stupid mistakes than I'd care to admit, here we are.

I quickly discovered that undercast conditions are far more complex than a simple temperature inversion. In the section below, I use data from six weather models (HRRR, RAP, NAM, GFS, ECMWF, and NBM), training three different machine learning models (Random Forest, Gradient Boosting, and XGBoost), after a simpler physics-based approach failed.

About Undercasts

While academic literature about undercast skies is sparse (it’s almost like climate scientists have something more important to study), there is a general consensus on what conditions are needed to produce undercast skies.

First and foremost, a temperature inversion is required, where temperature increases with height (rather than decreasing), trapping a cold layer of air at lower temperatures.

The following are good references on undercast conditions:

• ScienceDirect: Atmospheric study (S016980951930941X)
• Boundary-layer meteorology article (Springer)
• What is a cloud inversion? — EatSleepWild
• Mount Washington Observatory — Undercast
• Green Mountain Club — Temperature inversion & undercast

Determining if it’s undercast

To validate any model (ML or otherwise), we need to know if it was actually undercast. While I would prefer to just hike up Mt. Washington every day and collect data myself; I also would like to finish my PhD at some point, so we need a different solution.

Step 2 — Determine undercast conditions from the METAR report

Mount Washington is one of the few remaining staffed weather stations in the country. Every hour, a human, or cat, on the summit augments the station’s METAR report. These additional, man-made comments often contain information about cloud conditions. A key remark that appears in many undercast reports is TPS LWR — “tops lower” — which gives the coverage and the top height of any cloud deck sitting below the 6,288 ft summit.

An example report might read:

KMWN 211559Z 06010KT 70SM SCT/// OVC070 09/M06 RMK TPS LWR SCT040 SUN DMLY VSBL BINOVC

Hover or tab through any piece above to see what it means.

KMWN	Station. Mount Washington summit, 6,288 ft.
211559Z	Filed on the 21st at 15:59 UTC — just before local noon.
06010KT	Wind from 060° at 10 knots.
70SM	Visibility 70 statute miles. Likely estimated off distant landmarks. Automated stations stop at 10.
SCT///	Scattered layer, height unknown — the /// means the observer could not assign a height. This is not necessarily evidence the summit is in cloud.
OVC070	Overcast at 7,000 ft above the station — so ~13,300 ft above sea level. This is the lid overhead, and at 7,000 ft it is high enough to leave the view intact.
09/M06	Temperature 9 °C, dewpoint −6 °C.
RMK	Everything after this is the observer’s free text.
TPS LWR SCT040	The remark we care most about. “Tops lower”: a scattered deck whose tops reach 4,000 ft above sea level — 2,288 ft below the observer.
SUN DMLY VSBL	Sun dimly visible through the layer above.
BINOVC	“Breaks in overcast” — gaps in the lid.

This observation viewer shows 42 of these reports next to the webcam frames taken at the same time. The sample is not random, but instead has similar number of undercast (and non-undercast) days.

Thus, we want to determine which of the METAR remarks best correspond to undercast conditions, and will use the ~600 webcam-labeled dates to do so. That search produced five cuts/conditions:

Condition	Why
TPS LWR coverage is broken or overcast	At least 5/8 of the sky below the summit is cloud covered. "Scattered" cloud conditions didn't contain enough clouds to be considered undercast.
visibility > 40 SM	If visibility is too poor, then the summit is likely at least partially in the clouds.
no ceiling overhead, or one at or above 5,000 ft	A low lid overhead makes it more of a sandwich, than a true undercast. Is this "sandwich" is small enough, two layers blend together.
deck top at least 500 ft below the summit	If the clouds are too close to the summit, it no longer looks like an undercast, and you're mostly in the clouds.
lowest layer of any coverage overhead at or above 1,000 ft	The ceiling test only looks at broken and overcast layers, so it'll pass a report carrying FEW002 — cloud touching the summit. Even a few clouds directly at summit elevation can obscure a view.

What each cut does (individually)

These cuts were not the first thing I tried, nor were they generated randomly. The impacts of each cut are shown below.

Starting from “the observer mentioned a deck below the summit” and adding one condition at a time. Almost all of the recall is present at the start; the job of the cuts is to strip out false alarms without taking real days with them.

The impact of each cut individually (not after other are applied), is shown below.

Drop this one cut…	Extra false alarms	Extra real days caught	Precision	Verdict
broken or overcast coverage	+25	+4	0.38	Does the heavy lifting, but it is the one real trade — it costs four genuine undercast days.
lowest layer overhead ≥ 1,000 ft	+9	0	0.50	The best cut in the set. Removes nine false alarms and does not cost a single real day.
deck ≥ 500 ft below summit	+3	0	0.62	Modest but free.
visibility > 40 SM	+1	0	0.67	Nearly redundant here — the other cuts have already removed almost everything it would catch.
no low lid overhead	+1	0	0.67	Earns almost nothing on this sample. It stays because the sandwich case it guards against is real and these are only 22 days.

Scored against the full 589 hand-labeled days, the finished cuts have a precision of 0.70 and recall of 0.73. That is, when it says undercast it’s right about 70% of the time, and it catches about 73% of the real undercasts.

Those are numbers are determined by webcam interpretations, so it is worth looking at the actual webcam images. This gallery shows every labeled day that reported a deck below the summit, grouped by what the cuts did with it — what passes, what each cut throws out, and which of the discarded days were real undercasts after all.

Step 3 — Run the cuts over 29 years

We now have an adequate way of determining if it was undercast from METAR remarks. It's not great, but good enough, given we started with 22 undercast examples. Applying the undercast cuts to the 253,317 hourly observations from 1997 through 2026, 6,852 pass as undercast. That's over 300 times more than from manually labeling webcam images.

One caveat: the undercast rate is not constant over time (click to expand)

The fraction of METAR observations passing the cuts climbs over time. While it's possible that this is a rare positive consequence of anthropogenic climate change, it's probably just reporting bias. When we look at the rate of reports mentioning clouds decks TPS LWR, we see that the rate of reporting also increased. The data is noisy, given the low statistics, but the increase in undercast observations appears to be consistent with an increase in cloud deck reporting.

We have to be careful here, especially when training ML models. Without correction, a model would learn that recent years have more undercasts. The fix is in how the comparison cases are chosen — every undercast observation is matched against non-undercast observations drawn from the same year, same month, and same hour of day, so the calendar and the clock carry no signal and the weather parameters have to do the work.

Our final training set

Hand-labeling almost two years of webcam frames produced 589 days carrying 22 undercast examples. By contrast, using METAR observation remarks increased the record to 253,317 hourly observations carrying 6,852 — over 300× more positive examples, and enough that a classifier can see more than a handful of cases.

First Attempt: Simple Physics-based cuts

Scientific consensus is that undercast conditions are caused by temperature inversions. Initially, I figured it would be easy to query a couple of NOAA weather models, look for when temperature inversions were predicted, and voila, I have an undercast predictor. No fancy techniques required.

Turns out, predicting localized weather patterns is hard; especially in the mountains; especially on the most prominent peak for almost 1,000 miles; and especially on a mountain renowned for its harsh and unpredictable weather.

Based on my understanding of uncercast conditions, I came up with a couple conditions that I hoped would correlate strongly with undercasat conditions.

How they do

Scored on the base-rate holdout (a full unsampled year). In the sample, undercast occurs ~3% of the time:

While the physics-based approach works (it's about 3x better than random guessing), it's still not particularly useful. 3 times a small number is still a small number. Before abandoning this approach, we can investigate why it fails.

It becomes clear why these cuts fail when you look at the underlying distrubutions, separated by METAR-labeled undercast vs. non-undercast. These plots show the inversion strength, ceiling, visibility and boundary-layer depth for undercast hours against everything else, and lets you step through individual days model by model.

There are some clear patterns here (most notably the inversion plot); however the distributions overlap enough that no simple threshold can easily separate them.

So, you have some high-dimensional data with non-trivial correlations between parameters; let's throw some ML at it and see what sticks!

Developing the ML Model

Thanks to the METAR text, we now have a labeled dataset for training our machine learning model. Now we have to select parameters and a model.

Model selection & training

Three (XGBoost, Random Forest and Gradient Boosting) algorithms are trained independently for each of the seven weather model combinations above. You're only as good as your training data, and given the uncertainty in the undercast labeling technique, I'd have a hard time justifying any algorithms fancier than these vanilla-ML ones. Undercast conditions are rare (~3%), so the classes are heavily imbalanced. Rather than synthesizing samples, each one handles that with class weights (Random Forest’s balanced class weights, XGBoost’s scale_pos_weight, balanced sample weights for Gradient Boosting).

Two choices about how it is scored matter more than any of that.

Folds are grouped by week, and the holdouts are separated by date. Each observation is one hour and a day carries about 24 observations, so splitting rows at random lets a model learn a day’s weather from 03:50pm and be “tested” on 04:50pm. The two observations are clearly not independent. Grouping by date fixes that, but a multi-day inversion would make ajacent days correlated, so folds group by week instead, just to be safe. Any training row sharing a date with a holdout — plus a one-day buffer either side — is dropped outright, which costs about 17% of the training rows and leaves the holdouts as contiguous blocks of calendar time.

Negatives were subsampled 5:1 to build the training set., That thins the false alarms while keeping every true positive, so precision measured on the training set is inflated and means nothing. Each threshold is therefore fitted on the held-out year, which is unsampled, and the headline numbers come from the hand-labeled webcam days.

TODO: evaluate how changing the 5:1 sampling changes performance.

The importance of separating correlated observations (click to expand)

An initial analysis I performed showed ~0.8 precision. That number came from row leakage: folds were split by row rather than by date, so when a given forecast-hour row was scored, the other ~23 rows from the same day were sitting in the training set. The model didn’t have to forecast an undercast — it only had to recognize a day's conditions it had already memorized.

Below is that first-pass model evaluated both ways, each panel at its own F1-maximizing threshold. The top row is the leaky split, the bottom row is the date-grouped one. Reproducing the old split looks even better than the original writeup claimed — 0.9+ precision with nearly all undercast days caught — while the same model, same data, same algorithms scored honestly lands near 0.2 precision and misses most events. The old numbers were never forecast skill, just memorized days.

Note, this is the only data on this page still drawn from the first pass, so these numbers are not consistent with any other analysis in this writeup. The current pipeline groups folds by week and separates the holdouts by date, so the leaky split it illustrates cannot occur.

Model Results

ROC and precision–recall curves for the combined all-parameters model on the held-out year, one panel per algorithm. The three algorithms finish within 0.03 ROC-AUC of each other.

Top features

The top features for each model are shown below.

Note that this doesn't mean a ton nor is it particurarily interpretable, as as many parameters are, if not degenerate, strongly correlated. We investigate this in a later section.

Accuracy over Forecast Lead

A 1 hour forecast isn’t particularly useful; a 10 day forecast can’t be trusted. In the section below, I evaluate the performance of the model as a function of forecast lead. Each row in the testing holdout is a forecast made 1, 24, 48, 72, 96, 120 or 144 hours before the hour it describes, and the same model is used at all seven forecast lead points. Each lead gets its own F1-optimal threshold, fitted on the true-base-rate holdout.

All figures below are for Gradient Boosting. ROC-AUC is on the full unsampled holdout year.

Unsurprisingly, accuracy degrades with lead time. While ROC-AUC falls slowly, the precision drops quickly as lead time increases.

Two things are worth pulling out of the first table. Combining sources beats every individual one at every lead, and the margin widens with distance: at 1 h the best single source is within 0.05 AUC of the combination, by 120 h the gap is 0.09 even though only three models still reach that far.

Skill against forecast lead

As seen below, all metrics (save for precision) fall steadily over time.

Confusion matrices

Confusion matrices on the held-out year for each selected source, plus the 2-of-3 vote — a majority of the three hard calls, each made at its own threshold, not an average of their probabilities.

Pick a forecast lead to see how the same model degrades with distance. A single lead is scored at that lead’s F1-optimal threshold, while all leads” pools every row and uses the single global threshold.

Which ML model is best?

I chose to test three different ML models (and the consensus model), because it was cheap and it wasn't initially clear to me which model would have the best performance. As seen in the analysis above, the precision, recall, and ROC-AUC are pretty similar across all models.

Considering just the consensus model, an ensemble only pays when its members make different mistakes. If they agree, the vote can never overturn anything and all it does is average three copies of one opinion. So the first thing to look at is not the score but the agreement.

1. The model consensus vote does not help

On the combined model, at each member’s F1-optimal threshold:

Held-out year: 16,690 observations, 413 undercast. Webcam days: 4,122 observations, 154 undercast by human scoring.

It's pretty clear we're just playing with noise. All models have very similar performances, and there's no measurable benefit to any sort of ensemble.

2. Nothing else does better either

For kicks, I had ML try a couple other things as well. They all have similar performances. It seems pretty clear that fancier algorithms aren't providing any added benefit.

ROC-AUC on the held-out year. Histogram Gradient Boosting gets the raw columns with their gaps intact, since it splits on missingness natively; logistic regression gets median imputation and standardisation, because handing a linear model the −9999 sentinel the trees use would let one fill value swamp every coefficient.

3. The disagreement that matters is between weather models

However, the diffeerent weather models do disagree in different ways. I chose to investigate six different models, precicely because they have distinct physics and different resolutions. It makes sense that a 3 km regional and a global model at 0.25° will give different results, and provide new information.

The combined model reaches 0.908 [0.860, 0.951]. Against the six-way average that is +0.013 [+0.003, +0.026]; an improvement, but only slightly.

All models are effectively the same

There's no benefit to a vote, ensemble, or comparing different models from here on out. A single combined-source model with per-lead thresholds is as good as any ensemble. From here on out, I use Gradient Boosting.

Are we overfitting?

The combined model reads 213 feature columns, and the training split holds 31,009 rows. However, not all features or rows are independent.

Rows are not observations

Each undercast hour enters the training set once per forecast lead, but multiple observations from the same undercast event and not indipentent. They're not the same data point either though. Assuming the points are fully degenerate, we find our actual training set is reduces as follows:

Against 213 feature columns, the usual rule of thumb of ten events per parameter is off by more than an order of magnitude. We have more features than undercast days! However, feature columns are also strongly correlated and contain some degeneracies.

Some features are strongly correlated

213 columns is not 213 independent knobs. The combined model is six weather models’ columns stacked side by side, meaning a parameter such as the temperature at 500mb tmp_500mb, has six different versions: tmp_500mb_hrrr, tmp_500mb_gfs, tmp_500mb_nam, tmp_500mb_rap and tmp_500mb_ecmwf.

Clearly all six versions are going to be highly correlated, though some models perform better than others, which is why the combined model does better. Removing all duplicate parameters(across models), the 213 columns collapse to 78 distinct physical quantities, plus four calendar encodings (month and hour as sine/cosine). Forty-one of those quantities are carried by more than one source and the remaining thirty-seven are published by a single model.

When evaluating the correlation between same-quantity pairs, we find that the median is 0.872, against 0.181 for a randomly drawn pair of different quantities.

All 300 same-quantity, different-source pairs, binned by |Spearman r|. The dashed outline is the baseline: 19,317 randomly drawn pairs of different quantities, shown as a share of its own distribution so the two are comparable.

Performing principal component analysis on the on the 11,620 training rows where every source reports, over the 98 columns populated more than half the time, 29 components carry 95% of the variance and 21 carry 90%; the participation ratio is 8.7. There's more analysis that could be done here on parameter degeneracy, and which components are most important, but for now it seems the risk of over-training our model is low.

As seen in the two plots below, there's no clear benefit to trimming the model parameters or limitting the tree depth.

Each budget is the top k features by importance, retrained and scored on both untouched holdouts, plotted as change against the full 213-feature model. The grey band is the noise floor — ±0.005, the spread of the current configuration under five seeds.

Same construction as the chart above: change against the current configuration, with the ±0.005 reseed band.

Final Model

Combining everything, the highest-performing model is:

• Uses data from all weather models, not just one.
• Uses Gradient Boosting, since an ensemble does not improve performance.
• Keeps all features and trims it to a depth of 3; see overfitting and capacity checks.

Confusion matrices are shown below for each lead.

Held-out year, true 2.5% base rate, each lead at its own threshold.

Next Steps

You're only as good as your data, and our data is adequate at best. The model would benefit from a larger, and more accurate dataset. Any sort of additional data that could help with identifying historical undercast conditions would go a long way.

Might also be interesting to look into parameter importance more, and understand why different models perform better/worse, and what primary factors drive undercast conditions.

AI usage

The vast majority of the coding was done with AI. I made all high-level model design decisions and reviewed all analysis steps made by AI. AI was particularly helpful when it came to ML implementations and web visualizations, while it struggled with physics/weather interpretations and higher-level analysis and reasoning. The vast majority of the text on this page was written by me, with small sections of AI generated text (eg. for an overview of ML models).

Data availability

All scripts used for this analysis are available on GitHub. For access to the data, or with any questions, contact me at ehingerl@mit.edu.