The evaluation period goes from March 1, 2014 to August 23, 2014. To evaluate how my temperature forecasts have faired, I have compared them to forecasts by various sources: Environment Canada, The Weather Network, Intellicast, Accuweather, A Weather Moment and CBC Manitoba. I will not give the specific results of these sources, because I feel I should not do so if I do not have their permission. Instead, I will compare the results of my forecast with the average result across all sources.
First, the following table shows what percentage of temperature forecasts were PERFECT. For example, a forecast high of 33°C would be considered a perfect forecast if the actual high was between 32.5°C and 33.9°C.
Table 1: Percentage of temperature forecasts that were perfect (from March 1, 2014 to August 23, 2014)
My forecasts
|
Average of all sources
|
Range of all sources
|
|
Day
1 high
|
51%
|
41%
|
31-51%
|
Day
2 AM low
|
36%
|
34%
|
27-42%
|
Day
2 high
|
41%
|
35%
|
29-41%
|
Day
3 AM low
|
32%
|
28%
|
22-36%
|
Day
3 high
|
20%
|
23%
|
11-35%
|
Overall
|
36%
|
32%
|
26-36%
|
I am happy with the results. My forecasts had the highest percentage of perfect forecasts for both the day 1 and day 2 highs. May and July were the best months for the day 1 forecast with percentages of 58% ad 62% respectively. May was an exceptional month considering the average across all sources was just 32%. Interestingly, day 2 forecasts in July faired better than the day 1 forecasts. 66% of day 2 forecasts were perfect that month. Day 3 high forecasts continue to be my downfall, averaging slightly below average.
Overall, considering all the temperature forecasts from day 1 to day 3, July was the best month (47%) and March the worst (27%). This is to be expected because temperatures tend to be less predictable in the colder months versus the warmer months.
The last table below shows the percentage of forecasts that were WITHIN 1°C of the actual value. For example, a forecast high of 33°C would be considered within 1°C of actual if the actual high was between 31.5°C and 34.9°C.
Table 2: Percentage of temperature forecasts that were within 1°C of actual (from March 1, 2014 to August 23, 2014)
My forecasts
|
Average of all
sources
|
Range of all
sources
|
|
Day
1 high
|
81%
|
75%
|
64-81%
|
Day
2 AM low
|
66%
|
63%
|
56-67%
|
Day
2 high
|
67%
|
66%
|
57-70%
|
Day
3 AM low
|
59%
|
57%
|
49-62%
|
Day
3 high
|
55%
|
54%
|
40-69%
|
Overall
|
66%
|
63%
|
54-67%
|
Again, day 1 high forecasts continue to be my strength. July was the best month with 97% of day 1 forecasts being within 1°C of actual. March was the worst at 68%. Overall, all forecasts from day 1 day 3 were within 1°C of actual 66% of the time, above the average of 63% across all sources. July was my best month at 83% and May the worst at 53%.
I will continue to compare my forecasts with other forecasts for the foreseeable future. It has become a goal of mine to continue to remain above average and provide the best day 1 to day 3 temperature forecasts for Winnipeg. Unless interest is high, I don't plan to give another evaluation post like this one until next spring. Thanks for reading :)
In case anyone missed, here's this morning's weather update:
ReplyDeleteAnother cold one today. Lake-effect cloud and showers overnight helped keep our temperatures a couple degrees warmer than they could have dropped to. Some very chilly lows in southwestern Manitoba including Brandon at 3°C. All the way down to 1.6°C in Grandview and about 0.5°C at the Melita Manitoba Agriculture station. We'll likely remain under more cloud than sun today in Winnipeg with highs only around 17°C or so. Another chilly night tonight in the single digits.
Sun comes back for tomorrow and Thursday with highs of 23-24°C tomorrow and 27-28°C on Thursday. Note that the Canadian model is not as warm for Thursday. Cold front passes through on Friday with a chance of showers and perhaps a thunderstorm. That sunny forecast from EC is a little too optimistic. However, the front may pass through too early in the day for a decent storm chance. Models show it passing through already by midday, so the best risk for storms may end up to our east.
After that, all I can say is that conditions look quite variable, which is typical of a more fall-like pattern... Lots of ups and downs. Cooler for Saturday but warming back up for the beginning of the week. Possibly another cool down mid-week before warming up again.
Great job on your Winnipeg forecast, JJ! I can't believe how accurate your forecast is! No more looking at any forecasts except yours, Robs blog, a weather moments and occasionally EC's. Just curious, but where are you learning the techniques to get such an accurate forecast?
ReplyDeleteThanks!
I'd say when it comes to temperature forecasts, a lot of it comes down to experience, not just what I see the models are showing. Over time you get an idea of which models do better in certain situations and which do poorer in other situations... which helps decide which model may be better to trust at that given time. Also, if a particular model is having a bad week, I keep that into consideration when making my forecast. But yeah, overall, I'd say a lot of it comes down to experiences and looking at trends. This is the advantage of manual versus automatic computerized forecasts.
DeleteThanks again!
DeleteSo you don't really need to a heck of a lot about meteorology (not saying you don't!)for just temp forecasts, as a pose to something like predicting thunderstorm outbreaks or big precipitation events (witch you also do a great job on!:)?
Really sorry if I posted a million comments of the same thing. The comment was not displaying my computer until that last one at 12:31.
What I think Anon above is saying is you are a lot better than these computer generated forecasts but you might think twice before challenging them to a game of chess particularly a hundred simultaneous games.
DeleteYour comments about model situational reliability are very important as found by the hurricane forecasters. Jeff Masters blog going back to pre Katrina were very interested in keeping stats on the model performance. In fact isnt the european model a composite. Don
ReplyDelete