The Dominion Post in Wellington has run a secret accuracy trial over 30 days comparing forecasts from WeatherWatch.co.nz, MetService and NIWA. Both NIWA & MetService are based in Wellington, have multi millions in tax funding, access to all New Zealand data and dozens of meteorologists & scientists – yet WeatherWatch.co.nz has matched them accuracy-wise in this independent trial by The Dominion Post/Fairfax.
Full details – and reactions from WeatherWatch.co.nz, Metservice and NIWA – below…
When it comes to weather forecasts, it can pay to shop around.
To check if the washing will get wet, it might help to stop by the National Institute for Water and Atmospheric Research’s forecasts – whereas to discover how hot the day will be it could be best to try the MetService website, and how many blankets the bed requires overnight may be a WeatherWatch question.
The Dominion Post collated 30 days of weather forecasts from the three weather agencies, which all offer their predictions online. MetService and WeatherWatch also have smartphone apps, most of which are free.
The website forecasts for the three agencies – at two and four days ahead – were then compared for their accuracy to recorded weather information. Predictions for rainfall, maximum and minimum temperature were cross-checked.
For their short-range forecasts, all the agencies correctly predicted whether it would rain about two out of every three days.
But when it came to the most accurate, Niwa’s weather forecasts pipped the field, with a few more of their predictions matching the eventuating weather than rivals MetService and WeatherWatch.
Niwa’s weather graphics, launched online last year, are also slightly more conservative, offering only a 4 1/2-day outlook to WeatherWatch’s five and MetService’s nine days in advance.
Perhaps surprisingly, MetService’s long-range forecasts – at six days out – were more accurate than their short-range ones, with their computer- modelled forecasts picking the eventuating rain or shine on five out of every six days.
However, forecasting a rainy day appears to be the right call in the mid-June to mid-July period, with only eight dry days in the 30-day period.
Predicting the daily temperature high appears to be an easier task than picking the low. The three weather services are within one degree of the recorded maximum temperature for about two days out of three.
Here, MetService takes the crown for accuracy in the short- term, getting the day’s high within a degree more than 80 per cent of the time. Highlighting the challenges around forecasting, both Niwa’s and WeatherWatch’s four-day outlooks were ever-so- slightly more accurate than their two-day temperature outlooks.
Predicting the minimum temperature proved elusive, with all agencies’ estimates matching the recorded figures about half the time. But WeatherWatch and Niwa just edge out MetService in the accuracy stakes for estimating the day’s low. Forecasts for wind direction and speed vary in the details they give between MetService, Niwa and WeatherWatch and so were not compared.
Victoria University climate scientist and former forecaster James Renwick says there are two factors that can lead to vastly different weather predictions – the first is data about the climate right now.
“More effort goes into that step than anything else, because the closer you can get to the truth at the beginning of your forecast model, the closer to the truth the answer’s going to be,” he says.
“There’s a massive global infrastructure behind all this and billions of dollars are spent just observing the weather every day to get these starting conditions right.”
This data is then plugged into advanced computer models, which replicate how the atmosphere and the heat within it move around. There are a number of models available – and each might spit out slightly different results from the other.
Some agencies, such as Niwa, stick to one particular model, while others, including MetService, make their predictions after looking across several, Renwick says.
“Forecasting services then run these models dozens of times every day, changing the starting points just a little bit just to see how quickly the outcomes change.”
That’s why many forecasters talk in percentages about tomorrow’s weather, though the predictions are then simplified for public consumption.
Once the computers are done crunching the numbers, meteorologists then have the weather maps with their highs, lows and fronts, he says.
“That doesn’t mean you can accurately forecast rain will fall in Wellington. When you get down to really small scales, like individual towns in New Zealand, a lot of other factors come into play – like how strong the front is, how the mountains interact with the winds. We’re talking [accuracy] of two, three, four days max.”
A few decades ago, Kiwi meteorologists really struggled to get good information to plug into the models, with weather instruments so few and far between, Renwick says. “Twenty years ago, weather forecasts for the southern hemisphere were noticeably worse than they were in the northern hemisphere. But there’s been a lot of development since in satellite observations.
“Just because of chaotic effects, you really can’t get beyond about 10 days. Sometimes when there’s lots going on, the predictability may only last for four or five days.
“Probably the hardest thing for weather forecasters is when there’s nothing happening – not much wind around, but still the potential for a thunderstorm and localised flooding.”
In his old forecasting days, Renwick admits he took a slightly pessimistic take on what the models were telling him – and suspects many of today’s meteorologists do as well. “If there’s uncertainty, it’s better to say it’s going to rain and then have a fine day than to disappoint people.”
MetService – forecaster John Law:
The incredibly complex nature of weather means it’s very difficult for even the most skilled forecasters to get it 100 per cent accurate, particularly as you get further out in time. But our team of meteorologists work 24/7 to make them as accurate as possible. They are supported by constantly improving computer models from MetService’s forecast research team – we use a selection of the best global models as well as our own local models – and by the observation team’s hard work maintaining the network of weather stations and radars across New Zealand, so we get the most accurate information about what is happening right now.
There will always be areas of improvement to work on with weather forecasting, even for the most skilled meteorologists.
MetService verifies urban forecasts across 41 locations, for false alarms – rain being forecast and not occurring – and “probability of detection” – that rain was forecast and it happened.
In July 2014, false alarms were 28 per cent across the country, and probability of detection was 82 per cent.
Niwa – chief atmosphere, natural hazards and energy scientist Murray Poulter:
Niwa has been involved in weather forecasting for many years. It has the largest team of climate and weather scientists in New Zealand, a national climate station network, the national climate database and one of the most powerful supercomputers of its kind running world-class weather forecasting models over our complex terrain. Niwa uses sophisticated models run on our supercomputer producing “real time” highly detailed and precise localised forecasts.
Forecasting is a major global challenge, involving the atmosphere well beyond our shores and relies on dedicated collaborations with overseas agencies. The recent “Deepwave” experiment involving 100 international scientists and two research aircraft in the South Island is a good example of a collaboration aimed at further improving forecast accuracy.
Our forecasts will continue to improve as we invest in the necessary research and computing infrastructure, allowing us to make finer scale forecasts that are appropriate to New Zealand’s productive sectors.
WeatherWatch.co.nz – head analyst Philip Duncan:
WeatherWatch, with just two full-time forecasters, has managed to be as accurate – and in some cases more accurate – than both Niwa and MetService.
They have dozens of meteorologists and scientists and millions of dollars, including money from government funding.
This is huge for us as, unlike Niwa and MetService, we aren’t based in Wellington – they should have a much bigger advantage.
We don’t receive any funding from the government, no access to New Zealand data, yet here we are pretty much as accurate as them thanks to being fiercely passionate about accuracy and also thanks to the US government, who unlike the New Zealand government, believes weather data should be free for public good.
These results are fantastic for WeatherWatch.co.nz – but we are very motivated to get our accuracy up even further in Wellington and will do our best to do that as we head into 2015.
– The Dominion Post
on 11/08/2014 8:36am
What about Blue Skies Weather and Climate Services?
Maybe Tony’s weather forecasts should be included in the comparisons?
on 11/08/2014 10:18am
I believe they used only the main organisations that regularly send forecasts for free to the public and media. We had nothing to do with this trial and only found out about it when the results were sent to us last week!
on 10/08/2014 11:53pm
What observational data/material was uesd for this survey ? for eample showers can occur over a forecast area and miss the rainfall gauges, temperaures can vary over small distance. A 30 day survey is not long enough.
on 9/08/2014 11:36am
No one has mentioned MetVUW. I use those maps for trough rain especially in anticyclones, as they are often the best of the lot for where I live. They’re excellent for the Southerlies.
MetService forecasts have improved a lot where I live, and that is something to be grateful for. Kudos to NIWA who have been helping us out with this.
As for WeatherWatch, good on ya’s. Still No.1 with the social networking and look where everyone heads to when there is a storm raging away. It is always good to see what others are experiencing and I enjoy the photo’s, especially of big events. Keep up the good work you guys. WeatherWatch is also friendly towards the public, in fact you are the only public forecaster who is. I only have derogatory comments to make about the other two on that front, they’re still dinosaurs in the dark ages. Treating members of the public like they’re shit and refusing to respond to their queries is something I will never come to terms with. You need to be ready to do battle with your sword drawn before phoning those places.
The city might be one thing, but I think it’d be a very complex task to forecast the Wellington Region’s weather and get it right all of the time. You only have to look at the topography to know that. It has a lot of mountainous and orographic areas as well as the coastal and valley areas. Then there’s that strong gap wind. Throw in the influence of the Southern Alps and the Southern Ocean + Tasman Sea as well into the pot and it really boils away. Don’t we know it here, with wind that can blow from any direction at all.
I class WeatherWatch as one of the main forecasters and hope the day is not too distant when the others work in with you, properly. Weather stuff is paramount safety information and it should be freely disseminated for the public of NZ.
on 9/08/2014 3:43pm
Thanks heaps for your support and comments! With regards to MetVUW,they simply use American data only, no NZ data in those models (but they are very popoular – thats why we use the same data in our maps here at WW). The three listed in the news story are the only three forecasters in NZ who freely send information to the mainstream media nationwide on a regular basis. Other forecasters do not, or charge to do this.
on 9/08/2014 1:55am
A little funding would help. Give me WeatherWatch any day, no exaggerations and fluff.
on 9/08/2014 1:40am
Well done WW, the Dom’s trial here is in my opinion screwed from the start as they seemed totally ignorant of the facts that Phil made in his comments. These facts are huge when it comes to comparing like the Dom has done here. They should have made that stand out and asked the question why Government backed weather agencies are not, or only just, as good as the freelance WW team who have no Government funding whatsoever and absolutely nowhere near the staffing.
They really compared apples with pears with failing to mention these facts. The reporter also came over as biased to the Wellington based units and did not mention WW was based in Auckland.
So WW you can stand proud on this having really wiped the floor with the biggies. Well done..
on 9/08/2014 3:01am
To be fair this was a Wellington based newspaper trial about Wellington.
WW is based in Auckland but it doesn’t really matter where WW is based as you can get data anywhere you are based so thats not an issue really.
I follow the weather very closely and i remember on one occasion metservice forecast rain and WW at the same time forecast sun, i knew WW was wrong and mentioned it to them, within an hour they changed their forecast to reflect (basically) the metservice one.
My point is….enjoy all weather sites and make up your own mind.
I love them all and don’t like people saying that Niwa is crap or the metservice is crap or that WW is crap.
I think they enhance each other.
But i do agree with Phil that all data should be free like the US provides.