The National Operational Model Archive and Distribution System (NOMADS) system is the site which distributes most of the weather data generated at NOAA. NOMADS has been under strain, and the peole who run the site are making changes, which they hope will allow for continued smooth operations.

Many of the weather models offered by LuckGrib are obtained from NOMADS, including: GFS, HRRR, NAM, GEFS, NBM, RAP, CFS, RTOFS and NCOM.

A service change notice was published which details the changes. There was a meeting held many months ago which interested parties were able to attend. I attended the meeting and have been able to make changes on the server which minimize the impact of this change.

You may notice that some of the models now have a reduced set of parameters available. This is in an effort that LuckGrib continues operating smoothly. I’ll monitor how these changes play out over time.

Model Changes

The timing of the model downloads will be in flux for a little while. It may take several days after this change is introduced, to monitor and make adjustments to maintain timely updates.

The RAP model domain covering Alaska has been removed. This model received very little use, and Alaska is well covered by the HRRR, NAM and NBM models which are available.

CMC, the Canadian global ensemble, has had paramters removed. Of the two global ensemble models, GEFS has proven to be much more popular. CMC no longer has the following: all of its 850mb data, relative humidity @ 2m, along with CAPE and CIN.

The GFS ensemble no longer has: relative humidity @ 2m, temperature, relative humidity and vertical velocity @ 850mb, precipitable water.

The deprecated WW3 model has been further degraded. If you are still using this model, you should switch to the global ocean wave data in GFS / Ocean, which is the new location of the WW3 data, as described in the GFS v16 update article.

Background

NOMADS operates a publically accessible suite of servers which make the entire collection of weather data (GRIB files) freely available worldwide. Over time, the demand for this data has grown, as has the amount of data being generated.

The architecture of the system is that NOMADS receives its data from the various super computer centers around the USA. These super computers upload the weather data they generate to NOMADS through the same network channel that is available to everybody else - the public and the super computer centers share bandwidth.

At some times of day, this network channel is at capacity. Over time, the bandwidth required for the super computer centers to upload their data has increased. For example, the recent GFS upgrade considerably increased the size of the GFS data sets. This trend, where weather models generate more data over time, will continue.

The staff running NOMADS acknowledge that the proper solution to this capacity problem is that a second network connection be created, which the super computer centers can use to upload their data. This would create an architecture where the public downloads are on one connection, and the internal use of the system is on another. During the meeting, the NOMADS staff presented the second network interface as the ideal solution to this problem.

However, creating a second network connection apparently costs roughly $1.5 million dollars (as a one time cost, operation was presented as being negligible,) and it appears the US government has not prioritized this project and the money is not available. $1.5 million.

Throttling

The apporach being taken to mitigate the problem of oversubscribed bandwidth at NOMADS is to reduce the number of hits per minute each client to NOMADS is allowed to issue. The LuckGrib servers need to issue over 2,700 hits to NOMADS to obtain the GFS data that is required in order to make it available through the app. The new limit at NOMADS is 120 hits per minute.

(Note that while 2,700 hits to obtain the GFS data may seem like a lot, only roughly 2 GBytes of data is downloaded - exactly and only the data that is needed by the LG service. There is an alternate method that would require just over 120 hits, in total, however that approach would download over 60 GBytes of data, 2 of which is required and the rest of which would be thrown away during processing. The first, now slower method has less impact on this shared public resource.)

As a result of this new throttling, the model data at LuckGrib will be slightly delayed from when it was available before this change - by roughly 20 minutes for GFS. This delay varies according to the model. HRRR is available a few minutes later than it used to be, same with NAM and NBM.

Conclusion

Hopefully the folks at NOAA and NOMADS are able to receive some additional funding which will allow them to implement the network upgrades which are needed.