Author Topic: NWS Forecast Problems?  (Read 9830 times)

xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
NWS Forecast Problems?
« on: June 24, 2017, 05:14:20 PM »
Whenever I have problems, I assume the first cause is me. However, I'm simply asking if anyone else has had problems the last few days with the NWS Forecasts. I am mainly getting:
Quote
CLIENT ERROR
HTTP Error: Unsupported HTTP response status 504 Gateway Time-out (
 Essential data like temperature is mssing - program stops

The basic url I have been using for the last several months is:

https://graphical.weather.gov/xml/SOAP_server/ndfdXMLclient.php...location and other details

Perhaps the NWS has not only changed to security but also the domain? Or, as is usually the case, I've just screwed up something...

Gotta mow the back yard, just got a new mower! I'll research the "Gateway" time-out meaning if it's actually not what it appears to literally mean.

Additional info via Network Utility:
Loopup: graphical.weather.gov -> (e278.dscg.akamaiedge.net) ->23.36.68.92

Ping: PING 23.36.68.92 (23.36.68.92): 56 data bytes
64 bytes from 23.36.68.92: icmp_seq=0 ttl=55 time=83.870 ms
64 bytes from 23.36.68.92: icmp_seq=1 ttl=55 time=32.479 ms
...
64 bytes from 23.36.68.92: icmp_seq=9 ttl=55 time=32.187 ms
--- 23.36.68.92 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss

Traceroute: traceroute to 23.36.68.92 (23.36.68.92), 64 hops max, 72 byte packets
 1  10.0.1.1 (10.0.1.1)  0.469 ms  0.264 ms  0.169 ms
 2  10.0.0.1 (10.0.0.1)  1.534 ms  1.521 ms  1.779 ms
 3  www.xxx.yyy.zzz (www.xxx.yyy.zzz)  15.837 ms  14.290 ms  13.748 ms
 4  te5-3-ur01.mycity.ST.malt.comcast.net (www.xxx.yyy.zzz)  16.154 ms  14.375 ms  14.858 ms
 5  96.108.172.117 (96.108.172.117)  15.113 ms  14.313 ms  13.208 ms
 6  te-0-0-0-0-ur04.hattiesburg.ms.malt.comcast.net (68.85.203.21)  16.745 ms  14.327 ms  14.956 ms
 7  be-22258-cr02.dallas.tx.ibone.comcast.net (68.86.93.93)  45.036 ms  24.263 ms  24.890 ms
 8  be-11424-cr02.56marietta.ga.ibone.comcast.net (68.86.85.21)  35.018 ms  34.329 ms  34.893 ms
 9  hu-0-11-0-5-pe01.56marietta.ga.ibone.comcast.net (68.86.88.222)  32.728 ms  34.278 ms  34.929 ms
10  * * *
11  a23-36-68-92.deploy.static.akamaitechnologies.com (23.36.68.92)  33.930 ms  34.281 ms  35.090 ms

I think, with my minimal "knowledge" that this is saying the problem is with the "SOAP server". I think the "***" means there are three 'timeouts' or delays before the last server (deploy.static.akamaitechnologies.com) responds. ??? Is there a place/site that reports NWS server status? Direct Forecast access seems fine.
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
More than one way to skin a catfish (Re: NWS Forecast Problems?)
« Reply #1 on: June 24, 2017, 10:44:05 PM »
Dear X-Air and WeatherCat cyber weather data miners,

Whenever I have problems, I assume the first cause is me. However, I'm simply asking if anyone else has had problems the last few days with the NWS Forecasts. I am mainly getting:
Quote
CLIENT ERROR
HTTP Error: Unsupported HTTP response status 504 Gateway Time-out (
 Essential data like temperature is mssing - program stops

It is possible that the problem isn't the server you are using graphical.weather.gov, but another server that is being queried in the process of generating the forecast.

The basic url I have been using for the last several months is:

https://graphical.weather.gov/xml/SOAP_server/ndfdXMLclient.php...location and other details

For whatever it's worth, this is the forecast URL I've been using for years:

Code: [Select]
https://forecast.weather.gov/MapClick.php?lat=37.8511&lon=-122.15472&site=mtr&smap=1&unit=0&lg=en&FcstType=text
It works fine but will probably break when NWS finally succeeds in launching their new web forecast API.  The parameters appear to be reasonably obvious

Gotta mow the back yard, just got a new mower!

Well, I've still got my old mower, but I just got back from moving our yard.  The 10 day heat-wave definitely got the grass growing!

Cheers, Edouard  [cheers1]

xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: NWS Forecast Problems?
« Reply #2 on: June 24, 2017, 11:43:54 PM »
The "graphical" forecast is actually used for the multi-day, icon filled 'graphics-based' forecast, it's the upper part of my Forecast page. I think the one you use may be for the text display of the Forecast? The Summary forecast (Midnight to Noon and Noon to Midnight) actually always still works. By the way, I'm using the Leuven-NOAA stand-alone PHP scripts.

It is now working, of course... I'll upload a couple of screen shots after dinner. ;)

The site is now reporting the "Time-out" again. [rolleyes2]
« Last Edit: May 09, 2018, 07:39:32 PM by xairbusdriver »
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


wvdkuil

  • Strong Breeze
  • ***
  • Posts: 151
    • DW8113 (D8113)
    • IVLAAMSG47
    • Weatherstation Wilsele/Leuven/Belgium
  • Station Details: VantagePro2+ UV,Solar - Macbook/Mini
Re: NWS Forecast Problems?
« Reply #3 on: June 25, 2017, 12:13:42 PM »
Since last Thursday until now 2017-06-25 maybe 50% or more of the  graphical.weather.gov/xml/ server-requests end in that "gateway time-out" error
The server graphical.weather.gov is reachable normally but the back-end server fails.

This link https://graphical.weather.gov/xml/SOAP_server/ndfdXML.htm is the "human" interface to get the same xml data as the scripts do.
You click a few variables you want and press submit at the bottom.
Those requests fail also with the same frequency as the scripts.

Hope they will see the errors themselves and someone will do something about it shortly.

Wim
Support of Leuven-Template&Scripts: https://support.leuven-template.eu/
Support and demo for the PWS_Dashboard (also for WeatherCat)  https://pwsdashboard.com/ 
My PWS at home https://weer.sluispark.be/
And at my daughters house https://www.weerstation-herent.be/

xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: NWS Forecast Problems?
« Reply #4 on: June 25, 2017, 04:13:17 PM »
Thanks so much, Wim. I knew it wasn't your template, hat's why I hadn't contacted you. I still have your error reporting code enabled otherwise I would have understood even less! I really appreciate your confirming my suspicions and especially that I haven't screwed up something on my end. I haven't even touched the code since changing the "http" to "https" a couple of months ago. [computer]

Quote
Hope they will see the errors themselves and someone will do something about it shortly.
I did a minimalist search for a 'contact' link at the NWS site that might be appropriate, but I assumed they would be getting reports from commercial users by now. Perhaps the majority of heir users are hobbyists?! [banghead] I hope it doesn't indicate a problem with funding or lack of manpower at the organization! I hear reports that NOAA is getting negative comments about replacing their weather satellites. [rolleyes2]
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Speaking of which . . . . (Re: NWS Forecast Problems?)
« Reply #5 on: June 26, 2017, 11:36:49 PM »
Dear X-Air and WeatherCat web-based weather information seekers,

Now that you mention it, even the URL I use to get a forecast is having sporadic "issuez."  For example, It just generated the following error message:

Quote
Service Unavailable - Zero size object
The server is temporarily unable to service your request. Please try again later.

Reference #15.f532c517.1498516287.69c8395

Retrying after waiting a bit works fine, but it is new and does happen as often as once a day or more.  Perhaps the National Weather Service is doing some server upgrades associated with their attempt to launch their new API.  As a result, various services can be temporarily unavailable.

Cheers, Edouard  [cheers1]

xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: NWS Forecast Problems?
« Reply #6 on: June 27, 2017, 01:57:00 AM »
Quote
As a result, various services can be temporarily unavailable.
My opinion, also. I may just take down the Forecast page or create a new one with a usable text display. I hate to lose the Summary and Detailed versions, but they are also (I think) generated by the SOAP XML data.

Makes me wonder what are others are using for the NWS Forecast? No one else here seems to be affected! :o
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


wvdkuil

  • Strong Breeze
  • ***
  • Posts: 151
    • DW8113 (D8113)
    • IVLAAMSG47
    • Weatherstation Wilsele/Leuven/Belgium
  • Station Details: VantagePro2+ UV,Solar - Macbook/Mini
Re: NWS Forecast Problems?
« Reply #7 on: June 27, 2017, 05:59:02 AM »
Quote
As a result, various services can be temporarily unavailable.
My opinion, also. I may just take down the Forecast page or create a new one with a usable text display. I hate to lose the Summary and Detailed versions, but they are also (I think) generated by the SOAP XML data.

Makes me wonder what are others are using for the NWS Forecast? No one else here seems to be affected! :o
ALL NWS scripts are affected.  A small list of changes below:

The point-forecast => https://forecast.weather.gov/MapClick.php?lat=37.8511&lon=-122.15472&site=mtr&smap=1&unit=0&lg=en&FcstType=text

That is a 1 page forecast  with text and icons is affected as the whole architecture behind the scened will be changed. The cutover to the new format was scheduled for April, postponed to June a.s.o., is now scheduled for August. It is used in the Leuven scripts also as "plain-page forecast

The new structure needs  3 or 4 calls to the servers to get at the data, as in the past one was needed.  That will increase the number of errors, I think.

Icons: This and last year, there have been already multiple changes in the icon-structure and location of the icons.  NWS totally changed the icon structure when they introduced multipart icons with the .php extension => two six hour icons "glued together"

http => https  All links will be changed for that change

URL When doing all this the location (URL) is changing also

SOAP This source of information is not heavily used by scriptwriters, I think. 
https://graphical.weather.gov/xml/SOAP_server/ndfdXML.htm
But inside that wealth of information there are links for icons and alerts which has to change also.

Alerts The URL, location and structure are changing also. So those scripts, example the "Curly" scripts are and will  be adapted in coming months.

Other forums at least a dozen on wx and also some on ww:
http://www.wxforum.net/index.php?topic=32341.msg326185#msg326185
http://www.wxforum.net/index.php?topic=32287.msg325627#msg325627
http://www.wxforum.net/index.php?topic=31954.msg321304#msg321304

Wim
Support of Leuven-Template&Scripts: https://support.leuven-template.eu/
Support and demo for the PWS_Dashboard (also for WeatherCat)  https://pwsdashboard.com/ 
My PWS at home https://weer.sluispark.be/
And at my daughters house https://www.weerstation-herent.be/

xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: NWS Forecast Problems?
« Reply #8 on: June 27, 2017, 09:24:54 AM »
Thanks for the thorough and complete explanation, Wim! I must say I've usually skipped over the Graphical part of the display and mainly looked at the Summary table. If I wanted a more 'definite' time period in a day, the Detailed display was nice.

Of course, weather "forecasts", at least in the US, are not as accurate as our TV weather people imply. These changes don't help with the lack of apparent accuracy. I suspect the changes are more difficult than expected, also.

I've got lots of work to do in cleaning up my PHP to keep me busy for the next few weeks! I've ended up with severe code bloat! I think ~95% of it can be converted to functions. But we know how accurate 'forecasts' are!i  cmu:-)
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Thanks Wim for the detailed info (Re: NWS Forecast Problems?)
« Reply #9 on: June 27, 2017, 11:41:56 PM »
Dear Wim, X-Air and WeatherCat coders,

Thanks for the thorough and complete explanation, Wim!

Indeed thanks.  The updates were rather as I suspected, but I appreciate the detailed summery.

Cheers, Edouard  [cheers1]

xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: NWS Forecast Problems?
« Reply #10 on: June 27, 2017, 11:45:43 PM »
This gets me as close to the basic display as I can get using: https://forecast-v3.weather.gov/point/35.0702,-89.7363?mode=widget Better than nothing, I guess. [rolleyes2] At least it doesn't have the massive footer as in the MapClick version.

Of course, the NWS site seems to be working fine today... [banghead] [lol]
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


Blicj11

  • Storm
  • *****
  • Posts: 3941
    • EW3808
    • KUTHEBER6
    • Timber Lakes Weather
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP Data Logger | iMac (2019), 3.6 GHz Intel Core i9, 40 GB RAM, macOS Ventura 13.6 | Sharx SCNC2900 Webcam | WeatherCat 3.3 | Supportive Wife
Re: NWS Forecast Problems?
« Reply #11 on: June 28, 2017, 04:06:14 AM »
The only drawback I see to you pulling up the forecast in widget mode is that in addition to eliminating the footer, it also eliminates any weather warnings or hazardous weather outlooks in effect. I've been experimenting with my forecast and don't see a way to eliminate the footer but keep the warnings. Here's what mine looks like without the widget mode, allowing you to see the Red Flag Warning for fire danger.

https://forecast-v3.weather.gov/point/40.4612,-111.2283

Here is what the current page looks like, which is, in my opinion, a better graphic representation than the new and "improved" site.

https://forecast.weather.gov/MapClick.php?lon=-111.22832775115889&lat=40.46118454986899#.WVMWMsaZNUN

I wish they would just stop messing around and implement their changes. Of course that logic does not work so well when it comes to healthcare.
Blick


xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: NWS Forecast Problems?
« Reply #12 on: June 28, 2017, 03:55:52 PM »
Not a fan of either of the NWS options. Fortunately, the SOAP_server seems to have had its problems fixed... at least for ~48 hours! ThU32:-)

Except for the page header (includes the site nav menu), I have separate warning and forecast scripts:
Code: [Select]
<?php  include './noaa-warn/noaaWarning-start.php';  ?>
<h3 class="center top_pad bottom_pad">A Detailed Forecast</h3>
<?php  include './noaa_forecast/myforecast.php';  ?>
Even the page footer is another php script:
Code: [Select]
<?php  include './WX-pageFooter.php';  ?>It's about the simplest page on my site! [lol]
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Agreed - not progress (Re: NWS Forecast Problems?)
« Reply #13 on: June 28, 2017, 11:20:22 PM »
Dear Blick, X-Air and WeatherCat seekers of genuine progress,

The only drawback I see to you pulling up the forecast in widget mode is that in addition to eliminating the footer, it also eliminates any weather warnings or hazardous weather outlooks in effect.

. . . .

I wish they would just stop messing around and implement their changes. Of course that logic does not work so well when it comes to healthcare.

I second (or third) your gripe.  If the NWS went to all that trouble to change things - they definitely should have made it clearer that there were warnings.

Another case of  . . . . .

Oh well, . . . . . Edouard

xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: NWS Forecast Problems?
« Reply #14 on: July 21, 2017, 03:22:56 PM »
I think I've found a 'fix'! I created a minimalist, alternate "Forecast" page after growing tired of the NWS SOAP_server problems. Almost immediately, the SOAP_server seems to get fixed!

I guess what would be best would be to have (some one else build) a javascript to watch the output of the SOAP_server and switch to the alternate MapClick version. One problem (other than my total lack of interest and skill of javascript [rolleyes2] ) is that the SOAP_server doesn't always create the same erer! Sometimes it simply 'times out'. Usually it's just an empty file or one that lacks parts of the data and it ends up causing php undefined variables errors. Of course, the easiest solution would be to just display the MapClick version. [banghead]
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system