Trixology

Weather => General Weather Discussion => Topic started by: xairbusdriver on June 24, 2017, 05:14:20 PM

Title: NWS Forecast Problems?
Post by: xairbusdriver on June 24, 2017, 05:14:20 PM
Whenever I have problems, I assume the first cause is me. However, I'm simply asking if anyone else has had problems the last few days with the NWS Forecasts. I am mainly getting:
Quote
CLIENT ERROR
HTTP Error: Unsupported HTTP response status 504 Gateway Time-out (
 Essential data like temperature is mssing - program stops

The basic url I have been using for the last several months is:

https://graphical.weather.gov/xml/SOAP_server/ndfdXMLclient.php...location and other details

Perhaps the NWS has not only changed to security but also the domain? Or, as is usually the case, I've just screwed up something...

Gotta mow the back yard, just got a new mower! I'll research the "Gateway" time-out meaning if it's actually not what it appears to literally mean.

Additional info via Network Utility:
Loopup: graphical.weather.gov -> (e278.dscg.akamaiedge.net) ->23.36.68.92

Ping: PING 23.36.68.92 (23.36.68.92): 56 data bytes
64 bytes from 23.36.68.92: icmp_seq=0 ttl=55 time=83.870 ms
64 bytes from 23.36.68.92: icmp_seq=1 ttl=55 time=32.479 ms
...
64 bytes from 23.36.68.92: icmp_seq=9 ttl=55 time=32.187 ms
--- 23.36.68.92 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss

Traceroute: traceroute to 23.36.68.92 (23.36.68.92), 64 hops max, 72 byte packets
 1  10.0.1.1 (10.0.1.1)  0.469 ms  0.264 ms  0.169 ms
 2  10.0.0.1 (10.0.0.1)  1.534 ms  1.521 ms  1.779 ms
 3  www.xxx.yyy.zzz (www.xxx.yyy.zzz)  15.837 ms  14.290 ms  13.748 ms
 4  te5-3-ur01.mycity.ST.malt.comcast.net (www.xxx.yyy.zzz)  16.154 ms  14.375 ms  14.858 ms
 5  96.108.172.117 (96.108.172.117)  15.113 ms  14.313 ms  13.208 ms
 6  te-0-0-0-0-ur04.hattiesburg.ms.malt.comcast.net (68.85.203.21)  16.745 ms  14.327 ms  14.956 ms
 7  be-22258-cr02.dallas.tx.ibone.comcast.net (68.86.93.93)  45.036 ms  24.263 ms  24.890 ms
 8  be-11424-cr02.56marietta.ga.ibone.comcast.net (68.86.85.21)  35.018 ms  34.329 ms  34.893 ms
 9  hu-0-11-0-5-pe01.56marietta.ga.ibone.comcast.net (68.86.88.222)  32.728 ms  34.278 ms  34.929 ms
10  * * *
11  a23-36-68-92.deploy.static.akamaitechnologies.com (23.36.68.92)  33.930 ms  34.281 ms  35.090 ms

I think, with my minimal "knowledge" that this is saying the problem is with the "SOAP server". I think the "***" means there are three 'timeouts' or delays before the last server (deploy.static.akamaitechnologies.com) responds. ??? Is there a place/site that reports NWS server status? Direct Forecast access seems fine.
Title: More than one way to skin a catfish (Re: NWS Forecast Problems?)
Post by: elagache on June 24, 2017, 10:44:05 PM
Dear X-Air and WeatherCat cyber weather data miners,

Whenever I have problems, I assume the first cause is me. However, I'm simply asking if anyone else has had problems the last few days with the NWS Forecasts. I am mainly getting:
Quote
CLIENT ERROR
HTTP Error: Unsupported HTTP response status 504 Gateway Time-out (
 Essential data like temperature is mssing - program stops

It is possible that the problem isn't the server you are using graphical.weather.gov, but another server that is being queried in the process of generating the forecast.

The basic url I have been using for the last several months is:

https://graphical.weather.gov/xml/SOAP_server/ndfdXMLclient.php...location and other details

For whatever it's worth, this is the forecast URL I've been using for years:

Code: [Select]
https://forecast.weather.gov/MapClick.php?lat=37.8511&lon=-122.15472&site=mtr&smap=1&unit=0&lg=en&FcstType=text
It works fine but will probably break when NWS finally succeeds in launching their new web forecast API.  The parameters appear to be reasonably obvious

Gotta mow the back yard, just got a new mower!

Well, I've still got my old mower, but I just got back from moving our yard.  The 10 day heat-wave definitely got the grass growing!

Cheers, Edouard  [cheers1]
Title: Re: NWS Forecast Problems?
Post by: xairbusdriver on June 24, 2017, 11:43:54 PM
The "graphical" forecast is actually used for the multi-day, icon filled 'graphics-based' forecast, it's the upper part of my Forecast page. I think the one you use may be for the text display of the Forecast? The Summary forecast (Midnight to Noon and Noon to Midnight) actually always still works. By the way, I'm using the Leuven-NOAA stand-alone PHP scripts.

It is now working, of course... I'll upload a couple of screen shots after dinner. ;)

The site is now reporting the "Time-out" again. [rolleyes2]
Title: Re: NWS Forecast Problems?
Post by: wvdkuil on June 25, 2017, 12:13:42 PM
Since last Thursday until now 2017-06-25 maybe 50% or more of the  graphical.weather.gov/xml/ server-requests end in that "gateway time-out" error
The server graphical.weather.gov is reachable normally but the back-end server fails.

This link https://graphical.weather.gov/xml/SOAP_server/ndfdXML.htm is the "human" interface to get the same xml data as the scripts do.
You click a few variables you want and press submit at the bottom.
Those requests fail also with the same frequency as the scripts.

Hope they will see the errors themselves and someone will do something about it shortly.

Wim
Title: Re: NWS Forecast Problems?
Post by: xairbusdriver on June 25, 2017, 04:13:17 PM
Thanks so much, Wim. I knew it wasn't your template, hat's why I hadn't contacted you. I still have your error reporting code enabled otherwise I would have understood even less! I really appreciate your confirming my suspicions and especially that I haven't screwed up something on my end. I haven't even touched the code since changing the "http" to "https" a couple of months ago. [computer]

Quote
Hope they will see the errors themselves and someone will do something about it shortly.
I did a minimalist search for a 'contact' link at the NWS site that might be appropriate, but I assumed they would be getting reports from commercial users by now. Perhaps the majority of heir users are hobbyists?! [banghead] I hope it doesn't indicate a problem with funding or lack of manpower at the organization! I hear reports that NOAA is getting negative comments about replacing their weather satellites. [rolleyes2]
Title: Speaking of which . . . . (Re: NWS Forecast Problems?)
Post by: elagache on June 26, 2017, 11:36:49 PM
Dear X-Air and WeatherCat web-based weather information seekers,

Now that you mention it, even the URL I use to get a forecast is having sporadic "issuez."  For example, It just generated the following error message:

Quote
Service Unavailable - Zero size object
The server is temporarily unable to service your request. Please try again later.

Reference #15.f532c517.1498516287.69c8395

Retrying after waiting a bit works fine, but it is new and does happen as often as once a day or more.  Perhaps the National Weather Service is doing some server upgrades associated with their attempt to launch their new API.  As a result, various services can be temporarily unavailable.

Cheers, Edouard  [cheers1]
Title: Re: NWS Forecast Problems?
Post by: xairbusdriver on June 27, 2017, 01:57:00 AM
Quote
As a result, various services can be temporarily unavailable.
My opinion, also. I may just take down the Forecast page or create a new one with a usable text display. I hate to lose the Summary and Detailed versions, but they are also (I think) generated by the SOAP XML data.

Makes me wonder what are others are using for the NWS Forecast? No one else here seems to be affected! :o
Title: Re: NWS Forecast Problems?
Post by: wvdkuil on June 27, 2017, 05:59:02 AM
Quote
As a result, various services can be temporarily unavailable.
My opinion, also. I may just take down the Forecast page or create a new one with a usable text display. I hate to lose the Summary and Detailed versions, but they are also (I think) generated by the SOAP XML data.

Makes me wonder what are others are using for the NWS Forecast? No one else here seems to be affected! :o
ALL NWS scripts are affected.  A small list of changes below:

The point-forecast => https://forecast.weather.gov/MapClick.php?lat=37.8511&lon=-122.15472&site=mtr&smap=1&unit=0&lg=en&FcstType=text

That is a 1 page forecast  with text and icons is affected as the whole architecture behind the scened will be changed. The cutover to the new format was scheduled for April, postponed to June a.s.o., is now scheduled for August. It is used in the Leuven scripts also as "plain-page forecast

The new structure needs  3 or 4 calls to the servers to get at the data, as in the past one was needed.  That will increase the number of errors, I think.

Icons: This and last year, there have been already multiple changes in the icon-structure and location of the icons.  NWS totally changed the icon structure when they introduced multipart icons with the .php extension => two six hour icons "glued together"

http => https  All links will be changed for that change

URL When doing all this the location (URL) is changing also

SOAP This source of information is not heavily used by scriptwriters, I think. 
https://graphical.weather.gov/xml/SOAP_server/ndfdXML.htm
But inside that wealth of information there are links for icons and alerts which has to change also.

Alerts The URL, location and structure are changing also. So those scripts, example the "Curly" scripts are and will  be adapted in coming months.

Other forums at least a dozen on wx and also some on ww:
http://www.wxforum.net/index.php?topic=32341.msg326185#msg326185
http://www.wxforum.net/index.php?topic=32287.msg325627#msg325627
http://www.wxforum.net/index.php?topic=31954.msg321304#msg321304

Wim
Title: Re: NWS Forecast Problems?
Post by: xairbusdriver on June 27, 2017, 09:24:54 AM
Thanks for the thorough and complete explanation, Wim! I must say I've usually skipped over the Graphical part of the display and mainly looked at the Summary table. If I wanted a more 'definite' time period in a day, the Detailed display was nice.

Of course, weather "forecasts", at least in the US, are not as accurate as our TV weather people imply. These changes don't help with the lack of apparent accuracy. I suspect the changes are more difficult than expected, also.

I've got lots of work to do in cleaning up my PHP to keep me busy for the next few weeks! I've ended up with severe code bloat! I think ~95% of it can be converted to functions. But we know how accurate 'forecasts' are!i  cmu:-)
Title: Thanks Wim for the detailed info (Re: NWS Forecast Problems?)
Post by: elagache on June 27, 2017, 11:41:56 PM
Dear Wim, X-Air and WeatherCat coders,

Thanks for the thorough and complete explanation, Wim!

Indeed thanks.  The updates were rather as I suspected, but I appreciate the detailed summery.

Cheers, Edouard  [cheers1]
Title: Re: NWS Forecast Problems?
Post by: xairbusdriver on June 27, 2017, 11:45:43 PM
This gets me as close to the basic display as I can get using: https://forecast-v3.weather.gov/point/35.0702,-89.7363?mode=widget (https://forecast-v3.weather.gov/point/35.0702,-89.7363?mode=widget) Better than nothing, I guess. [rolleyes2] At least it doesn't have the massive footer as in the MapClick version.

Of course, the NWS site seems to be working fine today... [banghead] [lol]
Title: Re: NWS Forecast Problems?
Post by: Blicj11 on June 28, 2017, 04:06:14 AM
The only drawback I see to you pulling up the forecast in widget mode is that in addition to eliminating the footer, it also eliminates any weather warnings or hazardous weather outlooks in effect. I've been experimenting with my forecast and don't see a way to eliminate the footer but keep the warnings. Here's what mine looks like without the widget mode, allowing you to see the Red Flag Warning for fire danger.

https://forecast-v3.weather.gov/point/40.4612,-111.2283

Here is what the current page looks like, which is, in my opinion, a better graphic representation than the new and "improved" site.

https://forecast.weather.gov/MapClick.php?lon=-111.22832775115889&lat=40.46118454986899#.WVMWMsaZNUN

I wish they would just stop messing around and implement their changes. Of course that logic does not work so well when it comes to healthcare.
Title: Re: NWS Forecast Problems?
Post by: xairbusdriver on June 28, 2017, 03:55:52 PM
Not a fan of either of the NWS options. Fortunately, the SOAP_server seems to have had its problems fixed... at least for ~48 hours! ThU32:-)

Except for the page header (includes the site nav menu), I have separate warning and forecast scripts:
Code: [Select]
<?php  include './noaa-warn/noaaWarning-start.php';  ?>
<h3 class="center top_pad bottom_pad">A Detailed Forecast</h3>
<?php  include './noaa_forecast/myforecast.php';  ?>
Even the page footer is another php script:
Code: [Select]
<?php  include './WX-pageFooter.php';  ?>It's about the simplest page on my site! [lol]
Title: Agreed - not progress (Re: NWS Forecast Problems?)
Post by: elagache on June 28, 2017, 11:20:22 PM
Dear Blick, X-Air and WeatherCat seekers of genuine progress,

The only drawback I see to you pulling up the forecast in widget mode is that in addition to eliminating the footer, it also eliminates any weather warnings or hazardous weather outlooks in effect.

. . . .

I wish they would just stop messing around and implement their changes. Of course that logic does not work so well when it comes to healthcare.

I second (or third) your gripe.  If the NWS went to all that trouble to change things - they definitely should have made it clearer that there were warnings.

Another case of  . . . . .

Oh well, . . . . . Edouard
Title: Re: NWS Forecast Problems?
Post by: xairbusdriver on July 21, 2017, 03:22:56 PM
I think I've found a 'fix'! I created a minimalist, alternate "Forecast" page after growing tired of the NWS SOAP_server problems. Almost immediately, the SOAP_server seems to get fixed!

I guess what would be best would be to have (some one else build) a javascript to watch the output of the SOAP_server and switch to the alternate MapClick version. One problem (other than my total lack of interest and skill of javascript [rolleyes2] ) is that the SOAP_server doesn't always create the same erer! Sometimes it simply 'times out'. Usually it's just an empty file or one that lacks parts of the data and it ends up causing php undefined variables errors. Of course, the easiest solution would be to just display the MapClick version. [banghead]
Title: Re: NWS Forecast Problems?
Post by: xairbusdriver on May 09, 2018, 07:44:44 PM
I've given up on NOAA and their SOAP servers, I think they need a bigger 'soap dish'! I've simply deleted all but the graphical and the Detailed displays. The first is easy to get the 'big picture' and if details are desired, it's there. Since it's my site and I almost always went to the Detailed part, I saw no need for garbage or nothing in the other tab. I'm happy and what else matters?
I'll  be right there dear...
Title: Re: NWS Forecast Problems?
Post by: wvdkuil on May 09, 2018, 09:02:30 PM
I've given up on NOAA and their SOAP servers, I think they need a bigger 'soap dish'! I've simply deleted all but the graphical and the Detailed displays. The first is easy to get the 'big picture' and if details are desired, it's there. Since it's my site and I almost always went to the Detailed part, I saw no need for garbage or nothing in the other tab. I'm happy and what else matters?
I'll  be right there dear...
As I am not living in the US and also not that familiar with all the problems NWS/NOAA is facing I could need some help  and feed-back.

At the "beta" forecast script page https://www.weerstation-herent.be/saratoga/wsfct4/  there are two separate NWS/NOAA forecast scripts.
But if you want to test those scripts, feel free to do so. At the bottom of the https://www.weerstation-herent.be/saratoga/wsfct4/ page there is a download link.

Wim
Title: Re: NWS Forecast Problems?
Post by: xairbusdriver on March 20, 2019, 12:40:03 AM
For the last few weeks, the SOAP servers have been acting up (corrupted files/info) a large percentage of most days. It's that inconsistency that is frustrating to me. [banghead] I must admit that I haven't followed up on Mr. van der Kuil's suggestion. That may be the best solution. ;D

It is also frustrating to think that our own government can't seem to reliably provide this kind of information. [rolleyes2] I'm sure there are commercial entities that would accept my money for their forecasts. But this is, after all, simply a hobby for me. I doubt even a couple of folks on my street ever visit my web site. For my own purposes, I can interpret a forecast map enough to know the general weather for the next few days.

So, first question: Is there a reliable, free, US-based source for forecasts mainly for the US of A?

Second question: Is there a similar service but from another area of the world? I have 'played' with www.weather-forecast.com and could probably "borrow" their output and they might never discover my local address in order to send a 'cease and desist' order! [rolleyes2] :o :-\
Title: Re: NWS Forecast Problems?
Post by: saratogaWX on March 20, 2019, 04:57:13 PM
The premier free USA forecast source is from weather.gov (the NWS site).  There are (as Wim cited) two versions of my advforecast2.php script that use this data.  The V4 script is a 'page scraper' from the point-printable forecast produced by a MapClick.php URL -- this is the current 'production' version of the NWS website and is the best maintained, most reliable.

For a couple of years now, a 'replacement' for www.weather.gov has been in development/testing at https://forecast-v3.weather.gov/ and it relies on data provided via https://api.weather.gov/ -- my advforecast2.php V5 script uses this JSON API and relies on the lat/long to find the forecast.  The API is WAY more atomic than one would expect -- it requires multiple API accesses to gather the data for a forecast, but since much of the extra accesses can be safely cached (data doesn't change), it only resulted in some additional coding the gather/cache when a new forecast point is accessed.  The basic issue with api.weather.gov is it is still in BETA and so support and reliability are both lacking -- NWS local offices are sometimes lax in loading the point forecast data from the models into the api.weather.gov services, so sometime only a zone forecast is available for extended periods.  The production cutover is long overdue, and no word has been issued by the NWS on production timing for api.weather.gov.

The SOAP access to NWS data has been problematic for years, and I think they'll deprecate it in the future when the api.weather.gov is made productional.

Both of the above scripts only work for USA locations.  For worldwide locations, https://darksky.net/ has forecasts derived from various country models and provides a free, low-volume API key for developers (and hobbyists) -  My DS-forecast.php script uses their data.
One thing to note is that they only provide icons for Daytime weather but they have multilingual capability.

WeatherUnderground has deprecated their old API for a new weather.com (TWC) API that is free only to WU members who submit PWS data to them.  All others have to pay $$$ -- I have a WC-forecast.php script that handles their API.  It too is multilingual but only offers 5 days of forecasts (but both Day and Night icons).
Title: Re: NWS Forecast Problems?
Post by: The Grand Poohbah on March 20, 2019, 06:03:19 PM
Quote
Both of the above scripts only work for USA locations.  For worldwide locations, https://darksky.net/ has forecasts derived from various country models and provides a free, low-volume API key for developers (and hobbyists) -  My DS-forecast.php script uses their data.

I use the Dark Sky iOS App daily for local forecasts. It was also useful for the locations on our trip - Sydney NSW, Darwin NT and Brisbane OLD Australia; Bali, Indonesia, and Singapore. It's free if you don't mind the innocuous ads.
Title: Re: NWS Forecast Problems?
Post by: xairbusdriver on March 20, 2019, 06:42:59 PM
Thanks guys! I promise to take a look at all the suggestions before complaining again! [blush] I'd be happy to have just a plain text version or a table with data in consistent places. I don't mind (actually enjoy) putting stuff where and how I want. [lol]
Title: Re: NWS Forecast Problems?
Post by: xairbusdriver on March 20, 2019, 11:04:38 PM
I think I may be able to use this basic, text-only, table-based page (https://forecast.weather.gov/MapClick.php?lat=35.0704&lon=-89.7425&unit=0&lg=english&FcstType=text&TextType=1) for my purposes. Images and other 'computed' items can be displayed by parsing the text (which I already have some experience with my METAR displays). It should be relatively easy to allow user input for different (US, at least) locations, possibly with Zip Codes. Again, the key factor is reliable data in a consistent format. Simple display is only slightly less important.

Thanks, again, Wim and Ken! ThU5:-)