Data Sent to CWOP is Sometimes Sporadic

[Charlie Schorner]

I can't emphasize how much I enjoy WeatherCat! I've been using it reliably for many years. Recently, a data upload anomaly has appeared.

I send data to WeatherUnderground and CWOP, and the web pages seem to be as up-to-date as possible. Somewhere along the line, NOAA snags the data and publishes this page: http://www.wrh.noaa.gov/mesowest/getobext.php?sid=D4698. Every so often, their data feed simply stops. I think the problem began about six months ago, and I learned about it when I got an email from someone at NOAA asking if I'd please keep sending data. They had not received anything for a few days at that point, but both the CWOP page and the WU page were current. Restarting WeatherCat corrected the situation, but I never could figure out what caused it in the first place.

Now, during the day, I regularly check the NOAA page to make sure they're receiving data and, this morning I noticed that there had been a gap overnight. For some reason, data stopped after 8:19pm then, equally as mysteriously, began again at 5:04am. Both CWOP and WU show no data loss.

Has anyone else experienced this behavior?

[elagache]

Dear Charlie and WeatherCat data exporters,

I send data to WeatherUnderground and CWOP, and the web pages seem to be as up-to-date as possible. Somewhere along the line, NOAA snags the data and publishes this page: http://www.wrh.noaa.gov/mesowest/getobext.php?sid=D4698. Every so often, their data feed simply stops.

. . . .

Now, during the day, I regularly check the NOAA page to make sure they're receiving data and, this morning I noticed that there had been a gap overnight. For some reason, data stopped after 8:19pm then, equally as mysteriously, began again at 5:04am. Both CWOP and WU show no data loss.

. . . .

Has anyone else experienced this behavior?

Okay, first you've found something that at least I wasn't aware of.  At least for those of us in the western United States, we all should have a page like this if we submit data to CWOP.  Here is mine:

http://www.wrh.noaa.gov/mesowest/getobext.php?sid=D3835

Just substitute your CWOP submission id for D3835 to see your own page.

As to the data gaps . . . . . hmm, honestly I have no idea. 

One thing you might try is to change the CWOP server you upload to.  You can adjust this in the CWOP preferences.  It is a long shot, but at least it might help you diagnose the problem.  Another resource that might be handy is the list of CWOP server outages:

https://groups.google.com/forum/?hl=en#!forum/cwop-server-status-alerts

You might check to see if the server you are uploading to is having problems.

Let us know if you uncover anything.  I'm sure everyone who uploads to CWOP would like to know if there are any "issuez."

Cheers, Edouard 

[Charlie Schorner]

Thanks Edouard! The strange part about this is, if you look at the CWOP page, like this one: http://www.findu.com/cgi-bin/wxpage.cgi?call=DW4698&last=24 , the data have all beed properly uploaded while on the NOAA page, there's a huge gap in the data: http://www.wrh.noaa.gov/mesowest/getobext.php?sid=D4698 . And, of course, WeatherUnderground has all the data too: http://www.wunderground.com/personal-weather-station/dashboard?ID=KWAPRESC4#history/s20140806/e20140807/mcustom .

One thing I've noticed from time to time is, when I have a power outage that lasts long enough for all of my UPS's to run out, no matter how many hours go by, when WeatherCat starts up again, it'll fetch all of the data collected in the weather station and send it all to WU but not to CWOP. I have no idea if that is in any way related to this NOAA problem.

I'll try using a different server and see what happens. The problem is sporadic and does not occur on a regular basis, so I really won't know if it's been fixed until I see another gap, which might not happen for months.

[Randall75]

Hi Charlie
 WeatherCat will upload historical data to WU this can't be done a the moment to CWOP so if your power is off for a long period of time that data is going to be missing from CWOP and that is probably what is happening to the data at NOAA


cheers


 

[Charlie Schorner]

Thanks Randy! At least I now know that it's expected behavior.

I looked at various APRS pages and have determined that the server setting I'm using (cwop.aprs.net) is probably the best choice. And, looking at the data being sent, it seems to go to any one of the actual servers at any given time, but I'm not sure how the decision is made which one at which time. So, maybe there's some sort of time-out or log-in error or something that causes the data to not go to any of the servers? Hard to say at this point. I'll keep an eye on it, as I have been, and perhaps I'll just re-start WeatherCat every couple of days as a preventative measure.

[Blicj11]

Charlie:

I've gone back and reviewed my uploads to CWOP and don't see anything like what you are experiencing. I'm uploading to Server 1. Sorry I can't help, but thanks for pointing out the summary conditions weather page we each have at CWOP. I have never seen that before. Keep us informed.

[Charlie Schorner]

OK! I've realized that I lost track of what the original problem in this thread was! It's the information displayed on the NOAA MesoWest page that disappears from time to time. Data going to CWOP and WU are fine. Here's what I learned from the MesoWest FAQ:

MesoWest obtains nearly all of our citizen weather observations through
the Citizen Weather Observer Program (CWOP) (http://www.wxqa.com/).
We believe their website has information on how to get a system up and
running as well as communicating with the world. These CWOP observations
then get ingested into a system known as Meteorological Assimilation Data Ingest System, MADIS
(http://www-sdd.fsl.noaa.gov/MADIS/index.html) which in turn sends the data
to us here at Mesowest. As such Mesowest does not directly interact with
individual citizen weather stations.

Given this statement, I'd guess that my re-starting WeatherCat to get the data going again actually had no real effect. The data re-appeared after my re-start and I assumed cause and effect, but it was just coincidence. Next time I notice that the MesoWest page isn't getting data, I'll do nothing and see what happens. Maybe the problem is at the MADIS level?

[elagache]

Dear Charlie, Blick, Randall, and WeatherCat fans,

OK! I've realized that I lost track of what the original problem in this thread was! It's the information displayed on the NOAA MesoWest page that disappears from time to time. Data going to CWOP and WU are fine.

. . . .


Given this statement, I'd guess that my re-starting WeatherCat to get the data going again actually had no real effect. The data re-appeared after my re-start and I assumed cause and effect, but it was just coincidence. Next time I notice that the MesoWest page isn't getting data, I'll do nothing and see what happens. Maybe the problem is at the MADIS level?

I think you've assessed the situation correctly.  If the data reaches CWOP as it should, then you and WeatherCat have done all you could to make the data available to the NOAA.  Most likely there is a communication problem in between CWOP and the NOAA somewhere.

If you receive another email from NOAA, you might point out that indeed all the data is reaching CWOP.  That should allow them to fill in any missing data they might need from CWOP directly, and it might stimulate an effort to fix the problem that has to be on their end.

Cheers, Edouard 

[Bull Winkus]

It could be carried by an African swallow. Oh, sorry! Wrong script. Wrong dialogue. Just a moment. ? Yes, here it is.

It could be that the data interruption, precipitated by the re-start, did actually cause the NOAA's feed to resume. It's possible that the software runs through a sequence on startup of data that initiates the data forwarding. Though, even in this case, the problem that caused the lapse would have to be CWOP's or MADIS's.

Well, that concludes my half thought out analysis. Redactions at a future time may be necessary. Again, sorry for the confusion in the script. I really must sort this out. ? Carry on.

[Blicj11]

It could be carried by an African swallow. Oh, sorry! Wrong script. Wrong dialogue. Just a moment. ? Yes, here it is.

Herb:

You are usually good for a laugh and on many days, you are way better than the weather.

[Bull Winkus]

But, not today, eigh Blick! Lol!

[Blicj11]

Ha ha ha! I can see how you interpreted my observation but I didn't intend to send that message. So now you have given me two good laughs on the same day and I thank you for it. You, my friend, are on a roll. Keep 'em coming.

My apologies to Charlie (the OP) for not being more helpful, but when Herb is on the loose, I have to chime in.

[Charlie Schorner]

Actually, I'm surprised that the discussion lasted as long as it did before derailing. I had mis-interpreted the situation and am grateful to have not gotten bludgeoned as I would have on so many other forums. At least it had a chance of coming to some sort of conclusion, whether useful or not. I think that I'll just ignore the NOAA data gaps since I don't actually send the data directly to them. If I see gaps on my CWOP page, I'll know to take action (of some sort).

[Blicj11]

WeatherCatters are a nice lot to hang with. Nobody bludgeons here. The worst cudgeling I recall was when some hooligans stole Herb's weather station and all of that was directed at them, and even then, Herb retained his sense of humour.

[Bull Winkus]

Thanks Blick! My ego is suitably inflated now. Feels like I've had too much fried chicken. (grin)

On a more serious OT note: In reference to a previous conversation, I finally finished my little outdoor project today. My concrete step from the side of the carport to the somewhat elevated gravel driveway now has a grouted stone walkway. And, I found and bought a booklet from Amazon called, "Treat Your Own Neck" by Robin McKenzie. Although I haven't had time to read it thoroughly, I can say that it looks very well thought out and has some lovely illustrations. It might be worth your while. Here's a link. Best wishes!

http://www.amazon.com/gp/product/0987650416/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1