Author Topic: I'm *this* close to getting historical data imported yet?  (Read 4150 times)

TWSheppard

  • Gentle Breeze
  • **
  • Posts: 21
    • EW2262
    • IONSTITT1
    • Stittsville Weather
  • Station Details: Davis Vantage Pro 2, MacBook, 1 additional Temp/Hum sensor
I'm *this* close to getting historical data imported yet?
« on: March 02, 2013, 03:46:39 AM »
I've created historical data files going back to mid-2005 with some gaps in local data that I'm in the process of retrieving from Weather Underground. I've placed them in ~/Library/WeatherCatData/Location1/yyyy/n_WeatherCatData.cat.

However, only the data for 2013-01 thru 03 are loaded. The files in 2005 thru 2010 are ignored. I'm missing the end of 2010, 2011 and 2012 and will create those files once I retrieve the data from WU.

The console logs only indicate "Loading current month?".

Is there some magic I need to do to get the data files in earlier years to be recognized?

Thanks.

Here's the first few lines from 2010/1_WeatherCatData.cat:

VERS:3

0 t:010000 T:-5.056 Ti:22.556 T1:-15.000 D:-6.438 Pr:1011.481 W:0.000 Wd:86 Wc:-5.056 Wg:6.436 Ph:0.000 P:0.000 H:90 Hi:31.000 H1:0.000 Pm:0.000 V:4
1 t:010001 T:-5.056 Ti:22.556 T1:-15.000 D:-6.438 Pr:1011.481 W:3.218 Wd:80 Wc:-6.129 Wg:6.436 Ph:0.000 P:0.000 H:90 Hi:31.000 H1:0.000 Pm:0.000 V:4

TWSheppard

  • Gentle Breeze
  • **
  • Posts: 21
    • EW2262
    • IONSTITT1
    • Stittsville Weather
  • Station Details: Davis Vantage Pro 2, MacBook, 1 additional Temp/Hum sensor
Re: I'm *this* close to getting historical data imported yet?
« Reply #1 on: March 04, 2013, 06:48:31 PM »
I managed to get all the data imported from Virtual Weather Station on the PC, and filled in the gaps that VWS failed to log using data from Weather Underground.

It seems WC works backwards through the files and if it encounters a gap, it stops rather than skipping  to the older entries.

So all is good. Much happier with WC than I ever was with VWS.

WCDev

  • WeatherCat Developer
  • Administrator
  • Storm
  • *****
  • Posts: 2911
    • CW9739
    • ISCOTLAN25
    • Trixology
  • Station Details: Main Station: Vantage Pro-2, 24hr fars, solar, soil/leaf station, extra temp stations, no U.V. WeatherLink IP.
Re: I'm *this* close to getting historical data imported yet?
« Reply #2 on: March 04, 2013, 07:06:08 PM »
Sorry, missed this when you first posted.

Yes, as you've found, on boot WeatherCat scans backwards in time looking for data files. For each one it checks if the .hrs file exists (this is the hourly data), if not it creates one from the data file then carries on scanning until it finds a hole whereupon it stops. The only actual weather data it loads on boot is the current month (and the previous month if needed - i.e. graphs that are displaying data from last month) -  all other data is loaded as needed (for example scrolling back through time into a previous month will cause that month's data to be loaded if it isn't already loaded).

People are always looking to import data into WeatherCat - do you think you could put up a quick post detailing what you did?

Cheers,
Stu.

TWSheppard

  • Gentle Breeze
  • **
  • Posts: 21
    • EW2262
    • IONSTITT1
    • Stittsville Weather
  • Station Details: Davis Vantage Pro 2, MacBook, 1 additional Temp/Hum sensor
Re: I'm *this* close to getting historical data imported yet?
« Reply #3 on: March 04, 2013, 09:37:42 PM »
To import the data I had to write several scripts. Perl is my language of choice. As with most hack projects, I only have enough comments and error checking to satisfy my needs. Without a fair bit of work, the scripts would not be suitable for publishing.

However, if people need data extracted from Virtual Weather Station (http://www.ambientweather.com/virtualstation.html) or  Weather Underground (http://www.wunderground.com) I wouldn't mind assisting if they send me a message.

I'm not aware of a bulk data downloader from WU so I brute-forced the download of the two plus years of data that VWS "forgot" to record in its local database by requesting WU's pseudo-.csv files slowly and post-processing them. I downloaded a new day every 1-5 minutes so as not to place undue load on the WU servers.

SeaflyP

  • Gentle Breeze
  • **
  • Posts: 15
    • Seafly Memories
  • Station Details: Davis Vantage Vue Pro 2/USB Envoy/ iMac 27" i7 (2011)/MacOs 10.6
Re: I'm *this* close to getting historical data imported yet?
« Reply #4 on: March 07, 2013, 07:44:34 AM »
Has anyone a system for importing old wlink data into weathercat? I want to import the data I recorded since 2007. It should be possible by writing one or more unix (sed ?) scripts but I don't want to be re-inventing the wheel!
Cheers, Peter

ELO

  • Gentle Breeze
  • **
  • Posts: 75
    • KCOBRECK14
    • Weather in the Highlands
  • Station Details: Old Davis WMII, Heated Rain Collector, Mac Mini
Re: I'm *this* close to getting historical data imported yet?
« Reply #5 on: March 12, 2013, 08:47:01 PM »
I posted this note over in the Feature Request forum a little bit ago, but I've copied here for convenience:

Well... I still would love to see a data importer for WeatherLink files. I tried using LWC2 as part of a conversion process only to have it crash with an instruction exception when I ran it on Mac OS 10.8.2 and identified a WeatherLink file to convert. I've noticed that many folks talk about WeatherLink .WLK files; however, mine are old Mac WeatherLink files in a non human readable form with a .HOM extension. So that may be the source of the LWC2 crash and in the end a .wlk file converter/importer wouldn't help me.

All that being said, I have managed so far to get my WL data back to the first of 2013 into WeatherCat using an excel template I developed. Although it is still a multi-step process, converting a month's worth went relatively quickly once I had the template built. I'm certainly willing to share the process and template with anyone interested; however, keep in mind that I am working with old Davis Weather Monitor II data, which is less extensive than Davis Vantage or other stations, and my data logger was set for 30 archiving - 48 records per day or 1440 per 30 day month spaced at regular 30 minute intervals. That helped and is taken as a given in my template.

I'm sure this is not as elegant as a well done script and it still takes time & thought; however, it beats the heck for me out of no historical data and for sure trying to do something by hand. I'm now debating whether to tackle 2012, and if so, whether to go back to the beginning of my data in 2005. That still would be a major time undertaking.

A couple things I did learn in the process is to be sure the X_WeatherCatData.cat file actually has a .cat final extension. I am using TextWrangler for final cleanup & processing of my template output and it was appending an additional hidden .txt extension, which then kept WeatherCat from processing the file. Second, be sure the final file is saved with Unix Linefeeds. Again, TextWrangler was saving the files with Classic Mac Linefeeds, which caused a corrupt data error in WeatherCat.

Rick

SeaflyP

  • Gentle Breeze
  • **
  • Posts: 15
    • Seafly Memories
  • Station Details: Davis Vantage Vue Pro 2/USB Envoy/ iMac 27" i7 (2011)/MacOs 10.6
Re: I'm *this* close to getting historical data imported yet?
« Reply #6 on: March 14, 2013, 06:52:04 PM »
It's looking like I'll have to write a script afterall - but I'm afraid it will only work with data exported from wlink in text format!
Peter

ELO

  • Gentle Breeze
  • **
  • Posts: 75
    • KCOBRECK14
    • Weather in the Highlands
  • Station Details: Old Davis WMII, Heated Rain Collector, Mac Mini
Re: I'm *this* close to getting historical data imported yet?
« Reply #7 on: March 18, 2013, 12:19:23 AM »
My WLink files were non-text .HOM files; however, I still had Mac WeatherLink 5.x available and there is an export function available when you are browsing historical data. The export function allows you to select a time period and then exports data to a text file. That's what I did.

I then post processed the individually exported monthly files in an Excel spreadsheet and TextWrangler. I did generalize my spreadsheet so that it handles the Date/Time code (t:) and the T: Ti: D: Pr: W: Wc: Wg: Wd: P: H: Hi: and V: weathercat file parameters.  I've now completed the conversion of all my WeatherLink data back to 2005. My WLink data generally was at half hour intervals, but not in every case and there were some gaps where the station was shut down and data not saved in the logger, etc.

It took me a couple full evenings, but it's done and WeatherCat is properly reporting my year over year stats as shown here: http:www.oshlo.net/weather/yrsummary.html

Good luck!

Rick