Author Topic: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)  (Read 13834 times)

monymony

  • Strong Breeze
  • ***
  • Posts: 164
  • CoCoRaHS - MD-CC-16
    • EW3643
    • KMDELKTO10
    • Cecil County Maryland Weather
  • Station Details: Davis Vantage Pro 2
Hello,

I am currently a Weatherlink user with about six years of data I want to convert.  I have looked at some of the options available for conversion and I think I am up for converting my old data into the ".CAT" format used by WeatherCat.  As I am in the US, where we have to be different about everything (ugh!), all of my data are in Standard Measurements (i.e. inches, ºF, etc.).  Do I need to convert my data to metric to use the Weathercat data format or can I leave my data as is?  I noticed in my testing with Weathercat that all data appears to be stored in metric even though my settings are to display information in the US format of measurements.

Any suggestions would be appreciated.

Thanks!



Randall75

  • Storm
  • *****
  • Posts: 1205
  • CWOP-CW6734 WeatherUnderground-KOHNEWAR6
    • CW6734
    • KOHNEWAR6
    • Randy's Weather On The Hill
  • Station Details: Davis Vantage Pro 2 Plus.iMac i5 Sierra 10.12.6 8GB Ram, WeatherCat 3,Logitech 9000 Pro Web Cam
Re: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)
« Reply #1 on: August 08, 2013, 04:41:45 PM »
Hi MonyMony
 first welcome to the forum and yes all has to be converter to metric first that was my only problem when converted all my weather info into WeatherCat not for sure but I think that there was a converter in LWC that would convert WL in to it then when you went to WeatherCat it would convert it for automatic but I could be wrong on this will have to get the developer Stu  to give more input on this
cheers
 [cheers1]

monymony

  • Strong Breeze
  • ***
  • Posts: 164
  • CoCoRaHS - MD-CC-16
    • EW3643
    • KMDELKTO10
    • Cecil County Maryland Weather
  • Station Details: Davis Vantage Pro 2
Re: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)
« Reply #2 on: August 08, 2013, 06:47:39 PM »
Thanks Randall75 for the info and the welcome message!

I suspected the conversion to metric would be need.  I still have not tried getting the old copy of LWC work work at a middle-man for this conversion.  Worst case is I write my own conversion formulas and create the new .CAT files manually.



ELO

  • Gentle Breeze
  • **
  • Posts: 67
    • KCOBRECK14
    • Weather in the Highlands
  • Station Details: Old Davis WMII, Heated Rain Collector, Mac Mini
Re: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)
« Reply #3 on: August 08, 2013, 09:44:26 PM »
monymony,

Welcome to the forum! I never could get a copy of LWC to work for file conversion and finally had to resort to exporting my WeatherLink data to text files on a month to month basis from within WeatherLink. I then developed a semi manual/automatic process to convert the exported WL monthly text data to a file WeatherCat could understand using an Excel spreadsheet to automate the key steps such as date/time conversions, metric units conversions and tagging the final text files with the needed parameter prefixes.

Even so it was a fairly time consuming process for me to wade through 8 years of data (96 monthly data export files). My WeatherLink files had data saved at 30 minute intervals, which translates to about 1440 lines of data per monthly file. Once I had the process memorized, it was fairly easy to accomplish over several evenings while watching the nightly news or whatever. In the end it worked out and I have 8+ years of historical data on my site at http://www.oshlo.net/weather/yrsummary.html

You may will be able to find a better, more automated approach, but I'd be happy to share my process steps, Excel spreadsheet, and a couple months' worth of sample files if you get stuck.

Rick

monymony

  • Strong Breeze
  • ***
  • Posts: 164
  • CoCoRaHS - MD-CC-16
    • EW3643
    • KMDELKTO10
    • Cecil County Maryland Weather
  • Station Details: Davis Vantage Pro 2
Re: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)
« Reply #4 on: August 09, 2013, 03:24:30 AM »
Hi Rick,

Thank you for the offer.  I am very familiar with Microsoft Access so I am currently using this as my basic for this conversion. 

The fields "Ph is hourly precipitation" and  "P is total precipitation" have me stuck and I am hoping you or someone out there can clarify how these fields are used.

- First, I have to believe the format for these fields are in millimeters but cannot find anything to confirm this. 
- For Hourly Precip, is this a value for the sum of the past 60 minutes of records?  Does this value reset a midnight?
- For Total Precip, what is the starting point... first of the month, start of the day, etc?

Knowing these will help me extract the my data properly.  As I mentioned in my original post I have data going back to 2007 on a 5 minute interval so that equates to a lot of data.  Any help will be greatly appreciated.

Jay



ELO

  • Gentle Breeze
  • **
  • Posts: 67
    • KCOBRECK14
    • Weather in the Highlands
  • Station Details: Old Davis WMII, Heated Rain Collector, Mac Mini
Re: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)
« Reply #5 on: August 10, 2013, 03:39:02 AM »
Jay,

I didn't mess with the hourly precipitation (Ph) at all since I didn't have that data in WeatherLink, so I can't help you with that.

The Total Precipitation (P) is the cumulative precipitation from the first of the year expressed in millimeters. I took it out to 2 decimal places since 1/4 mm ~ .01" and we have many days with just a few hundredths of an inch of precipitation here in Breckenridge, CO.

As an example P was set 0 in my first record (midnight) on 1/1/12. It incremented for any time record that recorded any precipitation and finished at 51.82mm in the last record (midnight) for 1/31/12. Then P was 51.82mm for the first record for 2/1/12 and worked its way up to 105.66 for the last record for 2/29/12. WeatherCat then displays this in it's monthly history as 2.04" and 2.12" of precip in Jan & Feb 2012, respectively as you can see on the weather history page of my site.

Lastly, be sure the final file is saved with Unix style LFs and not Classic Mac LFs. I made that mistake initially since Mac was the default in TextWrangler and WeatherCat gagged on the resultant files.

I've attached a copy of the finalized Jan 2012 WeatherCatData.cat file I created for use with WC in case you would like to look through it.

I Hope this helps.

Rick

monymony

  • Strong Breeze
  • ***
  • Posts: 164
  • CoCoRaHS - MD-CC-16
    • EW3643
    • KMDELKTO10
    • Cecil County Maryland Weather
  • Station Details: Davis Vantage Pro 2
Re: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)
« Reply #6 on: August 12, 2013, 11:55:45 AM »
Thanks Rick!

This is very helpful as having a real example gives me a good target of how to convert my data.  I am quickly learning that the differences in how WC and WeatherLink store rain data has got to be why there is no official tools to convert.  I also would have never thought about the UNIX style formats so thanks for the tip. 

I guess I will find out this week if I can pull this conversion off.

Wish me luck!

Jay



ELO

  • Gentle Breeze
  • **
  • Posts: 67
    • KCOBRECK14
    • Weather in the Highlands
  • Station Details: Old Davis WMII, Heated Rain Collector, Mac Mini
Re: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)
« Reply #7 on: August 13, 2013, 02:36:03 AM »
Ok, good luck!  [bounce]

Let me know how it goes.

Rick

monymony

  • Strong Breeze
  • ***
  • Posts: 164
  • CoCoRaHS - MD-CC-16
    • EW3643
    • KMDELKTO10
    • Cecil County Maryland Weather
  • Station Details: Davis Vantage Pro 2
Re: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)
« Reply #8 on: August 13, 2013, 02:52:07 AM »
So far I have gotten 2010 thru current data over.  I have found out that there are small issues with the way WeatherLink exports data.  For example the rain totals for some days when totaled from the WeatherLink export do not match what is displayed in the Weatherlink program.  This is before any conversions to metric so I cannot explain it.  For me I am going to convert the data I have and then use the daily rain editor in WC to adjust my totals.  Otherwise with what turned out to be 650k records over 6+ years I would go completely crazy. 

Thanks again Rick for your examples... they were very helpful.

After conversion comes the webpage!



monymony

  • Strong Breeze
  • ***
  • Posts: 164
  • CoCoRaHS - MD-CC-16
    • EW3643
    • KMDELKTO10
    • Cecil County Maryland Weather
  • Station Details: Davis Vantage Pro 2
Re: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)
« Reply #9 on: August 14, 2013, 09:02:20 PM »
Quick update... I was able to get all of the data converted.  I did make a small mistake with the 1h Rain total in my query but that appears to be the only case of 'user error' and is easily fixed.

I am now completely cut over to WC and my WeatherLink days are behind me.

Jay



ELO

  • Gentle Breeze
  • **
  • Posts: 67
    • KCOBRECK14
    • Weather in the Highlands
  • Station Details: Old Davis WMII, Heated Rain Collector, Mac Mini
Re: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)
« Reply #10 on: August 15, 2013, 03:34:33 AM »
 [cheer]

Congrats Jay,

Glad you were able to get it done. Keep notes because others may ask!

Regards,
Rick

AstroSteve

  • Gentle Breeze
  • **
  • Posts: 42
    • KJ6IIW
    • KCAACTON3
  • Station Details: Davis Vantage Pro 2 Wireless, iMac 27"
Re: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)
« Reply #11 on: October 29, 2013, 08:30:24 PM »
Rick,

Any chance you can share you speadshet and procedure?

Steve

Blicj11

  • Storm
  • *****
  • Posts: 3029
    • EW3808
    • KUTHEBER6
    • Timber Lakes Utah
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP™ Data Logger | iMac (late 2013), 3.5 GHz Intel Core i7, 24 GB RAM, macOS Mojave | Sharx SCNC2900 Webcam | Supportive Wife
Re: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)
« Reply #12 on: October 29, 2013, 09:50:26 PM »
Steve:

Sorry, can't help you WeatherLink but wanted to welcome you to the WeatherCat forum. It's a helpful place and someone else is bound to have done what you are attempting.
Blick


ELO

  • Gentle Breeze
  • **
  • Posts: 67
    • KCOBRECK14
    • Weather in the Highlands
  • Station Details: Old Davis WMII, Heated Rain Collector, Mac Mini
Re: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)
« Reply #13 on: October 30, 2013, 12:24:18 AM »
Steve,

Here you are. This is the brief documentation I wrote back at the time; however, it's been a while since I've done it, so you are on your own for the most part. Hopefully this will give you a start.

Regards,
Rick


AstroSteve

  • Gentle Breeze
  • **
  • Posts: 42
    • KJ6IIW
    • KCAACTON3
  • Station Details: Davis Vantage Pro 2 Wireless, iMac 27"
Re: Importing Data from Weatherlink - Datafile Format (Standard v. Metric)
« Reply #14 on: October 30, 2013, 12:48:20 AM »
Rick,

Thanks for the info.  I'm actually converting from Weather Display so I'm still trying to figure out how to do that.  I've been saving all my data to a MySQL data base since 2005.  I already wrote a PHP script to move the data from my WD database to the WC one. Now I'm going to write one to create the WeatherCatData.cat files using the data from my WD database. I've got the outline done but was hung up on how to calculate the number at the beginning of each line, the record code I guess it's called. Thanks to your file, I now have that formula. I'll let everyone know if I get it to work.

Steve