Trixology
WeatherCat => WeatherCat Integration => Topic started by: monymony on August 08, 2013, 02:46:27 PM
-
Hello,
I am currently a Weatherlink user with about six years of data I want to convert. I have looked at some of the options available for conversion and I think I am up for converting my old data into the ".CAT" format used by WeatherCat. As I am in the US, where we have to be different about everything (ugh!), all of my data are in Standard Measurements (i.e. inches, ?F, etc.). Do I need to convert my data to metric to use the Weathercat data format or can I leave my data as is? I noticed in my testing with Weathercat that all data appears to be stored in metric even though my settings are to display information in the US format of measurements.
Any suggestions would be appreciated.
Thanks!
-
Hi MonyMony
first welcome to the forum and yes all has to be converter to metric first that was my only problem when converted all my weather info into WeatherCat not for sure but I think that there was a converter in LWC that would convert WL in to it then when you went to WeatherCat it would convert it for automatic but I could be wrong on this will have to get the developer Stu to give more input on this
cheers
[cheers1]
-
Thanks Randall75 for the info and the welcome message!
I suspected the conversion to metric would be need. I still have not tried getting the old copy of LWC work work at a middle-man for this conversion. Worst case is I write my own conversion formulas and create the new .CAT files manually.
-
monymony,
Welcome to the forum! I never could get a copy of LWC to work for file conversion and finally had to resort to exporting my WeatherLink data to text files on a month to month basis from within WeatherLink. I then developed a semi manual/automatic process to convert the exported WL monthly text data to a file WeatherCat could understand using an Excel spreadsheet to automate the key steps such as date/time conversions, metric units conversions and tagging the final text files with the needed parameter prefixes.
Even so it was a fairly time consuming process for me to wade through 8 years of data (96 monthly data export files). My WeatherLink files had data saved at 30 minute intervals, which translates to about 1440 lines of data per monthly file. Once I had the process memorized, it was fairly easy to accomplish over several evenings while watching the nightly news or whatever. In the end it worked out and I have 8+ years of historical data on my site at http://www.oshlo.net/weather/yrsummary.html
You may will be able to find a better, more automated approach, but I'd be happy to share my process steps, Excel spreadsheet, and a couple months' worth of sample files if you get stuck.
Rick
-
Hi Rick,
Thank you for the offer. I am very familiar with Microsoft Access so I am currently using this as my basic for this conversion.
The fields "Ph is hourly precipitation" and "P is total precipitation" have me stuck and I am hoping you or someone out there can clarify how these fields are used.
- First, I have to believe the format for these fields are in millimeters but cannot find anything to confirm this.
- For Hourly Precip, is this a value for the sum of the past 60 minutes of records? Does this value reset a midnight?
- For Total Precip, what is the starting point... first of the month, start of the day, etc?
Knowing these will help me extract the my data properly. As I mentioned in my original post I have data going back to 2007 on a 5 minute interval so that equates to a lot of data. Any help will be greatly appreciated.
Jay
-
Jay,
I didn't mess with the hourly precipitation (Ph) at all since I didn't have that data in WeatherLink, so I can't help you with that.
The Total Precipitation (P) is the cumulative precipitation from the first of the year expressed in millimeters. I took it out to 2 decimal places since 1/4 mm ~ .01" and we have many days with just a few hundredths of an inch of precipitation here in Breckenridge, CO.
As an example P was set 0 in my first record (midnight) on 1/1/12. It incremented for any time record that recorded any precipitation and finished at 51.82mm in the last record (midnight) for 1/31/12. Then P was 51.82mm for the first record for 2/1/12 and worked its way up to 105.66 for the last record for 2/29/12. WeatherCat then displays this in it's monthly history as 2.04" and 2.12" of precip in Jan & Feb 2012, respectively as you can see on the weather history page of my site.
Lastly, be sure the final file is saved with Unix style LFs and not Classic Mac LFs. I made that mistake initially since Mac was the default in TextWrangler and WeatherCat gagged on the resultant files.
I've attached a copy of the finalized Jan 2012 WeatherCatData.cat file I created for use with WC in case you would like to look through it.
I Hope this helps.
Rick
-
Thanks Rick!
This is very helpful as having a real example gives me a good target of how to convert my data. I am quickly learning that the differences in how WC and WeatherLink store rain data has got to be why there is no official tools to convert. I also would have never thought about the UNIX style formats so thanks for the tip.
I guess I will find out this week if I can pull this conversion off.
Wish me luck!
Jay
-
Ok, good luck! [bounce]
Let me know how it goes.
Rick
-
So far I have gotten 2010 thru current data over. I have found out that there are small issues with the way WeatherLink exports data. For example the rain totals for some days when totaled from the WeatherLink export do not match what is displayed in the Weatherlink program. This is before any conversions to metric so I cannot explain it. For me I am going to convert the data I have and then use the daily rain editor in WC to adjust my totals. Otherwise with what turned out to be 650k records over 6+ years I would go completely crazy.
Thanks again Rick for your examples... they were very helpful.
After conversion comes the webpage!
-
Quick update... I was able to get all of the data converted. I did make a small mistake with the 1h Rain total in my query but that appears to be the only case of 'user error' and is easily fixed.
I am now completely cut over to WC and my WeatherLink days are behind me.
Jay
-
[cheer]
Congrats Jay,
Glad you were able to get it done. Keep notes because others may ask!
Regards,
Rick
-
Rick,
Any chance you can share you speadshet and procedure?
Steve
-
Steve:
Sorry, can't help you WeatherLink but wanted to welcome you to the WeatherCat forum. It's a helpful place and someone else is bound to have done what you are attempting.
-
Steve,
Here you are. This is the brief documentation I wrote back at the time; however, it's been a while since I've done it, so you are on your own for the most part. Hopefully this will give you a start.
Regards,
Rick
-
Rick,
Thanks for the info. I'm actually converting from Weather Display so I'm still trying to figure out how to do that. I've been saving all my data to a MySQL data base since 2005. I already wrote a PHP script to move the data from my WD database to the WC one. Now I'm going to write one to create the WeatherCatData.cat files using the data from my WD database. I've got the outline done but was hung up on how to calculate the number at the beginning of each line, the record code I guess it's called. Thanks to your file, I now have that formula. I'll let everyone know if I get it to work.
Steve
-
My script works!!! If you have been using Weather Display and sending the data to a MySQL database you can run this script, do some copy and pasting, and be good to go. I've only done 2013 so far and it took about 15 minutes.
On the line with the mysql_connect statement, change DB path to the path to your DB, user to your MySQL user name and password to your password. On the next line, mysql_select_db("dbname",$db);, change dbname to your database. And last, change table in $equipsearch = "SELECT * FROM table ORDER BY month, day, hour, minute"; to your table.
Run the script, it could take awhile. The output will be by year and month. Copy the data, a month at a time, and paste it in to the proper .cat file, i.e. 10_WeatherCatData.cat in the 2013 folder. restart Weathercat and you should be good to go.
-
Hi Steve,
Welcome to the WeatherCat forum, Steve, and thanks for putting this together!
Stu probably has a better handle on requests, but to me it seems like an import tool for bringing in data from either another application of from Weather Underground is the number one feature request. Often you see someone ask about it in their first post, and then never hear from them again when they find out that you can't. You have to assume they went on to buy a different weather station software product.
Were you previously using the Mac or PC version of Weather Display. I'd assume that Brian would write the MySQL database the same for both, but was just curious.
Thanks!
Steve
-
Steve,
I had been using the PC version for years then tried the Mac version when I got my iMac a couple weeks ago. The Mac version was nice, but just seemed like a bad port of the PC one. The MySQL database seemed to be different on the Mac. When I started it up, it told me I was missing colomns, I don't remember which ones, so I figured I could live without the database. Then when I upgraded to Mavericks, Weather Display would suddenly stop gathering data. It thought it was but the numbers were never changing. I tried all sorst of fixes, reinstalling WD, the USB drivers, taking a hub off, but nothing worked. That's when I discovered WeatherCat.
At first I was looking at all the log files that WD created and tried to figure out a way to parse them and create the cat files but was overwhelmed by the amount of data. After I wrote the script to convert the database over it hit me that I could create the cat files from the database, like some people have done with Wunderground.
You do see a lot of people asking for a converter and other people saying they made a database or did it manually, but they don't say how or post the file. Obviously they will if you ask, like Rick did, but I figured it would be helpful to just post it. If anyone has a DB other than WD that they would like a script to convert, either to the WC database or cat file, I'd be more than happy to modify the one's I've writen for them.
Steve
-
Hi Steve
First welcome to WeatherCat
And yes I would love to have one that would convert my WeatherGround data to WeatherCat cat files so if you can get around to it please do
I set down and manually converted 1 year but it took me for ever having to convert everything metric then putting it in each column manually
Thanks
-
If anyone downloaded the file I posted, ther was a typo in the math for the dew point. It's has been corrected and updated above.
-
Randall,
My script would reqiure the data in be in a MySQL database. WeatherUnderground would be in a csv file I believe. If you have downloaded all the data you need, you can send me the file and I'll make a database out of it and see what I can do.
Steve
-
Here is the PHP script to convert a Virtual Weather System MySQL database to the .cat format.
-
I have added the ability to create the files directly on your web server, as long as it's Unix based. The files below will still output to the web browser, but will also create a directory on on your server for each year processed and the monthly file for that year. Once it's done, just copy all the directories and files over to where ever you have WeatherCat storing your .cat files. Restart WC and you should be good to go.
-
Nice work [tup]
-
Hey Seve
You wouldn't happen to have a converter from Excel to .cat files do you? [banghead] see attahed
Can these be converter into WeatherCat.cat files
thanks
cheers
[cheers1]
-
Randall,
Which file do you want converted, the CSV or SQL, or both?
The good news is, I have done the CSV, which looks like Wunderground data. At some point, I may pretty it up and have the script ask for data fields and units, but for now it's easier to write them for specific files.
To run the script, you need to upload the script and the csv file to the same directory on your server and run the script. The csv file needs to be called filename.csv.
If you want the sql converted too, let me know and I'll do it next week.
By the way, these scripts aren't pretty but they get the job done.
Steve
-
Hi Steve
Thanks Steve I will play with this one and see how it goes
but I think the one for sql that I have would work netter let me try this and see what happens
thanks again