Trixology
WeatherCat => WeatherCat Feature Requests => Topic started by: Ryan Hothersall on August 07, 2018, 10:17:43 AM
-
Myself and many others who have moved to WeatherCat from WeatherLink would dearly like a proper import tool for our historical data.
Sure it?s a one time use thing when you first set WeatherCat up, but if potential users thinking of trying WeatherCat know their data can be imported easily, it may just convince them that WeatherCat is the right software for them.
When I switched recently, I tried a app someone had made for data import, but I don?t think I did it right as all my historical data I tried to import doesn?t look right.
What do others think?.
-
i am wanting the same feature, so I can get rid of Weatherlink
-
1st: Warm welcome to oldcolonel! :)
2nd: I second an import tool for all kind of data (not only Weatherlink), although I personally do not need it. 8)
3rd: Very calm in the forum, we haven't heard anything from WCDev and Administrator a couple of weeks (months?). :-[
Cheers,
-
Dear oldcolonel, Reinhard, and WeatherCat faithful,
Let me join Reinhard in welcoming you to the WeatherCat forum! (http://www.canebas.org/WeatherCat/Forum_support_documents/Custom_emoticons/welcome_smiley.gif)
i am wanting the same feature, so I can get rid of Weatherlink
It is certainly important that everyone who wants this feature make their voice be heard. It is the more popular upgrades that make it into WeatherCat.
3rd: Very calm in the forum, we haven't heard anything from WCDev and Administrator a couple of weeks (months?). :-[
I have to assume that our fearless leader Stu (Stuart Ball, WeatherCat developer) is being really overwhelmed by his day job once more. I wanted to send him a quick email to confirm this over the weekend, but I never found the time myself! . . . :-[
Oh well, . . . . . Edouard
-
Hello everyone,
I'm also new here and considering to move from WeatherLink to another solution. Like everyone we struggle with the import of historial data (since 2006). I would also definitely appreciate an easy tool to take care of this migration. I've tried a tool that has been developped from one of the members. It's easy to use, but the data are not necessarily migrated on the right place.
My trial is close to his end. This is almost the only reason so far keep WeatherLink and to look forward for the right solution.
So why not giving priority to this development as far we seems to many within the same situation.
With all the best ...
-
Dear AlexisT and WeatherCat faithful,
I'm also new here and considering to move from WeatherLink to another solution.
Well let me welcome you to the WeatherCat forum nonetheless! (http://www.canebas.org/WeatherCat/Forum_support_documents/Custom_emoticons/welcome_smiley.gif)
Like everyone we struggle with the import of historial data (since 2006). I would also definitely appreciate an easy tool to take care of this migration. I've tried a tool that has been developped from one of the members. It's easy to use, but the data are not necessarily migrated on the right place.
My trial is close to his end. This is almost the only reason so far keep WeatherLink and to look forward for the right solution.
So why not giving priority to this development as far we seems to many within the same situation.
I wish I knew the answer to your question. WeatherCat evolutionary ancestor was able to import data from a number of weather programs. These importers disappeared in the transition to WeatherCat. I've wondered if Davis might have something to do with that. Davis is pursuing their own cloud-based system and might have put pressure on the 3rd party developers to not mess with their own software.
Before giving up completely, have you thought about contacting the person who wrote the small converter program? If this could be patched up to work properly, that might be the easiest and least disruptive way to make the transition to WeatherCat.
Once more welcome.
Cheers, Edouard [cheers1]
-
To simplify things I deleted my last post and combined it with this one.
A couple of years ago I needed to convert about 8 years worth of WeatherLink data over to WeatherCat.
The predecessor to WeatherCat, Lightsoft Weather Center (LWC), has a data importer app that will convert Davis WeatherLink data to LWC. The LWC data files are compatible with WeatherCat. Download here: https://download.cnet.com/Lightsoft-Weather-Center/3000-2130_4-179028.html
At the time I did this, my Mac was running OS X 10.11 El Capitan. I seem to recall either 10.11 not liking LWC or I didn't want LWC installing files on my current system, so I installed OS X 10.6 in VirtualBox and ran LWC there.
To refresh on how it worked, yesterday I installed OS X 10.6 in VirtualBox and ran LWC.
The data importer is accessible in the LWC menu, Tools, LWC Importer. Once selected, a prompt appeared notifying that to run LWC Importer (a separate app), LWC had to be shut down. For quicker access, I extracted the LWC Importer app from the LWC2.dmg and converted some WeatherLink files from 2011, it saved them to /Users/(username)/Library/LWCData/Location1/2011/
After running the importer the following will need to be done to the converted data files:
The files need to be renamed from *_LWC_Data.lw2 to *_WeatherCat.cat, leave the number at the beginning of the filename.
Replace the header (first 9 lines of text) at the beginning of each file with header lines from a WeatherCat data file. I don't remember if this was required in order for WeatherCat to use the converted data files or not, but I did it as a precaution.
Here is what the data converter looks like running on OS X 10.6:
(http://www.parsonsweather.com/files/sn1.png)
Converted data files:
(http://www.parsonsweather.com/files/sn2.png)
Ryan Hothersall, oldcolonel, wurzelmac, AlexisT and any other WeatherCat users wanting to import their WeatherLink data:
If you have issues or this seems too complex, let me know, we can make arrangements for you to send your WeatherLink data to me and I'll convert it.
-
Thank you very much for sharing this Dave13.
-
Thanks a lot.
I've tried so far. I still have an iMac running on OS X 10.6. But to run LWC I got problems with USB to recognize the Vantage Pro 2. As mentioned in the LWC manual, I've tried to update USB to Serial converters with Silabs VCP drivers. Unfortunately, the current version available for download needs at least OS X 10.9.
I look to find older version of VCP drivers but without success so far. In case you would know how to solve it, thanks by advance.
-
AlexisT,
Connecting your weather station to LWC is not necessary, you can run it with an unconnected station and convert your WeatherLink data.
When I first ran LWC, on the Station Communications prompt, I chose "Davis Vantage/Pro/Pro2...", left the rest blank and hit Done. After that another prompt popped up saying "No Weather Station Found", I selected Run and LWC started. It took about a minute for the LWC main window to appear. A prompt to do the initial setup appeared, I selected No, then went to Tools, LWC Importer.
To extract the LWC Importer and run it directly do the following:
Open the LWC2.1.dmg, click on Lightsoft Weather Center, right-click the LWC2 app and choose "Show Package Contents".
Go to /Contents/Resources/LWCPlugIns/Importer.bundle, right-click and choose "Show Package Contents".
Go to /Contents/Resources, copy the LWC_importer app to your desktop and run it there.
-
Thank you very much for sharing this Dave13.
x2!!
Cheers, Edouard [cheers1]
-
Thanks, thanks, thanks.
It succeeds. I had to run twice, the first time I got error messages on number of days. I took another BU of wlk file, and I got it. I only have a little issue it seems with barometric pressure the first 3 months this year.
Your solution is really working. I had the chance to still have a Mac running on OS X 10.6.
I only have now to change the trial period to a licence.
Great job !!!
PS Nevertheless, it still would be great to have a proper import function directly in the soft.
-
You?re welcome, glad to hear you got your WeatherLink data converted.
You can edit your WeatherCat data files to fix the incorrect barometer data. Don?t use the MacOS TextEdit app, it can change the formatting of the files which can cause issues. Use BBEdit, it will preserve the formatting.
My method of editing is to make a copy of the data file(s) in another folder to edit them, also make an additional backup copy in case something goes wrong. After editing, exit WeatherCat if it is running, replace the original data file(s) with the edited data file(s) and delete the corresponding .hrs file(s). This will trigger WeatherCat to read the edited file(s) when you run it.
BBEdit is a paid app but the basic editing features work in ?free mode?. You can get it in the Mac App Store or on the developer?s website here: https://www.barebones.com/products/bbedit/index.html
-
Dear AlexisT, Dave13, and WeatherCat troubleshooters,
Thanks, thanks, thanks.
It succeeds.
. . . . .
Great job !!!
Glad to hear that you were able to import all that data AlexisT and thanks for all the helpful advice Dave13! ThU32:-)
We like happy endings around here! [bounce]
Cheers, Edouard [cheers1]
-
When I switched to WeatherCat, I had the issue of needing to bring over around 9 years worth of data from Weather Display. Another WeatherCat user on this forum, Hugh Whalen, was more than happy to help.
I was glad to return the favor and help out a new WeatherCat user bring over their existing weather data.
-
To Dave13: [rockon] [woohoo] [cheers1]
Need I say more?
-
In my previous post I explained how to use the LWC software to convert data from WeatherLink, then how to edit and rename these files to use in WeatherCat. If you have several years worth of data, the file editing and renaming can take some time.
Also, I noticed in the WeatherCat data files that there are line numbers. The LWC data importer does not add line numbers, I don't know if WeatherCat will add the line numbers. A typical data file will have a little over 44,000 lines of data if WeatherLink was archiving every minute.
I wrote this script to run in Automator that will quickly edit the header, add line numbers and rename the files, making them ready to use in WeatherCat. In my tests the script took just a little over 2 seconds to complete.
*** I make no guarantees on whether or not this will work for you, I suggest you use copies of your converted data in case something goes wrong. ***
Download the two text files attached to this post and save them to your Desktop.
Create a folder named WX on the Desktop.
Copy all of the folders created by the LWC importer in /Users/(username)/Library/LWCData/Location1/ to the WX folder. There will be a folder for each year of data that was imported from WeatherLink.
Copy the wxcat-header.txt file attached to this post and paste it into each of the folders you copied into the WX folder.
Open the Automator app, choose Application in the document type list. Under Library section, choose Utilities, in the list to the right, double-click on Run Shell Script. Copy/Paste the contents of the WxLink-to-WxCat_script.txt file attached to this post into the Run Shell Script box.
Your automator app should look like the following:
(http://www.parsonsweather.com/files/a1.png)
On the first line of the script, change the year, which is the last directory of the file path, to the first year folder you have of converted data, currently it is set to 2010, so we'll use that in this scenario.
Click run in the top right corner of Automator. The script will edit the header, add line numbers and rename each of the files. Then it will discard the temporary files it created to do the work. Automator will beep when it is finished, all of the data in the 2010 folder will be ready to add to WeatherCat.
Change the year on the first line of the script to the next year, in this scenario, change it to 2011 and run the script. Repeat this for every year (folder) you have imported data from WeatherLink.
The script has 3 commands that repeat 12 times, one for each month. If you have a folder that does not have data for all 12 months, the script will create a 1KB sized file(s) for that month that contains only the header, delete those 1KB file(s).
Now you should have all of your imported WeatherLink data ready to use in WeatherCat.
Here is an explanation of what each line of the script does:
cd ~/Desktop/WX/2010
This sets the directory the rest of the script will run in. The last directory in the file path will be changed and the script ran again for each year (folder) of data to edit.
sed '1,10d' 1_LWC_Data.lw2 > 1_LWC_Data-1.lw2
This removes the LWC header and saves the edited data to a temporary file to use for the next step.
awk '{ print NR" "$0 }' 1_LWC_Data-1.lw2 > 1_LWC_Data-2.lw2
This adds line numbers and saves the edited data to a temporary file to use for the next step.
cat wxcat-header.txt 12_LWC_Data-2.lw2 > temp.txt && mv temp.txt 12_WeatherCat.cat
This inserts the WeatherCat header and saves the edited data to the file that will be used in WeatherCat.
rm *.lw2
rm *.txt
These two commands remove all the temp files used to do the editing.
I hope this helps for anyone wishing to import their WeatherLink data into WeatherCat.
-
Dear Dave and WeatherCat troubleshooters,
Also, I noticed in the WeatherCat data files that there are line numbers. The LWC data importer does not add line numbers, I don't know if WeatherCat will add the line numbers. A typical data file will have a little over 44,000 lines of data if WeatherLink was archiving every minute.
I wrote this script to run in Automator that will quickly edit the header, add line numbers and rename the files, making them ready to use in WeatherCat. In my tests the script took just a little over 2 seconds to complete.
. . . .
Thanks for taking the time to do this! [tup] It may not be necessary because I think WeatherCat retains an importer for LWC data, so LWC files are correctly converted. Still, anything to facilitate the process is definitely appreciated!
Cheers, Edouard [cheers1]
P.S. Your cautions are well warranted. Never use your only copy of data for conversions like this.
-
Edouard,
Thank you for the additional information, to confirm whether or not WeatherCat has an importer for LWC data, I did the following:
In my 10.6 virtual box that I used for testing LWC, I installed version 1.0.2 of WeatherCat, I had to use an old version that supported 10.6. When I ran WeatherCat the first time, it prompted to import the LWC data, and it did so very quickly.
I only confirmed this worked in WeatherCat 1.0.2, don't know if it works in version 2 or 3, but the newer versions may still retain the LWC data import.
If anyone wanting to import converted WeatherLink data from LWC, and it was done on a different Mac than the one they have WeatherCat installed, or ran LWC in a virtual box like I did, try the following:
Copy the Location1 folder the LWC importer creates in /Users/(username)/Library/LWCData/
On the Mac that has WeatherCat installed, exit WeatherCat if it is running.
Go to /Users/(username)/Library/ create a LWCData folder and paste the Location1 folder.
Run WeatherCat, it should detect and prompt if you would like to import the data, tell it to do so, it will import and complete the startup process.
If WeatherCat version 2 or 3 does not detect and do the import, then the script in my previous post can be used to edit the LWC data to use in WeatherCat.
Cheers, Dave [cheers1]
-
Recently decided to have another go at getting my historical data into weathercat.
I managed to borrow an old iMac that is running macOS 10.6.8, so no need to mess about with virtual machines.
Got Lightsoft Weather Centre going, without touching my weather station setup.
These Weatherlink files were created with the windows version.
I am getting an error when I try to import the files.
https://flic.kr/p/2nCznUP
https://flic.kr/p/2nCAJj2
-
I have made some progress...
I overcome the above problem by following the instructions earlier in this thread about going into the contents of the .dmg file for LWC and extracting the importer utility and running it on it's own. Everything worked perfectly and converted the files into the .lwc format.
I then was able to edit the first 9 lines of the text in each file with BBEdit. Changed the file name from the LWC name to X_WeatherCatData.cat (where X is the file number).
I then go into Library>WeatherCatData>Location1. I at first tried starting at my earliest data in 2008 and nothing showed up, so tried the other way from 2017 and the same thing. Data does not show up.
However I do have a 6 month gap in 2018, when I moved everything over from the windows machine to Mac. Would that gap be causing my issues?.
-
Ryan,
Scroll up and take a look at posts #17 and #18, then do the following:
Close WeatherCat, remove the converted files you edited, renamed and put in your WeatherCat data folder. Follow these steps from Post #18 using your unedited LWC data files:
Copy the Location1 folder the LWC importer creates in /Users/(username)/Library/LWCData/
On your WeatherCat Mac, go to /Users/(username)/Library/ create a LWCData folder and paste the Location1 folder.
Run WeatherCat, it should detect and prompt if you would like to import the data, tell it to do so, it will import and complete the startup process.
Hopefully this will import and WeatherCat will see all of your old data. Let us know how it goes.
-
I have sorted it out.
I found that LWC had created files for the months of missing data, so converted those files and put them in the weather cat year folders and the rest of my data is now showing up.
Took a lot of stuffing around, but I have gotten there.
-
Awesome! Glad to hear you now have all your old data imported. ThU5:-) [rockon]
-
Had one file that didn't work.
photo of error.
https://flic.kr/p/2nCNsig
There are a few gaps here and there in the data, but vast majority worked fine.
-
Congrats, Mr. Hothersall!! Well done! ThU5:-)