Author Topic: Proper data import for Davis WeatherLink  (Read 14489 times)

pbeaudet

  • Strong Breeze
  • ***
  • Posts: 198
    • WD6FGE
    • KCABURNE1
    • Burney Weather
  • Station Details: Dedicated Mac Mini 2014, 2.5GHz 8 GB DRAM 12.6.5, Davis Vantage Pro2, Solar, UV, Soil & Leaf, and the wireless Remote Anemometer / SHARX SCNC3905 Camera /Bloomsky 2 with Storm / PurpleAir Air Quality Sensor
Re: Proper data import for Davis WeatherLink
« Reply #15 on: June 15, 2019, 08:27:38 PM »
To Dave13:  [rockon] [woohoo] [cheers1]

Need I say more?

Dave13

  • Gentle Breeze
  • **
  • Posts: 63
    • CW7890
    • KTNPARSO2
    • ParsonsWeather.com
  • Station Details: Davis VP2
Script to edit LWC files (Re: Proper data import for Davis WeatherLink)
« Reply #16 on: June 16, 2019, 10:26:03 PM »
In my previous post I explained how to use the LWC software to convert data from WeatherLink, then how to edit and rename these files to use in WeatherCat. If you have several years worth of data, the file editing and renaming can take some time.

Also, I noticed in the WeatherCat data files that there are line numbers. The LWC data importer does not add line numbers, I don't know if WeatherCat will add the line numbers. A typical data file will have a little over 44,000 lines of data if WeatherLink was archiving every minute.

I wrote this script to run in Automator that will quickly edit the header, add line numbers and rename the files, making them ready to use in WeatherCat. In my tests the script took just a little over 2 seconds to complete.

*** I make no guarantees on whether or not this will work for you, I suggest you use copies of your converted data in case something goes wrong. ***

Download the two text files attached to this post and save them to your Desktop.

Create a folder named WX on the Desktop.

Copy all of the folders created by the LWC importer in /Users/(username)/Library/LWCData/Location1/ to the WX folder. There will be a folder for each year of data that was imported from WeatherLink.

Copy the wxcat-header.txt file attached to this post and paste it into each of the folders you copied into the WX folder.

Open the Automator app, choose Application in the document type list. Under Library section, choose Utilities, in the list to the right, double-click on Run Shell Script. Copy/Paste the contents of the WxLink-to-WxCat_script.txt file attached to this post into the Run Shell Script box.

Your automator app should look like the following:


On the first line of the script, change the year, which is the last directory of the file path, to the first year folder you have of converted data, currently it is set to 2010, so we'll use that in this scenario.

Click run in the top right corner of Automator. The script will edit the header, add line numbers and rename each of the files. Then it will discard the temporary files it created to do the work. Automator will beep when it is finished, all of the data in the 2010 folder will be ready to add to WeatherCat.

Change the year on the first line of the script to the next year, in this scenario, change it to 2011 and run the script. Repeat this for every year (folder) you have imported data from WeatherLink.

The script has 3 commands that repeat 12 times, one for each month. If you have a folder that does not have data for all 12 months, the script will create a 1KB sized file(s) for that month that contains only the header, delete those 1KB file(s).

Now you should have all of your imported WeatherLink data ready to use in WeatherCat.


Here is an explanation of what each line of the script does:

cd ~/Desktop/WX/2010
This sets the directory the rest of the script will run in. The last directory in the file path will be changed and the script ran again for each year (folder) of data to edit.

sed '1,10d' 1_LWC_Data.lw2 > 1_LWC_Data-1.lw2
This removes the LWC header and saves the edited data to a temporary file to use for the next step.

awk '{ print NR" "$0 }' 1_LWC_Data-1.lw2 > 1_LWC_Data-2.lw2
This adds line numbers and saves the edited data to a temporary file to use for the next step.

cat wxcat-header.txt 12_LWC_Data-2.lw2 > temp.txt && mv temp.txt 12_WeatherCat.cat
This inserts the WeatherCat header and saves the edited data to the file that will be used in WeatherCat.

rm *.lw2
rm *.txt
These two commands remove all the temp files used to do the editing.


I hope this helps for anyone wishing to import their WeatherLink data into WeatherCat.

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Thanks for the additional script! (Re: Script to edit LWC files)
« Reply #17 on: June 16, 2019, 11:46:20 PM »
Dear Dave and WeatherCat troubleshooters,

Also, I noticed in the WeatherCat data files that there are line numbers. The LWC data importer does not add line numbers, I don't know if WeatherCat will add the line numbers. A typical data file will have a little over 44,000 lines of data if WeatherLink was archiving every minute.

I wrote this script to run in Automator that will quickly edit the header, add line numbers and rename the files, making them ready to use in WeatherCat. In my tests the script took just a little over 2 seconds to complete.

. . . .

Thanks for taking the time to do this!  [tup]  It may not be necessary because I think WeatherCat retains an importer for LWC data, so LWC files are correctly converted.  Still, anything to facilitate the process is definitely appreciated!

Cheers, Edouard  [cheers1]

P.S. Your cautions are well warranted.  Never use your only copy of data for conversions like this.

Dave13

  • Gentle Breeze
  • **
  • Posts: 63
    • CW7890
    • KTNPARSO2
    • ParsonsWeather.com
  • Station Details: Davis VP2
LWC data to WeatherCat (Re: Script to edit LWC files)
« Reply #18 on: June 17, 2019, 01:33:58 AM »
Edouard,

Thank you for the additional information, to confirm whether or not WeatherCat has an importer for LWC data, I did the following:

In my 10.6 virtual box that I used for testing LWC, I installed version 1.0.2 of WeatherCat, I had to use an old version that supported 10.6. When I ran WeatherCat the first time, it prompted to import the LWC data, and it did so very quickly.

I only confirmed this worked in WeatherCat 1.0.2, don't know if it works in version 2 or 3, but the newer versions may still retain the LWC data import.


If anyone wanting to import converted WeatherLink data from LWC, and it was done on a different Mac than the one they have WeatherCat installed, or ran LWC in a virtual box like I did, try the following:

Copy the Location1 folder the LWC importer creates in /Users/(username)/Library/LWCData/

On the Mac that has WeatherCat installed, exit WeatherCat if it is running.

Go to /Users/(username)/Library/ create a LWCData folder and paste the Location1 folder.

Run WeatherCat, it should detect and prompt if you would like to import the data, tell it to do so, it will import and complete the startup process.


If WeatherCat version 2 or 3 does not detect and do the import, then the script in my previous post can be used to edit the LWC data to use in WeatherCat.


Cheers, Dave  [cheers1]

Ryan Hothersall

  • Gentle Breeze
  • **
  • Posts: 11
  • Station Details: Davis VP2plus with WeatherLink ip. Mac Mini & 27 imac 5k.
Re: Proper data import for Davis WeatherLink
« Reply #19 on: August 06, 2022, 02:26:42 AM »
Recently decided to have another go at getting my historical data into weathercat.

I managed to borrow an old iMac that is running macOS 10.6.8, so no need to mess about with virtual machines.

Got Lightsoft Weather Centre going, without touching my weather station setup.

These Weatherlink files were created with the windows version.

I am getting an error when I try to import the files.

https://flic.kr/p/2nCznUP

https://flic.kr/p/2nCAJj2

Ryan Hothersall

  • Gentle Breeze
  • **
  • Posts: 11
  • Station Details: Davis VP2plus with WeatherLink ip. Mac Mini & 27 imac 5k.
Re: Proper data import for Davis WeatherLink
« Reply #20 on: August 06, 2022, 12:22:53 PM »
I have made some progress...

I overcome the above problem by following the instructions earlier in this thread about going into the contents of the .dmg file for LWC and extracting the importer utility and running it on it's own. Everything worked perfectly and converted the files into the .lwc format.

I then was able to edit the first 9 lines of the text in each file with BBEdit. Changed the file name from the LWC name to X_WeatherCatData.cat (where X is the file number).

I then go into Library>WeatherCatData>Location1. I at first tried starting at my earliest data in 2008 and nothing showed up, so tried the other way from 2017 and the same thing. Data does not show up.

However I do have a 6 month gap in 2018, when I moved everything over from the windows machine to Mac. Would that gap be causing my issues?.

Dave13

  • Gentle Breeze
  • **
  • Posts: 63
    • CW7890
    • KTNPARSO2
    • ParsonsWeather.com
  • Station Details: Davis VP2
Re: Proper data import for Davis WeatherLink
« Reply #21 on: August 06, 2022, 08:51:24 PM »
Ryan,

Scroll up and take a look at posts #17 and #18, then do the following:

Close WeatherCat, remove the converted files you edited, renamed and put in your WeatherCat data folder. Follow these steps from Post #18 using your unedited LWC data files:

Copy the Location1 folder the LWC importer creates in /Users/(username)/Library/LWCData/

On your WeatherCat Mac, go to /Users/(username)/Library/ create a LWCData folder and paste the Location1 folder.

Run WeatherCat, it should detect and prompt if you would like to import the data, tell it to do so, it will import and complete the startup process.

Hopefully this will import and WeatherCat will see all of your old data. Let us know how it goes.

Ryan Hothersall

  • Gentle Breeze
  • **
  • Posts: 11
  • Station Details: Davis VP2plus with WeatherLink ip. Mac Mini & 27 imac 5k.
Re: Proper data import for Davis WeatherLink
« Reply #22 on: August 06, 2022, 09:46:43 PM »
I have sorted it out.

I found that LWC had created files for the months of missing data, so converted those files and put them in the weather cat year folders and the rest of my data is now showing up.

Took a lot of stuffing around, but I have gotten there.

Dave13

  • Gentle Breeze
  • **
  • Posts: 63
    • CW7890
    • KTNPARSO2
    • ParsonsWeather.com
  • Station Details: Davis VP2
Re: Proper data import for Davis WeatherLink
« Reply #23 on: August 06, 2022, 09:54:36 PM »
Awesome! Glad to hear you now have all your old data imported. ThU5:-) [rockon]

Ryan Hothersall

  • Gentle Breeze
  • **
  • Posts: 11
  • Station Details: Davis VP2plus with WeatherLink ip. Mac Mini & 27 imac 5k.
Re: Proper data import for Davis WeatherLink
« Reply #24 on: August 07, 2022, 01:25:32 AM »
Had one file that didn't work.

photo of error.
https://flic.kr/p/2nCNsig

There are a few gaps here and there in the data, but vast majority worked fine.

xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: Proper data import for Davis WeatherLink
« Reply #25 on: August 07, 2022, 04:00:36 AM »
Congrats, Mr. Hothersall!! Well done!  ThU5:-)
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system