Author Topic: Proper data import for Davis WeatherLink  (Read 971 times)

pbeaudet

  • Strong Breeze
  • ***
  • Posts: 191
    • WD6FGE
    • KCABURNE1
    • Burney Weather
  • Station Details: Dedicated Mac Mini, late 2012, 2.5GHz 8 GB DRAM 10.13.6, Davis Vantage Pro2, Solar, UV, Soil & Leaf, and the wireless Remote Anemometer / SHARX SCNC3905 Camera /Bloomsky 2 with Storm / PurpleAir Air Quality Sensor
Re: Proper data import for Davis WeatherLink
« Reply #15 on: June 15, 2019, 08:27:38 PM »
To Dave13:  [rockon] [woohoo] [cheers1]

Need I say more?

Dave13

  • Gentle Breeze
  • **
  • Posts: 30
    • CW7890
    • KTNPARSO2
    • ParsonsWeather.com
  • Station Details: Davis VP2
Script to edit LWC files (Re: Proper data import for Davis WeatherLink)
« Reply #16 on: June 16, 2019, 10:26:03 PM »
In my previous post I explained how to use the LWC software to convert data from WeatherLink, then how to edit and rename these files to use in WeatherCat. If you have several years worth of data, the file editing and renaming can take some time.

Also, I noticed in the WeatherCat data files that there are line numbers. The LWC data importer does not add line numbers, I don't know if WeatherCat will add the line numbers. A typical data file will have a little over 44,000 lines of data if WeatherLink was archiving every minute.

I wrote this script to run in Automator that will quickly edit the header, add line numbers and rename the files, making them ready to use in WeatherCat. In my tests the script took just a little over 2 seconds to complete.

*** I make no guarantees on whether or not this will work for you, I suggest you use copies of your converted data in case something goes wrong. ***

Download the two text files attached to this post and save them to your Desktop.

Create a folder named WX on the Desktop.

Copy all of the folders created by the LWC importer in /Users/(username)/Library/LWCData/Location1/ to the WX folder. There will be a folder for each year of data that was imported from WeatherLink.

Copy the wxcat-header.txt file attached to this post and paste it into each of the folders you copied into the WX folder.

Open the Automator app, choose Application in the document type list. Under Library section, choose Utilities, in the list to the right, double-click on Run Shell Script. Copy/Paste the contents of the WxLink-to-WxCat_script.txt file attached to this post into the Run Shell Script box.

Your automator app should look like the following:


On the first line of the script, change the year, which is the last directory of the file path, to the first year folder you have of converted data, currently it is set to 2010, so we'll use that in this scenario.

Click run in the top right corner of Automator. The script will edit the header, add line numbers and rename each of the files. Then it will discard the temporary files it created to do the work. Automator will beep when it is finished, all of the data in the 2010 folder will be ready to add to WeatherCat.

Change the year on the first line of the script to the next year, in this scenario, change it to 2011 and run the script. Repeat this for every year (folder) you have imported data from WeatherLink.

The script has 3 commands that repeat 12 times, one for each month. If you have a folder that does not have data for all 12 months, the script will create a 1KB sized file(s) for that month that contains only the header, delete those 1KB file(s).

Now you should have all of your imported WeatherLink data ready to use in WeatherCat.


Here is an explanation of what each line of the script does:

cd ~/Desktop/WX/2010
This sets the directory the rest of the script will run in. The last directory in the file path will be changed and the script ran again for each year (folder) of data to edit.

sed '1,10d' 1_LWC_Data.lw2 > 1_LWC_Data-1.lw2
This removes the LWC header and saves the edited data to a temporary file to use for the next step.

awk '{ print NR" "$0 }' 1_LWC_Data-1.lw2 > 1_LWC_Data-2.lw2
This adds line numbers and saves the edited data to a temporary file to use for the next step.

cat wxcat-header.txt 12_LWC_Data-2.lw2 > temp.txt && mv temp.txt 12_WeatherCat.cat
This inserts the WeatherCat header and saves the edited data to the file that will be used in WeatherCat.

rm *.lw2
rm *.txt
These two commands remove all the temp files used to do the editing.


I hope this helps for anyone wishing to import their WeatherLink data into WeatherCat.



elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 5239
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Thanks for the additional script! (Re: Script to edit LWC files)
« Reply #17 on: June 16, 2019, 11:46:20 PM »
Dear Dave and WeatherCat troubleshooters,

Also, I noticed in the WeatherCat data files that there are line numbers. The LWC data importer does not add line numbers, I don't know if WeatherCat will add the line numbers. A typical data file will have a little over 44,000 lines of data if WeatherLink was archiving every minute.

I wrote this script to run in Automator that will quickly edit the header, add line numbers and rename the files, making them ready to use in WeatherCat. In my tests the script took just a little over 2 seconds to complete.

. . . .

Thanks for taking the time to do this!  [tup]  It may not be necessary because I think WeatherCat retains an importer for LWC data, so LWC files are correctly converted.  Still, anything to facilitate the process is definitely appreciated!

Cheers, Edouard  [cheers1]

P.S. Your cautions are well warranted.  Never use your only copy of data for conversions like this.

Dave13

  • Gentle Breeze
  • **
  • Posts: 30
    • CW7890
    • KTNPARSO2
    • ParsonsWeather.com
  • Station Details: Davis VP2
LWC data to WeatherCat (Re: Script to edit LWC files)
« Reply #18 on: June 17, 2019, 01:33:58 AM »
Edouard,

Thank you for the additional information, to confirm whether or not WeatherCat has an importer for LWC data, I did the following:

In my 10.6 virtual box that I used for testing LWC, I installed version 1.0.2 of WeatherCat, I had to use an old version that supported 10.6. When I ran WeatherCat the first time, it prompted to import the LWC data, and it did so very quickly.

I only confirmed this worked in WeatherCat 1.0.2, don't know if it works in version 2 or 3, but the newer versions may still retain the LWC data import.


If anyone wanting to import converted WeatherLink data from LWC, and it was done on a different Mac than the one they have WeatherCat installed, or ran LWC in a virtual box like I did, try the following:

Copy the Location1 folder the LWC importer creates in /Users/(username)/Library/LWCData/

On the Mac that has WeatherCat installed, exit WeatherCat if it is running.

Go to /Users/(username)/Library/ create a LWCData folder and paste the Location1 folder.

Run WeatherCat, it should detect and prompt if you would like to import the data, tell it to do so, it will import and complete the startup process.


If WeatherCat version 2 or 3 does not detect and do the import, then the script in my previous post can be used to edit the LWC data to use in WeatherCat.


Cheers, Dave  [cheers1]