Author Topic: Editing weather data  (Read 311 times)

iccb

  • Gentle Breeze
  • **
  • Posts: 55
    • Säätilanne Eurassa
  • Station Details: Mac Mini (late 2012), Davis Vantage Pro2
Editing weather data
« on: January 06, 2019, 05:17:46 PM »
Hi all!

I had last year problems with Davis wind sensor giving way too much wind (over 50m/s). There values are in data and records. What is the best way to correct those values?

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 5097
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, MacBook Pro (Early 2011), macOS 10.13.6, WeatherCat 3
Data files can be edited. (Re: Editing weather data)
« Reply #1 on: January 06, 2019, 10:49:11 PM »
Dear iccb and WeatherCat "weather editors," . . .

I had last year problems with Davis wind sensor giving way too much wind (over 50m/s). There values are in data and records. What is the best way to correct those values?

For wind data there is basically no choice but to edit the WeatherCat data files themselves.  Fortunately, they are plain text files.  You'll find them in this folder:

~/Library/WeatherCatData/Location1/2018

The ~ is the UNIX convention for your own home folder.  The files have names like this 1_WeatherCatData.cat and look like this:

Code: [Select]
WeatherCat data file ***If you modify this file, be sure to delete the relevent .hrs file***.
t and V are not optional, all other fields are.
t is the day, hour and minute (2 digits each), T is outside temperature, Ti is internal temperature, T1 to T8 is auxiliary temperatures, D is dew point, Pr is barometric pressure, W is wind speed,
Wd is wind direction, Wc is wind chill, Wg is wind gust, Ph is hourly precipitation, P is total precipitation,
H is outside humidity, Hi is internal humidity, H1 to H8 are auxiliary humidity sensors, S is solar, CO21 to CO24 is CO2
Sm1 to Sm4 is soil moisture, Lw1 to Lw4 is leaf wetness, St1 to St4 is soil temperature,
Lt1 to Lt4 is leaf temperature, U is UV, Pm is monthly precipitation, Py is annual precipitation, Ed is daily ET, Em is monthly ET, Ey is yearly ET, C is current conditions (delimited by double quotes), V is validation.

VERS:3

    0 t:010000 T:8.83 Ti:18.94 D:7.12 Pr:1020.7551 W:0.00 Wd:328 Wc:8.83 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:2.07 Sy2:-3.41 Sy3:1.08 Sy4:739.00 Sy5:-7.61 C:"N/A" V:4
    1 t:010001 T:8.78 Ti:18.94 D:7.06 Pr:1020.7551 W:0.00 Wd:328 Wc:8.78 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:739.00 Sy5:-7.61 C:"N/A" V:4
    3 t:010003 T:8.78 Ti:18.94 D:7.06 Pr:1020.8228 W:0.00 Wd:328 Wc:8.78 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:739.00 Sy5:-7.61 C:"N/A" V:4
    5 t:010005 T:8.78 Ti:18.94 D:7.06 Pr:1020.8228 W:0.00 Wd:328 Wc:8.78 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:739.00 Sy5:-7.61 C:"N/A" V:4
    6 t:010006 T:8.78 Ti:19.00 D:7.06 Pr:1020.7551 W:0.00 Wd:328 Wc:8.78 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:622.00 Sy5:-7.56 C:"N/A" V:4
    8 t:010008 T:8.72 Ti:18.94 D:7.01 Pr:1020.7551 W:0.00 Wd:328 Wc:8.72 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:622.00 Sy5:-7.61 C:"N/A" V:4
    9 t:010009 T:8.67 Ti:18.94 D:6.96 Pr:1020.7212 W:0.00 Wd:328 Wc:8.67 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.77 Sy2:-0.00 Sy3:1.08 Sy4:622.00 Sy5:-7.50 C:"N/A" V:4
   10 t:010010 T:8.61 Ti:18.94 D:6.90 Pr:1020.7551 W:0.00 Wd:328 Wc:8.61 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.77 Sy2:-0.00 Sy3:1.08 Sy4:622.00 Sy5:-7.44 C:"N/A" V:4
. . . .

At the top are comments that should guide you in finding the errant data.  As noted in the comments, you must delete the associated  .hrs file to force WeatherCat to recreate the index file for the data file.

Caution is advised for this sort of exercise.  My recommendation is that first:

  • Quit WeatherCat
  • Make copies of the data files that you suspect have the errant data.
  • Start WeatherCat back up so you don't lose too much real-time collection.

You can then go through the copies of the data at your leisure.  You can use Apple's TextEdit to edit the files.  However, you might be better served to use a dedicated editor like Text Wrangler.  When you are satisfied that you corrected all the errors, repeat the general procedure:

  • Quit WeatherCat
  • Copy your edited files to replace the ones that have bad data in them
  • Delete the corresponding .hrs files
  • Now you can restart WeatherCat with all data corrected

Let us know if this procedure does the trick for you.

Cheers, Edouard

iccb

  • Gentle Breeze
  • **
  • Posts: 55
    • Säätilanne Eurassa
  • Station Details: Mac Mini (late 2012), Davis Vantage Pro2
Re: Editing weather data
« Reply #2 on: January 07, 2019, 06:00:35 PM »
Thanks for your helpfull post again Edouard!  [snow]

I will try to correct them that way!  ThU32:-)

Blicj11

  • Storm
  • *****
  • Posts: 3115
    • EW3808
    • KUTHEBER6
    • Timber Lakes Utah
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP™ Data Logger | iMac (late 2013), 3.5 GHz Intel Core i7, 24 GB RAM, macOS Mojave | Sharx SCNC2900 Webcam | Supportive Wife
Re: Editing weather data
« Reply #3 on: January 07, 2019, 06:26:50 PM »
Iccb

I have done exactly as Edouard outlines and it work perfectly. But, please be careful and only edit what you intend. I used BBedit because TextWrangler is no longer supported. They are from the same developer, and you can get a free 30-day eval of BBEdit, after which, if you do not want to pay for it, its editing functions will still work just fine. BBEdit is much easier for me to make sure I don't make an editing mistake than Apple's TextEdit, although either will work. BBEdit will let you open up your original file in one pane and your file you are editing in a second pane, so you can compare what you are doing.

Let us know how this project turns out.
Blick


iccb

  • Gentle Breeze
  • **
  • Posts: 55
    • Säätilanne Eurassa
  • Station Details: Mac Mini (late 2012), Davis Vantage Pro2
Re: Editing weather data
« Reply #4 on: February 03, 2019, 06:25:26 PM »
So, I have checked my data from year 2018 and 2019. First I though that this was very long operation, since there are sooooo much lines to check. Then I downloaded Atom text editor to change my very old paid version of Smultron editor, Smultron is also very good, I even buyed the latest version of Smultron from Mac App Store, but didn't liked that. Then I started to play with Atom, and see that I can open all my files and search from them and replace data that I would like to correct. It was very simply and fast way to do that.
Now I have correct data on my WeatherCat.

Here is also screenshot of Atom processing my files:


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 5097
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, MacBook Pro (Early 2011), macOS 10.13.6, WeatherCat 3
Thanks for the tip! (Re: Editing weather data)
« Reply #5 on: February 03, 2019, 10:47:44 PM »
Dear iccb and WeatherCat station caregivers,

So, I have checked my data from year 2018 and 2019. First I though that this was very long operation, since there are sooooo much lines to check. Then I downloaded Atom text editor to change my very old paid version of Smultron editor, Smultron is also very good, I even buyed the latest version of Smultron from Mac App Store, but didn't liked that. Then I started to play with Atom, and see that I can open all my files and search from them and replace data that I would like to correct. It was very simply and fast way to do that.

Thank you for sharing your experiences with the Atom editor.  I wasn't aware of it and indeed anyone else with a similar sort of data problem might find buying the Atom editor well worth it for the time it would save.

Cheers, Edouard  [cheers1]

Blicj11

  • Storm
  • *****
  • Posts: 3115
    • EW3808
    • KUTHEBER6
    • Timber Lakes Utah
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP™ Data Logger | iMac (late 2013), 3.5 GHz Intel Core i7, 24 GB RAM, macOS Mojave | Sharx SCNC2900 Webcam | Supportive Wife
Re: Editing weather data
« Reply #6 on: February 04, 2019, 08:25:24 AM »
Glad to see you have found a way to correct your data. Thanks for sharing info about Atom.
Blick


iccb

  • Gentle Breeze
  • **
  • Posts: 55
    • Säätilanne Eurassa
  • Station Details: Mac Mini (late 2012), Davis Vantage Pro2
Re: Editing weather data
« Reply #7 on: February 04, 2019, 09:23:43 AM »
Atom is open source and available free from https://atom.io
I was talking also Smultron, which is paid app from mac app store.

jennajon

  • Gentle Breeze
  • **
  • Posts: 55
    • FW2882
    • Hallbrook Estates Personal Weather Station
  • Station Details: Davis Vantage Pro2 Plus With FARS, Solar and UV sensors, MAC 27 in. Retina, WeatherCat vers. 3.04 build 91
Re: Editing weather data
« Reply #8 on: February 06, 2019, 10:39:38 AM »
Atom is open source and available free from https://atom.io
I was talking also Smultron, which is paid app from mac app store.

 [cheer] MANY thanks for this tip...I just spent over 4 hours with Apple on an issue that I ended up fixing myself!!! Could have used this.
TLDR: files in trash undeletable, solution was to boot into recovery and disable system integrity program!! Oh and of course, reenable it.  [banghead]
NEVER EVER USE SAFARI TECHNOLOGY PREVIEW.

Blicj11

  • Storm
  • *****
  • Posts: 3115
    • EW3808
    • KUTHEBER6
    • Timber Lakes Utah
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP™ Data Logger | iMac (late 2013), 3.5 GHz Intel Core i7, 24 GB RAM, macOS Mojave | Sharx SCNC2900 Webcam | Supportive Wife
Re: Editing weather data
« Reply #9 on: February 06, 2019, 05:20:23 PM »
jennajon:

I have had this problem in the past. Unfortunately, I cannot remember how I resolved it, but it didn't involve booting into recovery mode. So the point of this unhelpful post is to simply say you aren't the only person to have experienced undeletable files in the trash.
Grrr. 😡
Blick


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 5097
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, MacBook Pro (Early 2011), macOS 10.13.6, WeatherCat 3
More info on macOS System Integrity Protection (Re: Editing weather data)
« Reply #10 on: February 06, 2019, 11:31:00 PM »
Dear Jenna, Blick, and WeatherCat sys-admins,

[cheer] MANY thanks for this tip...I just spent over 4 hours with Apple on an issue that I ended up fixing myself!!! Could have used this.
TLDR: files in trash undeletable, solution was to boot into recovery and disable system integrity program!! Oh and of course, reenable it.  [banghead]
NEVER EVER USE SAFARI TECHNOLOGY PREVIEW.

Sorry to hear about your experiences.  Other World Computing has a piece on the macOS System Integrity Protection and how to control it:

https://blog.macsales.com/45473-we-explain-what-system-integrity-protection-on-mac-is-and-how-to-control-it

As mentioned in another post, I've gone back to my pre OS-X defensive strategy.  I now have a completely independent rescue system on another drive for all the important computers of the house (someday all of them.)  That way if I have a computer in trouble, I can boot off of a complete system and use whatever utilities I see fit.  Back in the 1990s it was common to have to boot off of your rescue drive and in those days run Norton Utilities to repair your hard drive.  I'm trying to get back to the same left of capability in dealing with an increasingly capricious operating system.

Cheers, Edouard