Trixology

WeatherCat => WeatherCat General Discussion => Topic started by: iccb on January 06, 2019, 05:17:46 PM

Title: Editing weather data
Post by: iccb on January 06, 2019, 05:17:46 PM
Hi all!

I had last year problems with Davis wind sensor giving way too much wind (over 50m/s). There values are in data and records. What is the best way to correct those values?
Title: Data files can be edited. (Re: Editing weather data)
Post by: elagache on January 06, 2019, 10:49:11 PM
Dear iccb and WeatherCat "weather editors," . . .

I had last year problems with Davis wind sensor giving way too much wind (over 50m/s). There values are in data and records. What is the best way to correct those values?

For wind data there is basically no choice but to edit the WeatherCat data files themselves.  Fortunately, they are plain text files.  You'll find them in this folder:

~/Library/WeatherCatData/Location1/2018

The ~ is the UNIX convention for your own home folder.  The files have names like this 1_WeatherCatData.cat and look like this:

Code: [Select]
WeatherCat data file ***If you modify this file, be sure to delete the relevent .hrs file***.
t and V are not optional, all other fields are.
t is the day, hour and minute (2 digits each), T is outside temperature, Ti is internal temperature, T1 to T8 is auxiliary temperatures, D is dew point, Pr is barometric pressure, W is wind speed,
Wd is wind direction, Wc is wind chill, Wg is wind gust, Ph is hourly precipitation, P is total precipitation,
H is outside humidity, Hi is internal humidity, H1 to H8 are auxiliary humidity sensors, S is solar, CO21 to CO24 is CO2
Sm1 to Sm4 is soil moisture, Lw1 to Lw4 is leaf wetness, St1 to St4 is soil temperature,
Lt1 to Lt4 is leaf temperature, U is UV, Pm is monthly precipitation, Py is annual precipitation, Ed is daily ET, Em is monthly ET, Ey is yearly ET, C is current conditions (delimited by double quotes), V is validation.

VERS:3

    0 t:010000 T:8.83 Ti:18.94 D:7.12 Pr:1020.7551 W:0.00 Wd:328 Wc:8.83 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:2.07 Sy2:-3.41 Sy3:1.08 Sy4:739.00 Sy5:-7.61 C:"N/A" V:4
    1 t:010001 T:8.78 Ti:18.94 D:7.06 Pr:1020.7551 W:0.00 Wd:328 Wc:8.78 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:739.00 Sy5:-7.61 C:"N/A" V:4
    3 t:010003 T:8.78 Ti:18.94 D:7.06 Pr:1020.8228 W:0.00 Wd:328 Wc:8.78 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:739.00 Sy5:-7.61 C:"N/A" V:4
    5 t:010005 T:8.78 Ti:18.94 D:7.06 Pr:1020.8228 W:0.00 Wd:328 Wc:8.78 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:739.00 Sy5:-7.61 C:"N/A" V:4
    6 t:010006 T:8.78 Ti:19.00 D:7.06 Pr:1020.7551 W:0.00 Wd:328 Wc:8.78 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:622.00 Sy5:-7.56 C:"N/A" V:4
    8 t:010008 T:8.72 Ti:18.94 D:7.01 Pr:1020.7551 W:0.00 Wd:328 Wc:8.72 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:622.00 Sy5:-7.61 C:"N/A" V:4
    9 t:010009 T:8.67 Ti:18.94 D:6.96 Pr:1020.7212 W:0.00 Wd:328 Wc:8.67 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.77 Sy2:-0.00 Sy3:1.08 Sy4:622.00 Sy5:-7.50 C:"N/A" V:4
   10 t:010010 T:8.61 Ti:18.94 D:6.90 Pr:1020.7551 W:0.00 Wd:328 Wc:8.61 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.77 Sy2:-0.00 Sy3:1.08 Sy4:622.00 Sy5:-7.44 C:"N/A" V:4
. . . .

At the top are comments that should guide you in finding the errant data.  As noted in the comments, you must delete the associated  .hrs file to force WeatherCat to recreate the index file for the data file.

Caution is advised for this sort of exercise.  My recommendation is that first:


You can then go through the copies of the data at your leisure.  You can use Apple's TextEdit to edit the files.  However, you might be better served to use a dedicated editor like Text Wrangler.  When you are satisfied that you corrected all the errors, repeat the general procedure:


Let us know if this procedure does the trick for you.

Cheers, Edouard
Title: Re: Editing weather data
Post by: iccb on January 07, 2019, 06:00:35 PM
Thanks for your helpfull post again Edouard!  [snow]

I will try to correct them that way!  ThU32:-)
Title: Re: Editing weather data
Post by: Blicj11 on January 07, 2019, 06:26:50 PM
Iccb

I have done exactly as Edouard outlines and it work perfectly. But, please be careful and only edit what you intend. I used BBedit because TextWrangler is no longer supported. They are from the same developer, and you can get a free 30-day eval of BBEdit, after which, if you do not want to pay for it, its editing functions will still work just fine. BBEdit is much easier for me to make sure I don't make an editing mistake than Apple's TextEdit, although either will work. BBEdit will let you open up your original file in one pane and your file you are editing in a second pane, so you can compare what you are doing.

Let us know how this project turns out.
Title: Re: Editing weather data
Post by: iccb on February 03, 2019, 06:25:26 PM
So, I have checked my data from year 2018 and 2019. First I though that this was very long operation, since there are sooooo much lines to check. Then I downloaded Atom text editor to change my very old paid version of Smultron editor, Smultron is also very good, I even buyed the latest version of Smultron from Mac App Store, but didn't liked that. Then I started to play with Atom, and see that I can open all my files and search from them and replace data that I would like to correct. It was very simply and fast way to do that.
Now I have correct data on my WeatherCat.

Here is also screenshot of Atom processing my files:

Title: Thanks for the tip! (Re: Editing weather data)
Post by: elagache on February 03, 2019, 10:47:44 PM
Dear iccb and WeatherCat station caregivers,

So, I have checked my data from year 2018 and 2019. First I though that this was very long operation, since there are sooooo much lines to check. Then I downloaded Atom text editor to change my very old paid version of Smultron editor, Smultron is also very good, I even buyed the latest version of Smultron from Mac App Store, but didn't liked that. Then I started to play with Atom, and see that I can open all my files and search from them and replace data that I would like to correct. It was very simply and fast way to do that.

Thank you for sharing your experiences with the Atom editor.  I wasn't aware of it and indeed anyone else with a similar sort of data problem might find buying the Atom editor well worth it for the time it would save.

Cheers, Edouard  [cheers1]
Title: Re: Editing weather data
Post by: Blicj11 on February 04, 2019, 08:25:24 AM
Glad to see you have found a way to correct your data. Thanks for sharing info about Atom.
Title: Re: Editing weather data
Post by: iccb on February 04, 2019, 09:23:43 AM
Atom is open source and available free from https://atom.io
I was talking also Smultron, which is paid app from mac app store.
Title: Re: Editing weather data
Post by: jennajon on February 06, 2019, 10:39:38 AM
Atom is open source and available free from https://atom.io
I was talking also Smultron, which is paid app from mac app store.

 [cheer] MANY thanks for this tip...I just spent over 4 hours with Apple on an issue that I ended up fixing myself!!! Could have used this.
TLDR: files in trash undeletable, solution was to boot into recovery and disable system integrity program!! Oh and of course, reenable it.  [banghead]
NEVER EVER USE SAFARI TECHNOLOGY PREVIEW.
Title: Re: Editing weather data
Post by: Blicj11 on February 06, 2019, 05:20:23 PM
jennajon:

I have had this problem in the past. Unfortunately, I cannot remember how I resolved it, but it didn't involve booting into recovery mode. So the point of this unhelpful post is to simply say you aren't the only person to have experienced undeletable files in the trash.
Grrr. 😡
Title: More info on macOS System Integrity Protection (Re: Editing weather data)
Post by: elagache on February 06, 2019, 11:31:00 PM
Dear Jenna, Blick, and WeatherCat sys-admins,

[cheer] MANY thanks for this tip...I just spent over 4 hours with Apple on an issue that I ended up fixing myself!!! Could have used this.
TLDR: files in trash undeletable, solution was to boot into recovery and disable system integrity program!! Oh and of course, reenable it.  [banghead]
NEVER EVER USE SAFARI TECHNOLOGY PREVIEW.

Sorry to hear about your experiences.  Other World Computing has a piece on the macOS System Integrity Protection and how to control it:

https://blog.macsales.com/45473-we-explain-what-system-integrity-protection-on-mac-is-and-how-to-control-it (https://blog.macsales.com/45473-we-explain-what-system-integrity-protection-on-mac-is-and-how-to-control-it)

As mentioned in another post, I've gone back to my pre OS-X defensive strategy.  I now have a completely independent rescue system on another drive for all the important computers of the house (someday all of them.)  That way if I have a computer in trouble, I can boot off of a complete system and use whatever utilities I see fit.  Back in the 1990s it was common to have to boot off of your rescue drive and in those days run Norton Utilities to repair your hard drive.  I'm trying to get back to the same left of capability in dealing with an increasingly capricious operating system.

Cheers, Edouard