Author Topic: Editing weather data  (Read 722 times)

iccb

  • Gentle Breeze
  • **
  • Posts: 56
    • Säätilanne Eurassa
  • Station Details: Mac Mini (late 2012), Davis Vantage Pro2
Editing weather data
« on: January 06, 2019, 05:17:46 PM »
Hi all!

I had last year problems with Davis wind sensor giving way too much wind (over 50m/s). There values are in data and records. What is the best way to correct those values?

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 5237
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Data files can be edited. (Re: Editing weather data)
« Reply #1 on: January 06, 2019, 10:49:11 PM »
Dear iccb and WeatherCat "weather editors," . . .

I had last year problems with Davis wind sensor giving way too much wind (over 50m/s). There values are in data and records. What is the best way to correct those values?

For wind data there is basically no choice but to edit the WeatherCat data files themselves.  Fortunately, they are plain text files.  You'll find them in this folder:

~/Library/WeatherCatData/Location1/2018

The ~ is the UNIX convention for your own home folder.  The files have names like this 1_WeatherCatData.cat and look like this:

Code: [Select]
WeatherCat data file ***If you modify this file, be sure to delete the relevent .hrs file***.
t and V are not optional, all other fields are.
t is the day, hour and minute (2 digits each), T is outside temperature, Ti is internal temperature, T1 to T8 is auxiliary temperatures, D is dew point, Pr is barometric pressure, W is wind speed,
Wd is wind direction, Wc is wind chill, Wg is wind gust, Ph is hourly precipitation, P is total precipitation,
H is outside humidity, Hi is internal humidity, H1 to H8 are auxiliary humidity sensors, S is solar, CO21 to CO24 is CO2
Sm1 to Sm4 is soil moisture, Lw1 to Lw4 is leaf wetness, St1 to St4 is soil temperature,
Lt1 to Lt4 is leaf temperature, U is UV, Pm is monthly precipitation, Py is annual precipitation, Ed is daily ET, Em is monthly ET, Ey is yearly ET, C is current conditions (delimited by double quotes), V is validation.

VERS:3

    0 t:010000 T:8.83 Ti:18.94 D:7.12 Pr:1020.7551 W:0.00 Wd:328 Wc:8.83 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:2.07 Sy2:-3.41 Sy3:1.08 Sy4:739.00 Sy5:-7.61 C:"N/A" V:4
    1 t:010001 T:8.78 Ti:18.94 D:7.06 Pr:1020.7551 W:0.00 Wd:328 Wc:8.78 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:739.00 Sy5:-7.61 C:"N/A" V:4
    3 t:010003 T:8.78 Ti:18.94 D:7.06 Pr:1020.8228 W:0.00 Wd:328 Wc:8.78 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:739.00 Sy5:-7.61 C:"N/A" V:4
    5 t:010005 T:8.78 Ti:18.94 D:7.06 Pr:1020.8228 W:0.00 Wd:328 Wc:8.78 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:739.00 Sy5:-7.61 C:"N/A" V:4
    6 t:010006 T:8.78 Ti:19.00 D:7.06 Pr:1020.7551 W:0.00 Wd:328 Wc:8.78 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:622.00 Sy5:-7.56 C:"N/A" V:4
    8 t:010008 T:8.72 Ti:18.94 D:7.01 Pr:1020.7551 W:0.00 Wd:328 Wc:8.72 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.76 Sy2:-0.00 Sy3:1.08 Sy4:622.00 Sy5:-7.61 C:"N/A" V:4
    9 t:010009 T:8.67 Ti:18.94 D:6.96 Pr:1020.7212 W:0.00 Wd:328 Wc:8.67 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.77 Sy2:-0.00 Sy3:1.08 Sy4:622.00 Sy5:-7.50 C:"N/A" V:4
   10 t:010010 T:8.61 Ti:18.94 D:6.90 Pr:1020.7551 W:0.00 Wd:328 Wc:8.61 Wg:0.00 Ph:0.00 P:0.00 H:89 Hi:59 S:0 U:0.0 T1:8.9 H1:90.0 Lw4:0.0 Pm:0.00 Py:0.00 Ed:0.00 Em:0.00 Ey:0.00 Sy1:1.77 Sy2:-0.00 Sy3:1.08 Sy4:622.00 Sy5:-7.44 C:"N/A" V:4
. . . .

At the top are comments that should guide you in finding the errant data.  As noted in the comments, you must delete the associated  .hrs file to force WeatherCat to recreate the index file for the data file.

Caution is advised for this sort of exercise.  My recommendation is that first:

  • Quit WeatherCat
  • Make copies of the data files that you suspect have the errant data.
  • Start WeatherCat back up so you don't lose too much real-time collection.

You can then go through the copies of the data at your leisure.  You can use Apple's TextEdit to edit the files.  However, you might be better served to use a dedicated editor like Text Wrangler.  When you are satisfied that you corrected all the errors, repeat the general procedure:

  • Quit WeatherCat
  • Copy your edited files to replace the ones that have bad data in them
  • Delete the corresponding .hrs files
  • Now you can restart WeatherCat with all data corrected

Let us know if this procedure does the trick for you.

Cheers, Edouard

iccb

  • Gentle Breeze
  • **
  • Posts: 56
    • Säätilanne Eurassa
  • Station Details: Mac Mini (late 2012), Davis Vantage Pro2
Re: Editing weather data
« Reply #2 on: January 07, 2019, 06:00:35 PM »
Thanks for your helpfull post again Edouard!  [snow]

I will try to correct them that way!  ThU32:-)

Blicj11

  • Storm
  • *****
  • Posts: 3203
    • EW3808
    • KUTHEBER6
    • Timber Lakes Utah
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP™ Data Logger | iMac (late 2013), 3.5 GHz Intel Core i7, 24 GB RAM, macOS Mojave | Sharx SCNC2900 Webcam | Supportive Wife
Re: Editing weather data
« Reply #3 on: January 07, 2019, 06:26:50 PM »
Iccb

I have done exactly as Edouard outlines and it work perfectly. But, please be careful and only edit what you intend. I used BBedit because TextWrangler is no longer supported. They are from the same developer, and you can get a free 30-day eval of BBEdit, after which, if you do not want to pay for it, its editing functions will still work just fine. BBEdit is much easier for me to make sure I don't make an editing mistake than Apple's TextEdit, although either will work. BBEdit will let you open up your original file in one pane and your file you are editing in a second pane, so you can compare what you are doing.

Let us know how this project turns out.
Blick


iccb

  • Gentle Breeze
  • **
  • Posts: 56
    • Säätilanne Eurassa
  • Station Details: Mac Mini (late 2012), Davis Vantage Pro2
Re: Editing weather data
« Reply #4 on: February 03, 2019, 06:25:26 PM »
So, I have checked my data from year 2018 and 2019. First I though that this was very long operation, since there are sooooo much lines to check. Then I downloaded Atom text editor to change my very old paid version of Smultron editor, Smultron is also very good, I even buyed the latest version of Smultron from Mac App Store, but didn't liked that. Then I started to play with Atom, and see that I can open all my files and search from them and replace data that I would like to correct. It was very simply and fast way to do that.
Now I have correct data on my WeatherCat.

Here is also screenshot of Atom processing my files:


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 5237
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Thanks for the tip! (Re: Editing weather data)
« Reply #5 on: February 03, 2019, 10:47:44 PM »
Dear iccb and WeatherCat station caregivers,

So, I have checked my data from year 2018 and 2019. First I though that this was very long operation, since there are sooooo much lines to check. Then I downloaded Atom text editor to change my very old paid version of Smultron editor, Smultron is also very good, I even buyed the latest version of Smultron from Mac App Store, but didn't liked that. Then I started to play with Atom, and see that I can open all my files and search from them and replace data that I would like to correct. It was very simply and fast way to do that.

Thank you for sharing your experiences with the Atom editor.  I wasn't aware of it and indeed anyone else with a similar sort of data problem might find buying the Atom editor well worth it for the time it would save.

Cheers, Edouard  [cheers1]

Blicj11

  • Storm
  • *****
  • Posts: 3203
    • EW3808
    • KUTHEBER6
    • Timber Lakes Utah
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP™ Data Logger | iMac (late 2013), 3.5 GHz Intel Core i7, 24 GB RAM, macOS Mojave | Sharx SCNC2900 Webcam | Supportive Wife
Re: Editing weather data
« Reply #6 on: February 04, 2019, 08:25:24 AM »
Glad to see you have found a way to correct your data. Thanks for sharing info about Atom.
Blick


iccb

  • Gentle Breeze
  • **
  • Posts: 56
    • Säätilanne Eurassa
  • Station Details: Mac Mini (late 2012), Davis Vantage Pro2
Re: Editing weather data
« Reply #7 on: February 04, 2019, 09:23:43 AM »
Atom is open source and available free from https://atom.io
I was talking also Smultron, which is paid app from mac app store.

jennajon

  • Gentle Breeze
  • **
  • Posts: 82
    • FW2882
    • Hallbrook Estates Personal Weather Station
  • Station Details: Davis Vantage Pro2 Plus With FARS, Solar and UV sensors, MAC 27 in. Retina, WeatherCat vers. 3.04 build 91
Re: Editing weather data
« Reply #8 on: February 06, 2019, 10:39:38 AM »
Atom is open source and available free from https://atom.io
I was talking also Smultron, which is paid app from mac app store.

 [cheer] MANY thanks for this tip...I just spent over 4 hours with Apple on an issue that I ended up fixing myself!!! Could have used this.
TLDR: files in trash undeletable, solution was to boot into recovery and disable system integrity program!! Oh and of course, reenable it.  [banghead]
NEVER EVER USE SAFARI TECHNOLOGY PREVIEW.

Blicj11

  • Storm
  • *****
  • Posts: 3203
    • EW3808
    • KUTHEBER6
    • Timber Lakes Utah
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP™ Data Logger | iMac (late 2013), 3.5 GHz Intel Core i7, 24 GB RAM, macOS Mojave | Sharx SCNC2900 Webcam | Supportive Wife
Re: Editing weather data
« Reply #9 on: February 06, 2019, 05:20:23 PM »
jennajon:

I have had this problem in the past. Unfortunately, I cannot remember how I resolved it, but it didn't involve booting into recovery mode. So the point of this unhelpful post is to simply say you aren't the only person to have experienced undeletable files in the trash.
Grrr. 😡
Blick


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 5237
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
More info on macOS System Integrity Protection (Re: Editing weather data)
« Reply #10 on: February 06, 2019, 11:31:00 PM »
Dear Jenna, Blick, and WeatherCat sys-admins,

[cheer] MANY thanks for this tip...I just spent over 4 hours with Apple on an issue that I ended up fixing myself!!! Could have used this.
TLDR: files in trash undeletable, solution was to boot into recovery and disable system integrity program!! Oh and of course, reenable it.  [banghead]
NEVER EVER USE SAFARI TECHNOLOGY PREVIEW.

Sorry to hear about your experiences.  Other World Computing has a piece on the macOS System Integrity Protection and how to control it:

https://blog.macsales.com/45473-we-explain-what-system-integrity-protection-on-mac-is-and-how-to-control-it

As mentioned in another post, I've gone back to my pre OS-X defensive strategy.  I now have a completely independent rescue system on another drive for all the important computers of the house (someday all of them.)  That way if I have a computer in trouble, I can boot off of a complete system and use whatever utilities I see fit.  Back in the 1990s it was common to have to boot off of your rescue drive and in those days run Norton Utilities to repair your hard drive.  I'm trying to get back to the same left of capability in dealing with an increasingly capricious operating system.

Cheers, Edouard

xairbusdriver

  • Storm
  • *****
  • Posts: 2303
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.14.4/WC 3.0.4
Re: Editing weather data
« Reply #11 on: April 16, 2019, 01:33:57 AM »
We've had a lot of rain over the past Fri, Sat and just a wee bit Sun. There has been a flood advisory out for over 30 days!! :o Beautiful Sun all today! [cheer]

However, after the Mac emitted its "You-just-got-a-probably-urgent-email!" chime and I checked for new missives, I discovered that it was WC telling me that the bucket had tipped, again. After the second time I decided to see what was urinating in the rain collector (which has the bird spikes/lightening arrestors).

Turns out it was not an animal but a vegetative problem. There was the usual collection of pollen, but there was also a nearly perfectly round seed pod from either the Maple or the Crepe Myrtle almost perfectly sealing the collectors drain! The water level in the collector was within an inch of the top! I can't rule out that the seed pod placement might have been from a bird carrying the seed and then opening its beak in a cry of surprise when attempting to alight on the spikes. [lol] At any rate, I "pulled the plug" and now have an all-time high rain rate: "26.18 in/hr at 3:36 pm"! I assume that rate is possibly a local record, I'll contact the local NWS office tomorrow to see if there is any kind of reward...

I will also be doing some table editing tomorrow. [rolleyes2] I hope I can find the 'rate' value as easily as the amount values, hopefully I can simply delete any 'rate' above the 5.33 earlier this month. I'm pretty sure the several short but heavy rains last week never got above 1"/per hr.

Any suggestions for spreading the total rain over this fairly short time span? It's seen in todays total as only 1.14". I think simply adding a few hundredths over the 48 hours would be best? I can't even be sure of any zero amounts being accurate. I simply don't know when the drain became clogged... [coffee]

I'll also be checking the collector after every sprinkle for the next few months... [rolleyes2] [banghead]

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 5237
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Been there . . . . .(Re: Editing weather data)
« Reply #12 on: April 16, 2019, 10:16:03 PM »
Dear X-Air and WeatherCat station caregivers,

. . . . .
However, after the Mac emitted its "You-just-got-a-probably-urgent-email!" chime and I checked for new missives, I discovered that it was WC telling me that the bucket had tipped, again. After the second time I decided to see what was urinating in the rain collector (which has the bird spikes/lightening arrestors).

Turns out it was not an animal but a vegetative problem.
. . . .
At any rate, I "pulled the plug" and now have an all-time high rain rate: "26.18 in/hr at 3:36 pm"!

Been there and had that happen . . . . . .

http://athena.trixology.com/index.php?topic=2028.msg18618#msg18618


Any suggestions for spreading the total rain over this fairly short time span? It's seen in todays total as only 1.14". I think simply adding a few hundredths over the 48 hours would be best? I can't even be sure of any zero amounts being accurate. I simply don't know when the drain became clogged... [coffee]

Perhaps someone else can chime in with a clever strategy.  I just left the anomaly for posterity.  If your station is up for long enough the oddity will be washed out in the totality of data.

Besides, it is obviously the lazy man's solution! . . . . .  ;D

I'll also be checking the collector after every sprinkle for the next few months... [rolleyes2] [banghead]

My collector cone is easy to reach and I do keep an eye on it during the rainy season.  During the dry season I keep a cap on it to avoid all the other things that could fall in it.  In California, when we say dry season we definitely mean dry season!

Cheers, Edouard  [cheers1]


xairbusdriver

  • Storm
  • *****
  • Posts: 2303
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.14.4/WC 3.0.4
Re: Editing weather data
« Reply #13 on: April 16, 2019, 10:43:02 PM »
Quote
During the dry season I keep a cap on it to avoid all the other things that could fall in it.
No real dry season around here. It could rain practically any week of the year. But I am thinking an inverted cone made of screening material might help keep items like this from blocking the hole. The built-in central screen obviously did not do its duty. It was actually floating in the water! The design has openings in the bottom but the top third of the contraption is a cone of plastic, so there is enough air in there to allow it to float right-side up. Thus, additional large items could easily get collected and block the drain. Maybe simply drilling a few hole in that part of the removable strainer will prevent the 'Noah's Ark' effect. I do check the buckets once a month in Spring just to keep the muck (dust/pollen/etc) cleaned out! We seem to get anything not open to the rain become coated with a yellow layer of pollen!! ;D

"NOT EDITING THE INVALID DATA?!" I've never heard of such a thing!!! How are you going to keep your Junior Weatherman license if that advice gets out?!  ::)

OK, I can see your reasoning, while reclining in an easy chair... but I certainly don't want that ridiculous 26.13 inches per hour "data point" left in the "All-time High/Low" collection! [blush]

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 5237
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
My example wasn't so extreme. (Re: Editing weather data)
« Reply #14 on: April 17, 2019, 11:15:52 PM »
Dear X-Air and WeatherCat station caregivers,

. . . .
I am thinking an inverted cone made of screening material might help keep items like this from blocking the hole.
. . . .

Unfortunately there is a sticky wicket associated with any sort of mesh screens.  Droplets will accumulate and never fall into your rain gauge.  When the rains stop, the droplets evaporate resulting in your rainfall counts being constantly lower than actual.  Where you are at the difference may be negligible, but in dry places it is preferable to keep an eye on the rain gauge and stick to the Davis design for the screen.  It does avoid trapping droplets and allowing them to evaporate.

"NOT EDITING THE INVALID DATA?!" I've never heard of such a thing!!! How are you going to keep your Junior Weatherman license if that advice gets out?!  ::)

If you look at my posting back in 2016 there was no need to edit really:

http://athena.trixology.com/index.php?topic=2028.msg18618#msg18618

There was so little water in my rain gauge that the peak rate wasn't out of the ordinary.  Since it wasn't setting any records, it was easier to just leave the "white lie" in place so that the rainfall totals were correct.

Cheers, Edouard  [cheers1]