Any way to merge data?

[bcurry]

Hi all,
I have a new computer, a Mac Studio, and I wiped and sold my old iMac. I got the old Weathercat folder off the Documents folders but did not back up the libraries and so forth. If you have a look at my webpage, I have the new data coming in (under the Past tab), and I have a link to a page that the older computer threw, and I have the data all on my new Mac, but not in Library form, just inside an old Weathercat folder.
Is there any way to take the 2014, 2015 etc info and have it displayed along with the new 2022 data?
I'm not sure what I'm asking is clear, but any help would be appreciated.
Bill

https://www.billcurry.ca/weather/

[elagache]

Dear Bill and WeatherCat users who migrate from time to time,

Hi all,
I have a new computer, a Mac Studio, and I wiped and sold my old iMac. I got the old Weathercat folder off the Documents folders but did not back up the libraries and so forth.
. . . . .
Is there any way to take the 2014, 2015 etc info and have it displayed along with the new 2022 data?
I'm not sure what I'm asking is clear, but any help would be appreciated.

I sure wish you had posted your question before making this transition.  We have a guide on the WeatherCat Wiki on how to move an installation from one Mac to another:

https://wiki.trixology.com/index.php?title=Moving_a_WeatherCat_installation_from_one_computer_to_another

Your old data was stored in the folder:

~/Library/WeatherCatData

If you didn't preserve this folder somehow, then unfortunately no, all your data was lost when you wiped out your old Mac's hard drive.

If you preserved any of these library folders, you should be able to get WeatherCat to recognize them by putting them back into the folders that WeatherCat is expecting.

Alternatively, did you have any backup scheme that you were using on your old Mac?  If you didn't wipe the backups, the data might still be there.  Time Machine would allow you to recover those folders for example.

Sorry,  . . . When in doubt, definitely post a question on the forum.  That's what we are here for.

Edouard

[bcurry]

Hi Edouard,
Thanks for the reply.
I had a rather large and quick failure on my old iMac, got the new machine and probably acted too quickly to wipe it. I had a major photography project to complete and things moved quickly!
I will look at my backups via Time Machine, but that may not be there either…
The way it looks now is OK, just not ideal.
Lesson learned,
Bill

[elagache]

Dear Bill and WeatherCat troubleshooters,

I will look at my backups via Time Machine, but that may not be there either…
. . . .

If you didn't wipe out the Time Machine archives from the old machine, the data still should be there.  However, it might be chore to find it.  If you can find the data folder in the archives, we should be able to tell you how to merge the old data so that you have previous years represented.

Cheers, Edouard

[bcurry]

Hi all,
I found the data in Time Machine!
I will need to know-do I now copy as in the instructions linked above, or do I just need the wcdata file?I have the past years now in the location folder but so far still only seeing current data?
Thanks for any help…
Bill

[elagache]

Hi Bill,

I found the data in Time Machine!
I will need to know-do I now copy as in the instructions linked above, or do I just need the wcdata file?I have the past years now in the location folder but so far still only seeing current data?

Well, I don't know of anyone who has gotten into exactly this predicament.  I'm guessing on the solution and we might need to refine it depending on how WeatherCat reacts to the changes.  You need to access a subfolder from your archives and merge the contents of that folder into the current WeatherCat folder.  The path is the same:

~/Library/WeatherCatData/Location1

As before, ~ refers to your home directory on your old Mac (in Time Machine) and your new Mac (where WeatherCat is storing data now.)

Inside this folder you should find a group of subfolders with names corresponding to years.   Here is what my folder looks like:

Code: [Select]

2009 2013 2017 2021 Journal.xml
2010 2014 2018 2022
2011 2015 2019 Calibration.xml
2012 2016 2020 Imagery
You should be able to simply copy the folders of past years into your new Mac's folder.  It is prudent to quit WeatherCat before doing this.  Dealing with 2022 is more tricky.  If you look at that folder you should see a list of flies like this:

Code: [Select]

1_WeatherCatData.cat 4_WeatherCatData.cat
1_WeatherCatData.cat.hrs 4_WeatherCatData.cat.hrs
2_WeatherCatData.cat 5_WeatherCatData.cat
2_WeatherCatData.cat.hrs 5_WeatherCatData.cat.hrs
3_WeatherCatData.cat 6_WeatherCatData.cat
3_WeatherCatData.cat.hrs 6_WeatherCatData.cat.hrs
Again you should be able to copy the files from January to May (1 to 5 prefix) into the 2022 folder on your new Mac.  Again, do this with WeatherCat not running.

That still might leave you with some data from June that is missing.  You might be able to retrieve this as well; however, it is more complex still.  Let us know if you want to do to those lengths.

Let us know if these instructions are sufficient and by all means - if it works!

Cheers, Edouard

[bcurry]

Hi Edouard et al,
Thanks so much for your help.
I got the past statistics to load and it now displays properly on the page (see link below).
I do note that if I wanted it to be completely accurate, I would have to "tweak" the May readings for 2022. I think what is happening is that the program is reading the two weeks at the end of May since my switch-over, but did not import the first two weeks of May (I'm guessing this going by the temperature and number of rain dates).
I am happy with the way it is, but I assume there is probably a way to massage the data to include those two weeks?
The important thing, by the way, was to put the data in the Location 1 folder in the Library WHILE WEATHERCAT WAS NOT RUNNING! When I then ran WeatherCat it immediately found and displayed the changed information.
Thanks again for all the help.
Bill
https://www.billcurry.ca/weather/statistics.html

[elagache]

Dear Bill and WeatherCat users who like complete data sets,

I do note that if I wanted it to be completely accurate, I would have to "tweak" the May readings for 2022. I think what is happening is that the program is reading the two weeks at the end of May since my switch-over, but did not import the first two weeks of May (I'm guessing this going by the temperature and number of rain dates).
I am happy with the way it is, but I assume there is probably a way to massage the data to include those two weeks?

It depends on why there is a hole in your data.  If there was a significant gap during which you were changing machines, you might have lost some data and you will have to accept that.

Were you collecting data for the first 2 weeks of May?  If so, can you retrieve the 5_WeatherCatData.cat file from your backup archive?  In order to have all the data for May you would need to create a new file that merges the contents of the file from the last 1/2 of May with the first.  If you open these files with a text editor, they look like this:

Code: [Select]

WeatherCat data file ***If you modify this file, be sure to delete the relevent .hrs file***.
t and V are not optional, all other fields are.
t is the day, hour and minute (2 digits each), T is outside temperature, Ti is internal temperature,                               T1 to T8 is auxiliary temperatures, D is dew point,                               Pr is barometric pressure, W is wind speed,
Wd is wind direction, Wc is wind chill, Wg is wind gust, Ph is hourly precipitation, P is total precipitation,
                              H is outside humidity, Hi is internal humidity, H1 to H8 are auxiliary humidity sensors, S is solar, CO21 to CO24 is CO2
Sm1 to Sm4 is soil moisture, Lw1 to Lw4 is leaf wetness, St1 to St4 is soil temperature,
                              Lt1 to Lt4 is leaf temperature, U is UV, Pm is monthly precipitation, Py is annual precipitation, Ed is daily ET, Em is monthly ET, Ey is yearly ET, C is current conditions (delimited by double quotes), V is validation.

VERS:3

    0 t:010000 T:9.61 Ti:20.89 D:7.04 Pr:1011.9844 W:0.00 Wd:86 Wc:9.61 Wg:0.00 Ph:0.00 P:0.00 H:84 Hi:62 S:0 U:0.0 T1:7.8 H1:89.0 Lw4:0.0 Pm:0.00 Py:104.65 Ed:0.00 Em:0.00 Ey:233.43 Sy1:3.73 Sy2:-0.00 Sy3:235.43 Sy4:35.92 Sy5:-6.44 C:"N/A" V:4
    1 t:010001 T:9.61 Ti:20.89 D:7.04 Pr:1012.0521 W:0.00 Wd:86 Wc:9.61 Wg:0.00 Ph:0.00 P:0.00 H:84 Hi:62 S:0 U:0.0 T1:7.8 H1:89.0 Lw4:0.0 Pm:0.00 Py:104.65 Ed:0.00 Em:0.00 Ey:233.43 Sy1:2.56 Sy2:-0.00 Sy3:0.00 Sy4:0.00 Sy5:-6.50 C:"N/A" V:4
    3 t:010003 T:9.72 Ti:20.94 D:7.15 Pr:1012.0521 W:0.00 Wd:86 Wc:9.72 Wg:0.00 Ph:0.00 P:0.00 H:84 Hi:62 S:0 U:0.0 T1:7.8 H1:89.0 Lw4:0.0 Pm:0.00 Py:104.65 Ed:0.00 Em:0.00 Ey:233.43 Sy1:2.56 Sy2:-0.00 Sy3:0.00 Sy4:0.00 Sy5:-6.44 C:"N/A" V:4
    4 t:010004 T:9.67 Ti:20.94 D:6.92 Pr:1012.0182 W:0.00 Wd:86 Wc:9.67 Wg:0.00 Ph:0.00 P:0.00 H:83 Hi:62 S:0 U:0.0 T1:7.8 H1:89.0 Lw4:0.0 Pm:0.00 Py:104.65 Ed:0.00 Em:0.00 Ey:233.43 Sy1:2.57 Sy2:-0.00 Sy3:0.00 Sy4:0.00 Sy5:-6.56 C:"N/A" V:4
    5 t:010005 T:9.67 Ti:20.94 D:7.09 Pr:1012.0182 W:0.00 Wd:86 Wc:9.67 Wg:0.00 Ph:0.00 P:0.00 H:84 Hi:62 S:0 U:0.0 T1:7.8 H1:89.0 Lw4:0.0 Pm:0.00 Py:104.65 Ed:0.00 Em:0.00 Ey:233.43 Sy1:2.57 Sy2:-0.00 Sy3:0.00 Sy4:0.00 Sy5:-6.50 C:"N/A" V:4
    6 t:010006 T:9.72 Ti:20.94 D:7.15 Pr:1011.9844 W:0.00 Wd:86 Wc:9.72 Wg:1.61 Ph:0.00 P:0.00 H:84 Hi:61 S:0 U:0.0 T1:7.8 H1:89.0 Lw4:0.0 Pm:0.00 Py:104.65 Ed:0.00 Em:0.00 Ey:233.43 Sy1:2.57 Sy2:-0.00 Sy3:0.00 Sy4:0.00 Sy5:-6.56 C:"N/A" V:4
   10 t:010010 T:9.72 Ti:20.94 D:6.97 Pr:1012.0860 W:0.00 Wd:93 Wc:9.72 Wg:3.22 Ph:0.00 P:0.00 H:83 Hi:61 S:0 U:0.0 T1:7.8 H1:89.0 Lw4:0.0 Pm:0.00 Py:104.65 Ed:0.00 Em:0.00 Ey:233.43 Sy1:2.57 Sy2:-0.00 Sy3:0.00 Sy4:0.00 Sy5:-6.56 C:"N/A" V:4

If you have both files, you should be able take the data of the file (everything but the comments and white space at the top) from the last 2 weeks of May and append it to the end of the file from the earlier part of May.

The important thing, by the way, was to put the data in the Location 1 folder in the Library WHILE WEATHERCAT WAS NOT RUNNING!

Yes, that is extremely important.  It is also preferable to use a standard text editor like BBEdit, though you can get away with TextEdit if you put it in plain text mode and are careful.  Finally as the instructions at the top of the file report: If you modify this file, be sure to delete the relevent .hrs file.  If you don't WeatherCat won't index the new data.

Yes, this is getting geeky, so keep backups of everything and be extra careful if you are bold enough to try this.

By all means do let us know if you can pull off this last bit of data rescue!

Cheers, Edouard

[bcurry]

Hi Edouard et al,
I went back to my Time Machine backup and went two days back from the date of the crash that caused the issue with the old iMac, and found a good copy of the WeatherCat data file. I copied the Location1 info for May to that date and then edited the Location1 on my new machine for May using BBEdit (and trashed the old .hrs file) and when I restarted WeatherCat, it now displays the updated information.
Yay!
I think that's about as far as I can go as I am now seeing that I may have missed two or three days, but that was in the timeframe when the iMac was giving me the issues and all kinds of things got messed up, so I'm thinking I'm good now.
I enjoy WeatherCat immensely as the data is fun to look at and my website gives me just what I want to see from my weather station.
Thanks again for all the help.
Cheers,
Bill
 https://www.billcurry.ca/weather

[xairbusdriver]

Thanks for the update! Another successful restoration guide from the experienced WC users group. (Even while battling their own physical 'restoration' processes! ) It really helps verify that things can be returned to near normal thanks to the great programming of Stu! He put a lot of wisdom and logic into designing a system that stores our info allows us to restore things when disaster strikes. And a big thanks to the experts here who have meticulously recorded the steps sometimes needed after our own mistakes.

[elagache]

Dear Bill, X-Air and WeatherCat faithful,

I went back to my Time Machine backup and went two days back from the date of the crash that caused the issue with the old iMac, and found a good copy of the WeatherCat data file. I copied the Location1 info for May to that date and then edited the Location1 on my new machine for May using BBEdit (and trashed the old .hrs file) and when I restarted WeatherCat, it now displays the updated information.
Yay!

Congratulations!  Honestly, you are the first person to actually try this.  I was reasonably confident it would work, but this sort of data gap simply had never happened before.  As X-Air points out, it is a real testimony to our fearless leader Stu (Stuart Ball, WeatherCat developer.)  WeatherCat is so well designed and implemented that even allows you to splice data together like this.

I think that's about as far as I can go as I am now seeing that I may have missed two or three days, but that was in the timeframe when the iMac was giving me the issues and all kinds of things got messed up, so I'm thinking I'm good now.

Once your station has been running long enough, a few days of lost data isn't enough to distort the overall picture of the weather at your location.

I enjoy WeatherCat immensely as the data is fun to look at and my website gives me just what I want to see from my weather station.

It is no small exaggeration that WeatherCat has really changed my life.  When I put up my weather station is was mostly a curiosity.  These days, I'm constantly referring to WeatherCat data in order to decide how to keep the house comfortable and safe.  If I had only known how useful a personal weather station actually is - I would have bought one much sooner!

Thanks for sharing your success story!

Cheers, Edouard

[bcurry]

Hi all,
Just a little further to my geekiness on weather data-
As I described before I was only missing 3 days worth of data since the crash and then the restoration with the help of those here in the WeatherCat forums.
Since my station has been running since 2012 and online since 2013, a couple days data loss should not really matter.
Except...
This May was very dry here, and with the data loss the month showed 39.6 mm of rain and 5 rain days.
What got me going was that that would make May 2022 the driest on record since 2013.
What to do?
Using the WeatherCat tools, I found an answer:
I used the daily rain editor to add the one day of the lost that was rainy (I keep a weather journal and also submit to CoCoRaHS, please don't judge me...) and added in the 12.7 mm that SHOULD have been there. WeatherCat accepted the new information (even though the day wasn't actually in there) and I now see that May 2022 is the third lowest month and the correct 52.3 mm of rain is now being displayed.
WeatherCat is a wonderful program and I love the way we can play with data to make it as precise as possible.
Fun stuff!

Bill

If you wish to see the stats:
https://www.billcurry.ca/weather

[Blicj11]

I love it when Edouard guides a WeatherCatter to a successful outcome. Thanks to both of you for documenting your adventure here.