Author Topic: Importing data From Weather Display  (Read 8552 times)

Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Importing data From Weather Display
« on: June 06, 2017, 12:36:44 PM »
I have data from August 2008 to today in Weather Display files. I have looked at the documentation and format of the 6_WeatherCatData.cat file.

I know enough Swift/XCode that I can write a program to extract the data from Weather Display files and put it into .cat files. I would like to confirm a few things before I do this. Any insights would be helpful.

1.  I would need to create the proper directory structure with a separate directory for each year. Correct?

2.  In each directory the files are named X_WeatherCatData.cat with the X being 1...12. Correct?

3.  If I change a file I must delete the X_WeatherCatData.cat.hrs file? Correct?

4.  Upon starting WeatherCat will create a new X_WeatherCatData.cat.hrs file for ALL of the new X_WeatherCatData.cat files. This will bring all my historical data into WeatherCat. Correct?

5. The data in the X_WeatherCatData.cat files is in UTF8 (plain text). Correct?

6. The headers in the X_WeatherCatData.cat files are optional as the important data is in the key:value pairs. Correct? I will put them in but I was just curious.

7. The line separators in the X_WeatherCatData.cat files are /n. Correct?

8. The first entry in each line of a X_WeatherCatData.cat file is a number. This is just a line number and has no significance. Correct? Do I need to put this  in? If so, what is it?

9. There are keys in the X_WeatherCatData.cat file that I do not understand. They are Sy1 to Sy5. I think they may be System status codes but I do not know what they are. Can anyone explain what they are?

Thanks in advance.

Hugh

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Appear to be on the mark. (Re: Importing data From Weather Display)
« Reply #1 on: June 06, 2017, 11:33:48 PM »
Dear Hugh and WeatherCat "do it yourself" types,

I have data from August 2008 to today in Weather Display files. I have looked at the documentation and format of the 6_WeatherCatData.cat file.

I know enough Swift/XCode that I can write a program to extract the data from Weather Display files and put it into .cat files. I would like to confirm a few things before I do this. Any insights would be helpful.

Well now!  You are a go getter!!  If you manage to write this program would you be willing to share it with any other Weather Display users who might want to switch to WeatherCat?

1.  I would need to create the proper directory structure with a separate directory for each year. Correct?

Correct.  For historical reasons the path to the directories of the years is:

~/Library/WeatherCatData/Location1/

2.  In each directory the files are named X_WeatherCatData.cat with the X being 1...12. Correct?

Correct

3.  If I change a file I must delete the X_WeatherCatData.cat.hrs file? Correct?

Correct

4.  Upon starting WeatherCat will create a new X_WeatherCatData.cat.hrs file for ALL of the new X_WeatherCatData.cat files. This will bring all my historical data into WeatherCat. Correct?

Correct.  Actually the *.cat.hrs are the index files which are generated to access the data.

5. The data in the X_WeatherCatData.cat files is in UTF8 (plain text). Correct?

I think this very likely, but I don't have a handy tool to confirm this 100%.  If this concerns you, check with the developer Stuart Ball by contacting technical support on the trixology website.

6. The headers in the X_WeatherCatData.cat files are optional as the important data is in the key:value pairs. Correct? I will put them in but I was just curious.

Same comment as before.  I'm just a beta-tester and don't know every last bit of the software, but my understanding was that comments were simply to help the users and weren't required.

7. The line separators in the X_WeatherCatData.cat files are /n. Correct?

I believe so as well, but to be 100% sure ask Stu.

8. The first entry in each line of a X_WeatherCatData.cat file is a number. This is just a line number and has no significance. Correct? Do I need to put this  in? If so, what is it?

Unfortunately I don't know the answer to that one as well.  If I was in your position I would simply replicate the file format as it is a minor addition.

9. There are keys in the X_WeatherCatData.cat file that I do not understand. They are Sy1 to Sy5. I think they may be System status codes but I do not know what they are. Can anyone explain what they are?

Actually these are what is called "synthetic channels."  These are values that are derived from the physical data using AppleScript coding.  They are actually a very useful addition to WeatherCat.  You can read up on synthetic channels starting on page 56 of the current WeatherCat manual.

Once more welcome and by all means feel free to ask any additional questions you might have.

Cheers, Edouard

Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Re: Importing data From Weather Display
« Reply #2 on: June 07, 2017, 12:54:27 AM »
I will certainly be willing to share it. However, I think it would only be useful to people who have both Weather Display and Vantage Pro hardware.

Thanks for the informative reply.

Hugh

iccb

  • Gentle Breeze
  • **
  • Posts: 71
    • S??tilanne Eurassa
  • Station Details: Mac Mini (late 2018), Davis Vantage Pro2
Re: Importing data From Weather Display
« Reply #3 on: June 07, 2017, 02:53:35 PM »
This is great thing Hugh!!   [woohoo]
I have data in weather display from 2011 to 2014 and it would be nice to get those in weathercat.

Do you have Hugh, wd for mac or windows?


Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Re: Importing data From Weather Display
« Reply #4 on: June 07, 2017, 06:32:19 PM »
I started off in Windows and switched to WD Mac a few years ago.

I will work on it this week and keep you posted.

Hugh

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Thank you! (Re: Importing data From Weather Display)
« Reply #5 on: June 07, 2017, 11:23:20 PM »
Dear Hugh and WeatherCat weather data "historians,"

I will certainly be willing to share it. However, I think it would only be useful to people who have both Weather Display and Vantage Pro hardware.

You might well be correct, but the Davis is the most popular station around, so there is likely to be a few WeatherCat users with orphaned data that they would like to retrieve.

I hope indeed you find the task straightforward and I thank you for your efforts!

Cheers, Edouard  [cheers1]

Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Re: Importing data From Weather Display
« Reply #6 on: June 08, 2017, 01:23:07 AM »
Edouard,

As I am writing the program I encountered one more thing I am not sure of.

Ph is hourly precipitation
Pm is monthly precipitation
Py  is yearly precipitation

What is P? In the header file it says  total precipitation. But total  for what? For the day?

I am making significant progress. Hope to have it finished next week.

Hugh

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Total daily rain (Re: Importing data From Weather Display)
« Reply #7 on: June 08, 2017, 11:29:59 PM »
Dear Hugh and WeatherCat "DIY" types, . . . .

As I am writing the program I encountered one more thing I am not sure of.

. . .

What is P? In the header file it says  total precipitation. But total  for what? For the day?

P is total daily precipitation.  Also, I assume you knew this but just in case, WeatherCat stores all data in metric units.

I am making significant progress. Hope to have it finished next week.

Glad to hear the project isn't posing any major challenges thus-far!

Best of luck with this project!  [tup]

Cheers, Edouard  [cheers1]

Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Re: Importing data From Weather Display
« Reply #8 on: June 09, 2017, 12:56:13 AM »
Edourd,

That's what I thought but better to ask than assume.

Thanks,

Hugh

iccb

  • Gentle Breeze
  • **
  • Posts: 71
    • S??tilanne Eurassa
  • Station Details: Mac Mini (late 2018), Davis Vantage Pro2
Re: Importing data From Weather Display
« Reply #9 on: June 19, 2017, 10:21:23 AM »
Hello Hugh!

How is your project going?  ThU32:-)

Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Re: Importing data From Weather Display
« Reply #10 on: June 19, 2017, 12:20:13 PM »
It's coming along quite well. At this point I can...
Import WeatherDisplay data for a month into a M_WeatherCatData.cat file.

It looks exactly like the WeatherCat data file that WeatherCat produces but I have not yet tested it by moving it into the WeatherCat directory.

I can search and get all related files from Weather Display.

I still have to do the following:

Write code to go through all the files and export them into the proper directory structure. That is a structure which puts each year's files in the appropriate directory names for the year. ie. put all 2012 files in a directory called 2012.

Write the interface code so that a person can put in the input directory (where the Weather Display files are located) and the output directory (where the X_WeatherCatData.cat should be placed.)

There is also some error handling that needs to be done.

Also, it appears that Weather Display stores data in the units that the user specifies. So, for example, I use all metric units but some people may use all English units or a mixture of both. Someone may have temperature in Fahrenheit but pressure in mb rather than inches. It appears that WeatherCat always stores the metric data and if required converts it to the format the user wishes to display. So, the program would have to know the units that the user specified and do the conversion. I have not put in the  code to do this as all my data is metric so I don't have test data for other formats.

One final thing. I don't know if users with different sensors (solar, uv, soil temperature) have a different file format than I do. So, if someone wants to use the program I would have to have examples of their data files to ensure that the conversion is done correctly.

I hope to be finished  this week but problems can always occur. The first 90% of software development takes 90% of the time. The last 10% takes the other 90%.  :-\

It takes around 12 to 14 seconds to import a month of data on a Mid 2010 iMac 27-inch with a 2.93 GHz processor and a Hard Disk Drive.

Next week I will have a 2017 iMac 27-inch with a 3.5 GHz processor and 512 GB SSD. I suspect the time to process will drop. [bounce]

Hugh

iccb

  • Gentle Breeze
  • **
  • Posts: 71
    • S??tilanne Eurassa
  • Station Details: Mac Mini (late 2018), Davis Vantage Pro2
Re: Importing data From Weather Display
« Reply #11 on: June 19, 2017, 07:50:12 PM »
Thx Hugh for your long and thorough answer!

It sounds that You are doing super good importer software! =)  ThU32:-)
I have davis vantage pro2 with UV and solar sensors and my WD data is also in metrics, so if You need any datafiles, I can send them to you!

P.S. U got early Christmas when You get the new iMac!!   lol(1) ThU5:-)

Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Re: Importing data From Weather Display
« Reply #12 on: June 19, 2017, 08:02:14 PM »
The program converted all WeatherDisplay files from August 2008 to June 2017 ( a total of 107 months ) in 18 minutes. It also placed them all in the correct directories.

I am now visually inspecting the files to look for any obvious errors before I attempt to put them into WeatherCat.

Progress ...

Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Re: Importing data From Weather Display
« Reply #13 on: June 19, 2017, 08:04:30 PM »
Thx Hugh for your long and thorough answer!

It sounds that You are doing super good importer software! =)  ThU32:-)
I have davis vantage pro2 with UV and solar sensors and my WD data is also in metrics, so if You need any datafiles, I can send them to you!

P.S. U got early Christmas when You get the new iMac!!   lol(1) ThU5:-)

I will probably need another set of files to test once I have convinced myself that the conversion of my files was done correctly. I will email you later in the week when I have reached that point.

Thanks,

Hugh

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Thanks for the update! (Re: Importing data From Weather Display)
« Reply #14 on: June 20, 2017, 12:05:34 AM »
Dear Hugh and WeatherCat fans who want to take their data with them,

It's coming along quite well. At this point I can...
Import WeatherDisplay data for a month into a M_WeatherCatData.cat file.

Congratulations!  ThU5:-)

I hope to be finished  this week but problems can always occur. The first 90% of software development takes 90% of the time. The last 10% takes the other 90%.  :-\

Ain't that the truth!!   [banghead]

Cheers, Edouard