Author Topic: Importing data From Weather Display  (Read 8555 times)

Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Re: Importing data From Weather Display
« Reply #15 on: June 20, 2017, 01:56:56 PM »
I have converted and moved  over all my Weather Display data from August 2008 to May 2017 inclusive.

WeatherCat successfully interpreted all the data. I can see data from every month as well as NOAA reports, etc.

Some of the data is bad and I am checking to see why. So far, it has  all been bad data in Weather Display, not a conversion problem.

Hugh

Blicj11

  • Storm
  • *****
  • Posts: 3941
    • EW3808
    • KUTHEBER6
    • Timber Lakes Weather
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP Data Logger | iMac (2019), 3.6 GHz Intel Core i9, 40 GB RAM, macOS Ventura 13.6 | Sharx SCNC2900 Webcam | WeatherCat 3.3 | Supportive Wife
Re: Importing data From Weather Display
« Reply #16 on: June 20, 2017, 05:41:11 PM »
Nice progress. Thanks for keeping us in the loop.
Blick


Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Re: Importing data From Weather Display
« Reply #17 on: June 20, 2017, 06:30:47 PM »
Update:

Some of the data in my Weather Display files was bad. In one case there were a number of Nulls in the Weather Display files (not many, three instances in two files out of a total of 107 files). This caused the program to crash. I manually removed the Nulls but will fix the program to automatically remove them before they are processed.

In another case 3 observations of Pressure were wrong in a three minute (observation) period. The pressure went from 1013.0 to 0.0 to 976.3 to 1015.0 and back to 1013.0. The effect of this was to report the minimum pressure at my station was 0.0 mb. Sort of like halfway to the moon. I fixed this by manually editing the Weather Display file.

One thing that caused a number of problems. I lost data from July 2013 to July of 2015 because of a backup failure. Weather Display has an option to download data from Weather Underground if you have been uploading data to that site. As I have done that, I downloaded the data. However, this data does not have exactly the same format as regular Weather Display files. When I process this data I get some very weird results. For example, it shows the maximum rain day is March 13, 2014 with 345.4 mm of rain. Essentially I have had to discard the data downloaded from Weather Underground.

When I discard the months from July 2013 to July 2015 it creates a problem. WeatherCat will create X_WeatherCatData.cat.hrs files only back to August 2015. It will not create data back farther than that even if the X_WeatherCatData.cat files exist.  This should not be a problem for anyone who has continuous files going up to the time they switched over. That is, it stops when there are missing months.

I have written to weathercat_support@trixology.com asking if there is any way to work around this.

I still have to do the user interface so that people can input the locations of their Weather Display files and the location where they want their converted files to be stored. I will probably do that Thursday or Friday.

Hugh

Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Re: Importing data From Weather Display
« Reply #18 on: June 20, 2017, 07:22:13 PM »
Just received a reply from WeatherCat support. The workaround is to create X_WeatherCatData.cat files for the missing months with just the header in the file.

Appeared to have worked flawlessly.

I will try to add this to the conversion program to fill in missing months.

Fast and informative service. I am very impressed!
 [cheer]

Hugh

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Dear Hugh, Blick, and WeatherCat faithful,

Sorry about your troubles with lost data, but glad the process is working overall.

Fast and informative service. I am very impressed!
 [cheer]

Yes indeed, there are good reasons to stick with this cat!  [cat]

Cheers, Edouard  [cheers1]

Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Re: Importing data From Weather Display
« Reply #20 on: June 21, 2017, 12:15:26 AM »
Later this week I will probably be looking for beta testers, if anyone is interested.

Looks like everything with the original Weather Display files transitioned well.

Hugh

iccb

  • Gentle Breeze
  • **
  • Posts: 71
    • S??tilanne Eurassa
  • Station Details: Mac Mini (late 2018), Davis Vantage Pro2
Re: Importing data From Weather Display
« Reply #21 on: June 21, 2017, 09:25:12 AM »
Sounds that things going pretty well! I have also good experience from weathercat support...

I can beta test if Hugh approves me...  ThU5:-) lol(1)

Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Re: Importing data From Weather Display
« Reply #22 on: June 21, 2017, 08:19:50 PM »
This is very odd. I was looking at the records for Windspeed and some other variables. Weather Display shows a maximum wind gust of 93.2 kmh on December 30 of last year at 8:46 in the morning. However, WeatherCat shows the maximum gust for that day at 50 kmh for the same time. I looked at the log files for Weather Display for that exact time and it shows 50.

Hmmm 50 mph = 80.5 kph

50 knots = 92.6 kph

Whaat?????

So, I check the maximum gust for today. It occurred at 12:45 PM. In WeatherCat it is recorded in the log as 45.06 kmh. 28 mph is exactly 46.06 kmh. So I look at Weather Display for the same time. The data is 24.0. 24.0 mph is 38.62 kmh. 24.0 knots is 44.45 kmh. But all the windspeeds in the Weather Display log files have 0 after the decimal place. If you convert mph to knots then 28 mph = 24.33 knots = 46.06 kmh.

Wow.  All my wind data is in knots in the Weather Display files. How did that happen I have no idea. I will, however have to do the file conversion over again.

Sigh.

Hugh

Blicj11

  • Storm
  • *****
  • Posts: 3941
    • EW3808
    • KUTHEBER6
    • Timber Lakes Weather
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP Data Logger | iMac (2019), 3.6 GHz Intel Core i9, 40 GB RAM, macOS Ventura 13.6 | Sharx SCNC2900 Webcam | WeatherCat 3.3 | Supportive Wife
Re: Importing data From Weather Display
« Reply #23 on: June 21, 2017, 08:41:43 PM »
All my wind data is in knots in the Weather Display files. How did that happen I have no idea. I will, however have to do the file conversion over again.

I hate it when one makes such discoveries. Sorry, mate.
Blick


Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Re: Importing data From Weather Display
« Reply #24 on: June 21, 2017, 09:03:27 PM »
All my wind data is in knots in the Weather Display files. How did that happen I have no idea. I will, however have to do the file conversion over again.

I hate it when one makes such discoveries. Sorry, mate.

Took me 10 minutes to change the code and 20 minutes to run it. Everything looks OK now...

So no major disaster. That's just what checking the results of a program run is for.  [coffee]

montever

  • Calm
  • *
  • Posts: 8
    • DW3169
    • KCAOAKLA11
  • Station Details: VantagePro2, MacBook Air.
Re: Importing data From Weather Display
« Reply #25 on: June 21, 2017, 11:49:52 PM »
Thank you, Hugh. 

I am a brand new convert from WD.   I would willing to serve as beta tester for your add on to bring all my WD files over.   

John

Blicj11

  • Storm
  • *****
  • Posts: 3941
    • EW3808
    • KUTHEBER6
    • Timber Lakes Weather
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP Data Logger | iMac (2019), 3.6 GHz Intel Core i9, 40 GB RAM, macOS Ventura 13.6 | Sharx SCNC2900 Webcam | WeatherCat 3.3 | Supportive Wife
Re: Importing data From Weather Display
« Reply #26 on: June 21, 2017, 11:57:37 PM »
Welcome to WeatherCat Montever!
Blick


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Thanks and welcome! (Re: Importing data From Weather Display)
« Reply #27 on: June 22, 2017, 12:26:15 AM »
Thank you once more Hugh for your work on this converter!

Welcome John (montever) to the WeatherCat forum!

Cheers, Edouard  [cheers1]

Hugh Whalen

  • Gentle Breeze
  • **
  • Posts: 64
    • INBFLOWE2
  • Station Details: Vantage Pro 2 with UV and Solar sensors. 2017 27" iMac, 3.5 GHz i5, 16GB RAM, 512 GB SSD
Re: Importing data From Weather Display
« Reply #28 on: June 23, 2017, 01:19:10 AM »
I now need some examples of other people?s  Weather Display files to proceed with further testing. Right now, I need the following files for any ONE MONTH

MYYYYlg.txt
MYYYYindoorlog.txt
MYYYYvantagelog.txt
MYYYYextralog.csv

M is the month (from 1 to 12), YYYY is the year so, for example, files for April 2011 would begin with 42011.

The MYYYYlg.txt file is required. The others are not required but if you have them I would like them.

You need to upload the files to my Dropbox account. If you email me at hwhalen@unb.ca I will tell you how do this.

Don?t? read beyond here unless you really care about the details of how this is done. It?s long and boring unless you are a programming geek!

************
The Weather Display MYYYYlg.txt file is a plain text file and contains the following information  - day, month, year, hour, minute, temperature, humidity, dew point, barometer, wind speed, gust speed, wind direction, rain last minute, daily rain, monthly rain, yearly rain, heat index.

The MYYYYindoorlog.txt is a plain text file and contains the following information -  day, month, year, hour, minute, indoor temperature, indoor humidity.

The MYYYYvantagelog.txt file is a plain text file and contains the following information ? day, month, year, hour, minute, solar radiation, UV, daily ET, soil moisture, soil temperature

The MYYYYextralog.csv file is a Comma Separated Variable file and contains the following information -  day, month, year, hour, minute, 9 temperature readings numbered 1 to 9, 9 humidity readings numbered 1 to 9.

Using these 4 files we can either retrieve or calculate the information that goes into a WeatherCat log file. This file is called X_WeatherCatData.cat where X is the month. You will note that there is no information in the file name to indicate the year. WeatherCat puts 12 months of data into a directory with the name of the year. Eg the directory is named 2016.

The X_WeatherCatData.cat file is a plain text file which contains the following information, described in a header in each file. The header reads:

WeatherCat data file ***If you modify this file, be sure to delete the relevent .hrs file***.
t and V are not optional, all other fields are. 
t is the day, hour and minute (2 digits each), T is outside temperature, Ti is internal temperature, T1 to T8 is auxiliary temperatures, D is dew point, Pr is barometric pressure, W is wind speed, 
Wd is wind direction, Wc is wind chill, Wg is wind gust, Ph is hourly precipitation, P is total precipitation,
H is outside humidity, Hi is internal humidity, H1 to H8 are auxiliary humidity sensors, S is solar, CO21 to CO24 is CO2
Sm1 to Sm4 is soil moisture, Lw1 to Lw4 is leaf wetness, St1 to St4 is soil temperature,
Lt1 to Lt4 is leaf temperature, U is UV, Pm is monthly precipitation, Py is annual precipitation, Ed is daily ET, Em is monthly ET, Ey is yearly ET, C is current conditions (delimited by double quotes), V is validation.

VERS:3


An examination of the files indicates that the following information is in the Weather Display files but not the X_WeatherCatData.cat file: year, month, rain for minute, heat index. This is because WeatherCat uses the file name for the month and the directory location for the year.  WeatherCat calculates the heat index from the other information in the X_WeatherCatData.cat file.

The following information is included in the X_WeatherCatData.cat file but not in the Weather Display files: hourly precipitation, wind chill, monthly ET, yearly ET, Current Conditions. Therefore, we must calculate this from information that we retrieved from the Weather Display files.


Other Issues

The original data files from Weather Display may have errors. There can be several of these. (Please note that this is not a criticism of Weather Display. The errors can occur at many places. The Davis weather sensors may have errors. The transmission of the data may have problems in many places. Transmission to the ISS. Transmission to the console. Transmission to the computer, etc. )

Each Weather Display monthly data file has 30 days x 24 hours x 60 minutes = 43,200 data samples.  Each sample can have 7 to 23 data items. So, say an average of 12. We have 4 files with data so we have 43,200 x 4 files x 12 observations = 2,073,000 data points each month. We can have errors in any of these.

The errors are rare but they take several forms:

Null values in the data samples. In 84 months of data I had three instances of null data.  So, in 2,073,000 x 84 = 174,132,000 data points I had 3 errors. Nonetheless, each one would crash the program. So, I must search through all the data points and eliminate the nulls.

Weather Display files may also have redundant data. If Weather Display crashes it may not have properly recorded the last time it saved data. When it restarts it my go to my logger and reload the data. So, there will be redundant data. I must remove all the redundant data. Again, this does not happen often but I must deal with it.

WeatherCat uses the X_WeatherCatData.cat files to calculate other things. This data is stored in X_WeatherCatData.cat.hrs files. I really don?t know much about this but I think it contains statistical and other calculated data. The data in the X_WeatherCatData.cat.hrs files comes from the X_WeatherCatData.cat files.

All WeatherCat data while it may be displayed in different units, is stored in metric units (degrees Celsius, kilometers per hour, pressure in mb or hPa (which are the same), rainfall in mm, evapotranspiration in mm). Data in Weather Display may be stored in different units (temperature in Fahrenheit or Celsius, speed in mph, kph or knots per hour, rainfall in mm or inches, evapotranspiration in inches or mm, etc. So, we must know what units Weather Display has stored the data. Then we must translate the units into metric when we import data.

So, we do not need to create the X_WeatherCatData.cat.hrs files as WeatherCat does this for us. There is, however, one thing we must be careful of. There can be no holes in the sequence of X_WeatherCatData.cat files. Here is an example. If I have X_WeatherCatData.cat files from September 2006 to August 2014 and October 2014 to the present, WeatherCat will only create X_WeatherCatData.cat.hrs files back to October, 2014. It will not go back before this because it is missing the September 2014 file. Since that file is missing it assumes there is nothing before it.

So, we must fill in all missing data files from the start of our Weather Display files up to the month we start using WeatherCat. There doesn?t need to be any data in the files.  They only need to have the standard header.

End of a long and rambling post. Time to go to bed.

 [sleep]

Hugh



montever

  • Calm
  • *
  • Posts: 8
    • DW3169
    • KCAOAKLA11
  • Station Details: VantagePro2, MacBook Air.
Re: Importing data From Weather Display
« Reply #29 on: June 23, 2017, 09:06:24 PM »
Thanks, Hugh.  I sent you an email and can send you the relevant .txt files from WD Logs.

John