I now need some examples of other people?s Weather Display files to proceed with further testing. Right now, I need the following files for any ONE MONTH
MYYYYlg.txt
MYYYYindoorlog.txt
MYYYYvantagelog.txt
MYYYYextralog.csv
M is the month (from 1 to 12), YYYY is the year so, for example, files for April 2011 would begin with 42011.
The MYYYYlg.txt file is required. The others are not required but if you have them I would like them.
You need to upload the files to my Dropbox account. If you email me at hwhalen@unb.ca I will tell you how do this.
Don?t? read beyond here unless you really care about the details of how this is done. It?s long and boring unless you are a programming geek!
************
The Weather Display MYYYYlg.txt file is a plain text file and contains the following information - day, month, year, hour, minute, temperature, humidity, dew point, barometer, wind speed, gust speed, wind direction, rain last minute, daily rain, monthly rain, yearly rain, heat index.
The MYYYYindoorlog.txt is a plain text file and contains the following information - day, month, year, hour, minute, indoor temperature, indoor humidity.
The MYYYYvantagelog.txt file is a plain text file and contains the following information ? day, month, year, hour, minute, solar radiation, UV, daily ET, soil moisture, soil temperature
The MYYYYextralog.csv file is a Comma Separated Variable file and contains the following information - day, month, year, hour, minute, 9 temperature readings numbered 1 to 9, 9 humidity readings numbered 1 to 9.
Using these 4 files we can either retrieve or calculate the information that goes into a WeatherCat log file. This file is called X_WeatherCatData.cat where X is the month. You will note that there is no information in the file name to indicate the year. WeatherCat puts 12 months of data into a directory with the name of the year. Eg the directory is named 2016.
The X_WeatherCatData.cat file is a plain text file which contains the following information, described in a header in each file. The header reads:
WeatherCat data file ***If you modify this file, be sure to delete the relevent .hrs file***.
t and V are not optional, all other fields are.
t is the day, hour and minute (2 digits each), T is outside temperature, Ti is internal temperature, T1 to T8 is auxiliary temperatures, D is dew point, Pr is barometric pressure, W is wind speed,
Wd is wind direction, Wc is wind chill, Wg is wind gust, Ph is hourly precipitation, P is total precipitation,
H is outside humidity, Hi is internal humidity, H1 to H8 are auxiliary humidity sensors, S is solar, CO21 to CO24 is CO2
Sm1 to Sm4 is soil moisture, Lw1 to Lw4 is leaf wetness, St1 to St4 is soil temperature,
Lt1 to Lt4 is leaf temperature, U is UV, Pm is monthly precipitation, Py is annual precipitation, Ed is daily ET, Em is monthly ET, Ey is yearly ET, C is current conditions (delimited by double quotes), V is validation.
VERS:3
An examination of the files indicates that the following information is in the Weather Display files but not the X_WeatherCatData.cat file: year, month, rain for minute, heat index. This is because WeatherCat uses the file name for the month and the directory location for the year. WeatherCat calculates the heat index from the other information in the X_WeatherCatData.cat file.
The following information is included in the X_WeatherCatData.cat file but not in the Weather Display files: hourly precipitation, wind chill, monthly ET, yearly ET, Current Conditions. Therefore, we must calculate this from information that we retrieved from the Weather Display files.
Other Issues
The original data files from Weather Display may have errors. There can be several of these. (Please note that this is not a criticism of Weather Display. The errors can occur at many places. The Davis weather sensors may have errors. The transmission of the data may have problems in many places. Transmission to the ISS. Transmission to the console. Transmission to the computer, etc. )
Each Weather Display monthly data file has 30 days x 24 hours x 60 minutes = 43,200 data samples. Each sample can have 7 to 23 data items. So, say an average of 12. We have 4 files with data so we have 43,200 x 4 files x 12 observations = 2,073,000 data points each month. We can have errors in any of these.
The errors are rare but they take several forms:
Null values in the data samples. In 84 months of data I had three instances of null data. So, in 2,073,000 x 84 = 174,132,000 data points I had 3 errors. Nonetheless, each one would crash the program. So, I must search through all the data points and eliminate the nulls.
Weather Display files may also have redundant data. If Weather Display crashes it may not have properly recorded the last time it saved data. When it restarts it my go to my logger and reload the data. So, there will be redundant data. I must remove all the redundant data. Again, this does not happen often but I must deal with it.
WeatherCat uses the X_WeatherCatData.cat files to calculate other things. This data is stored in X_WeatherCatData.cat.hrs files. I really don?t know much about this but I think it contains statistical and other calculated data. The data in the X_WeatherCatData.cat.hrs files comes from the X_WeatherCatData.cat files.
All WeatherCat data while it may be displayed in different units, is stored in metric units (degrees Celsius, kilometers per hour, pressure in mb or hPa (which are the same), rainfall in mm, evapotranspiration in mm). Data in Weather Display may be stored in different units (temperature in Fahrenheit or Celsius, speed in mph, kph or knots per hour, rainfall in mm or inches, evapotranspiration in inches or mm, etc. So, we must know what units Weather Display has stored the data. Then we must translate the units into metric when we import data.
So, we do not need to create the X_WeatherCatData.cat.hrs files as WeatherCat does this for us. There is, however, one thing we must be careful of. There can be no holes in the sequence of X_WeatherCatData.cat files. Here is an example. If I have X_WeatherCatData.cat files from September 2006 to August 2014 and October 2014 to the present, WeatherCat will only create X_WeatherCatData.cat.hrs files back to October, 2014. It will not go back before this because it is missing the September 2014 file. Since that file is missing it assumes there is nothing before it.
So, we must fill in all missing data files from the start of our Weather Display files up to the month we start using WeatherCat. There doesn?t need to be any data in the files. They only need to have the standard header.
End of a long and rambling post. Time to go to bed.
Hugh