File/Record structures?

[Minoic]

Hello,
after reading a lot in the LWC forum and here I understand that the only way to import data (WeatherLink only) is through LWC and that WC will 'automatically' take these .lw2 files over.

I have two and a half years of .wlk files but I also have three years of data which I tried to put into files to be read by LWC.

I placed a file into a folder called 2009 under Location 1. The file (few lines) looks like this:
--------------------------------------------------------------------------------------------------------------
LWC data file ***If you modify this file, be sure to delete the relevant .hrs file***.
t and V are not optional, all other fields are.
t is the day, hour and minute (2 digits each), T is outside temperature, Ti is internal temperature, T1 to T8 is auxiliary temperatures, D is dew point, Pr is barometric pressure, W is wind speed,
Wd is wind direction, Wc is wind chill, Wg is wind gust, Ph is hourly precipitation, P is total precipitation,
H is outside humidity, Hi is internal humidity, H1 to H8 are auxiliary humidity sensors, S is solar,
Sm1 to Sm4 is soil moisture, Lw1 to Lw4 is leaf wetness, St1 to St4 is soil temperature,
Lt1 to Lt4 is leaf temperature, U is UV, C is current conditions (delimited by double quotes), V is validation.

VERS:3

t:151904 T:05.70 Ti:21.00 W:00.00 Wd:050 Ph:00.00 H:82 Hi:56 V:4
t:151934 T:05.30 Ti:21.00 W:09.10 Wd:285 Ph:00.00 H:83 Hi:57 V:4
t:152004 T:05.10 Ti:21.00 W:00.00 Wd:040 Ph:00.00 H:84 Hi:57 V:4
t:152034 T:05.40 Ti:21.00 W:05.40 Wd:300 Ph:00.00 H:81 Hi:57 V:4
t:152104 T:05.00 Ti:21.00 W:00.00 Wd:295 Ph:00.00 H:86 Hi:57 V:4
t:152134 T:04.70 Ti:21.00 W:00.00 Wd:315 Ph:00.00 H:87 Hi:57 V:4
t:152204 T:04.90 Ti:20.90 W:10.30 Wd:330 Ph:00.00 H:86 Hi:57 V:4
t:152234 T:04.90 Ti:20.80 W:00.00 Wd:210 Ph:00.00 H:86 Hi:57 V:4
----------------------------------------------------------------------------------------------------------

Of course LWC is not recognizing this file because i cannot see it when using the Data Editor within the program.

Questions:

Is there a way to get the data into . lw2 files manually?
If not possible without a converter from the WC developers (I've heard that a .csv discussion for WC got started), should I hold on to that data for later?
Assuming that I import my .wlk data into LWC and then move to WC, will it be possible to import older data later?
Should I forget the whole thing?

Thanks

[elagache]

Hello again George,

I can sympathize with your data plight.  Alas, I can't help with your problem.  Unlike you, it took me less than a day to get completely fed-up with WeatherLink.  I found LWC the following day and never turned back . . . . .   

However, there are some other forum visitors who know more about this.  They are more infrequent visitors though.  So you'll need to have some patience for this phase of the project. 

So just play it cool until then . . . 

Cheers, Edouard 

[WCDev]

Hello George, welcome to the forum.

Yes, LWC can import (some versions of) WeatherLink data via the importer found in its 'Tools' menu. I'd suggest first you import the data into LWC to give a complete data set from now back to the start of your WeatherLink data then run WeatherCat to import the LWC data.

Note that WeatherCat offers to import LWC data the first time it's run, so if you've already run it, it won't offer again. To get around this, delete the WeatherCatData folder found in your ~/Library and run WeatherCat again - it'll see no WeatherCat data and offer to import your LWC data.

If you have data older than your WeatherLink data that you want to bring over via some other method - a script for example, you'll have to make sure there are no holes in the data set. WeatherCat determines the start of the data when it finds a month with no data - when it boots it scans backwards from todays date looking for a data file for each month - when it doesn't find a file, it stops scanning. So for example if you had data files for October 2010 and also December 2010 onwards, WeatherCat would think your data-set started in December 2010. If you filled in the hole at November 2010 then it would also find the October file.

Cheers,
Stu.

[Minoic]

Hello Stu,
thanks for the quick reply.

I had thought of it but didn't try to bridge the gap of few months.

One question:
Do these files need to be populated with data or can they be empty? In other words is it only the missing file or the data that makes the difference?

I didn't run WeatherCat yet because I wanted to have a clean data set before that.

thanks
George

[WCDev]

Hello George,
The absolute bare minimum is the following:

Code: [Select]

WeatherCat data file ***If you modify this file, be sure to delete the relevent .hrs file***.
t and V are not optional, all other fields are.
t is the day, hour and minute (2 digits each), T is outside temperature, Ti is internal temperature, T1 to T8 is auxiliary temperatures, D is dew point, Pr is barometric pressure, W is wind speed,
Wd is wind direction, Wc is wind chill, Wg is wind gust, Ph is hourly precipitation, P is total precipitation,
H is outside humidity, Hi is internal humidity, H1 to H8 are auxiliary humidity sensors, S is solar,
Sm1 to Sm4 is soil moisture, Lw1 to Lw4 is leaf wetness, St1 to St4 is soil temperature,
Lt1 to Lt4 is leaf temperature, U is UV, Pm is monthly precipitation, Py is annual precipitation, Ed is daily ET, Em is monthly ET, Ey is yearly ET, C is current conditions (delimited by double quotes), V is validation.

VERS:3

Note that a carriage return is needed after the VERS:3 line.


If you do this, the statistics for that month will be zero's or dashes. To improve on this, you can enter one line per day (say timed at midday) or for better data two lines per day - one with the minimums and one with the maximums/totals. That way you'll end up with more accurate records but very little data. You can replace 'days' with 'weeks' if you'd prefer, but obviously it's less data so queries that return dates/times will be less accurate. It's up to you

Cheers,
Stu.

[Minoic]

Hello Stu,
data is clean and WeatherCat is running since two hours!!!!

Had to convert approx. 50K lines of old data into the LWC files, then imported the .wlk files and finally transferred the folder to the machine that is supposed to run WeatherCat from now on.
After starting WC the LWC files were automatically backed up and the party started. I decided to change the WeatherLink IP logger to a 15 minutes interval. From that point on WC is logging records every minute although the Data Viewer/Editor shows records every few minutes.

Any ideas what this can be? I can roll back anytime because if have all data ready to restart if need be.

thanks
George

[WCDev]

Hi George,
Good to hear you got the data across.

WeatherCat's sampling rate is independent of the logger period - by the sounds of it adaptive sampling is on - in this case a sample won't be recorded until the data changes (or the sampling period is reached, whichever comes first) - more info in the manual on page 128.

Cheers,
Stu.

[Minoic]

Stu,
got it now. I should have read the manual!!!!!

Thanks