Author Topic: Querying the DataLogger??  (Read 2823 times)

SalemBottomWeather

  • Calm
  • *
  • Posts: 5
    • N3OYL
    • KMDWESTM23
  • Station Details: Vantage Vue>Davis USB Logger>MeteoBridge
Querying the DataLogger??
« on: November 25, 2015, 12:04:52 AM »
Hi all,

I have installed the trial version of WeatherCat on my MacBook Pro and am considering a purchase.  The PC software is reading and displaying the live stream from my station effortlessly!!!  But have a couple of questions about the historical data and I am hoping that someone can help me understand.

First, I do not intend to run WeatherCat 24/7?.rather use it to pull down the saved data from the logger and then run reports, plots, and charts.  Will also be viewing the live stream but will be running WC 4 or 5 times a week; each session lasting anywhere from 5 minutes to a couple of hours.  One of my primary objectives is to build and maintain a complete historical record.

Therefore, I am really trying to understand how and when WeatherCat retrieves data from the DataLogger.

*) Does WC only query the Logger at startup?and only if some amount of time has passed since the last query?  How much time needs to pass?
*) Is the most the most recent hour retrieved?  When I view the data in the DataViewer, the most recent hour is always missing.
*) If the ?Delete Log After Historical Download? is *not* checked, does it download the entire contents of the logger each and every time?  Because it takes 3 or 5 minutes to complete and runs through all 512, I suspect this is what is happening.
*) If the ?Delete Log After Historical Download? is checked, does WC perform any kind of verification that it has successfully pulled all data before deleting?  Otherwise, any data that may have been lost in transmission (and not successfully downloaded) would be erased.
*) Let?s assume WC has *not* been running for 4 or 5 days (and assume that the DataLogger is not ?full?): will WC retrieve all data stored in the logger or just the last day or two?
*) Are there any known reasons why or when WC would not retrieve data from the Logger?

My setup is a Vantage Vue > USB Davis Logger > MeteoBridge.

It will be greatly appreciated if anyone is willing to explain some of these details and help me understand how WC interfaces with the DataLogger!!!  As far as the rest of the product, it seems fantastic and well worth the investment!

Bull Winkus

  • Storm
  • *****
  • Posts: 782
    • EW0095
    • KARHORSE2
    • WU for Horseshoe Bend, Arkansas
  • Station Details: Davis Wireless Vantage Pro 2, iMac 24"
Re: Querying the DataLogger??
« Reply #1 on: November 25, 2015, 03:41:54 AM »
Downloaded with WeatherCat (WC) was the WeatherCat User Manual. If you'll reference the Table of Contents, you'll see that Station Options starts on page 68. Flipping through the pages from there, Davis Vantage/Vue Range and a granular explanation of the data-logger's operation starts on page 70.

The Historical Download is not the same as the real time fetch that WC does while running. If WC has been offline, then upon starting it catches up by downloading everything in the data-logger. Thereafter, the program gets live updates. I don't know the exact interval, but it seems almost real time to me. If you are seeing it to be behind by an hour, then there must be a setting wrong somewhere.

The data-logger is simply a memory buffer. If one does not check the flag to "Delete Log After Historical Download," then the data remains while the logger continues to fill, eventually using all of its memory. And you are correct, this increases the amount of time that WC takes to download and parse the data with its historical record. (It builds a timeline, eliminating duplicates) The reason for the option to not erase is, if your downloaded data were to be lost due to HD failure or some other misfortune, you still would have the 2560 samples in the data-logger queue that would automatically be loaded when WC starts. The logger, meanwhile, is still in the process of gathering data. You would then be able to restore from backup the most recently saved history and carry on without lost data. But suppose your last saved backup was a week old. ? Well, if your data-logger's logging interval was set to 15-minute intervals, the logger, when full, would contain 640 hours or 26 days of data, bridging the gap with your missing backup data. However, if you're using Time Machine's 1 hour system of backup, then you would not need to maintain such an extensive record on the data-logger's queue, and could probably let the program Delete the Log After Historical Download. When it would delete the log again without a restart is something someone else may answer. I really don't know.

I hope I've cleared at least a few things up. Perhaps someone else can fill in the missing pieces.

 [cheers1]
Herb

Blicj11

  • Storm
  • *****
  • Posts: 3941
    • EW3808
    • KUTHEBER6
    • Timber Lakes Weather
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP Data Logger | iMac (2019), 3.6 GHz Intel Core i9, 40 GB RAM, macOS Ventura 13.6 | Sharx SCNC2900 Webcam | WeatherCat 3.3 | Supportive Wife
Re: Querying the DataLogger??
« Reply #2 on: November 25, 2015, 04:38:15 AM »
First, welcome to the Forum and I hope you enjoy your 14 day trial period with WeatherCat. I started out that way and then purchased. Glad I did.

Second, there is some excellent additional information about WeatherCat in the Wiki at http://wiki.trixology.com/index.php/Main_Page

Specifically, take a look at this article that explains the data logger sampling options available through WeatherCat: http://wiki.trixology.com/index.php/Capacity_of_the_Davis_data-logger_to_store_weather_data_depending_on_the_chosen_sampling_interval
Blick


AssortedNuggets

  • Calm
  • *
  • Posts: 4
  • Station Details: Davis II
Re: Querying the DataLogger??
« Reply #3 on: November 25, 2015, 05:13:21 PM »
Excellent, my first post and my second question is already answered on the front page.

Brilliant ....

SalemBottom Weather, are you using the same USB logger that the PC was using for the Davis?

So no difference between the PC and Mac data stream from the logger? Weather cat can read it just fine?

Thank you

Blicj11

  • Storm
  • *****
  • Posts: 3941
    • EW3808
    • KUTHEBER6
    • Timber Lakes Weather
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP Data Logger | iMac (2019), 3.6 GHz Intel Core i9, 40 GB RAM, macOS Ventura 13.6 | Sharx SCNC2900 Webcam | WeatherCat 3.3 | Supportive Wife
Re: Querying the DataLogger??
« Reply #4 on: November 25, 2015, 05:26:10 PM »
Welcome to the forum Assorted Nuggets!

The physical USB data logger from Davis is exactly the same for the PC and the Mac. The only difference is in the accompanying software, which is not used for anything if you are using WeatherCat. The only thing the Mac WeatherLink software allows you to do that you cannot do with WeatherCat is your initial setup on a Davis Envoy. If you are using the Vue or Vantage stations, you don't need the Mac software from Davis.
Blick


AssortedNuggets

  • Calm
  • *
  • Posts: 4
  • Station Details: Davis II
Re: Querying the DataLogger??
« Reply #5 on: November 25, 2015, 07:39:00 PM »
And more questions answered, thank you, one Christmas present down....imminent purchase ....

Many thanks for speedy reply.

A

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Running down your questions in detail (Re: Querying the DataLogger??)
« Reply #6 on: November 25, 2015, 11:35:14 PM »
Dear SalemBottomWeather, Herb, Blick, AssortedNuggets, and WeatherCat Davis fans,

Since you went into great detail in your questions, let me try to answer them in detail.  I'm not the WeatherCat developer, but I'm a longtime user with a Davis Vantage Pro-2.

*) Does WC only query the Logger at startup?and only if some amount of time has passed since the last query?  How much time needs to pass?

In short yes.  If WeatherCat is running, then it gets current conditions directly from the station so there is no need to involve the data logger.  The amount of time needed before it will query the data logger depends on sampling rate for WeatherCat (see preferences/misc2.)  So if you have your sampling rate set to 5 minutes and you quit WeatherCat for less than 5 minutes, WeatherCat will not need to sample the data logger.

*) Is the most the most recent hour retrieved?  When I view the data in the DataViewer, the most recent hour is always missing.

I'm not sure what you mean by this.  What is the sample rate you have chosen?

*) If the ?Delete Log After Historical Download? is *not* checked, does it download the entire contents of the logger each and every time?  Because it takes 3 or 5 minutes to complete and runs through all 512, I suspect this is what is happening.

In short yes.  The data logger is in effect a circular buffer.  WeatherCat downloads the entire logger and then compares the timestamps of the data it already has to the contents of the logger.  It then matches up what is missing from the WeatherCat data and fills the missing portions from the data logger.

*) If the ?Delete Log After Historical Download? is checked, does WC perform any kind of verification that it has successfully pulled all data before deleting?  Otherwise, any data that may have been lost in transmission (and not successfully downloaded) would be erased.

I don't believe WeatherCat does this, but there is no reason to delete the data logger during normal operation.  The only reason purge the data logger is if you suspect there is some corrupted data.  Another time you might want to do this is if you have performed some maintenance on your station that might have introduced erroneous data.  Corrupted data loggers are extremely rare.  I'm not aware of any WeatherCat user ever having this occur and my station is over 5 year old now.

*) Let?s assume WC has *not* been running for 4 or 5 days (and assume that the DataLogger is not ?full?): will WC retrieve all data stored in the logger or just the last day or two?

As I said, WeatherCat will not purge the logger unless you explicitly request it to do so.  WeatherCat retrieves all the data, but then matches that data to what is missing and only adds what was not already existing to the WeatherCat database.

*) Are there any known reasons why or when WC would not retrieve data from the Logger?

You could ask the WeatherCat developer for full details, but the only reason WeatherCat wouldn't download data is some sort of malfunction which are vanishingly rare.

First, I do not intend to run WeatherCat 24/7?.rather use it to pull down the saved data from the logger and then run reports, plots, and charts.  Will also be viewing the live stream but will be running WC 4 or 5 times a week; each session lasting anywhere from 5 minutes to a couple of hours.  One of my primary objectives is to build and maintain a complete historical record.

Most of us do not run our stations like this because it limits what WeatherCat can do.  However, when our house was damaged by a lightning strike, I was forced to download data periodically as you propose for about a week until I was able to make repairs to my data connection.  WeatherCat performed as you desire it, perfectly.  I lost no data by allowing my Davis station to store the conditions in the data logger and then running WeatherCat periodically to download the data onto my Mac.  You can set up your station in this way in full confidence.

Cheers, Edouard

SalemBottomWeather

  • Calm
  • *
  • Posts: 5
    • N3OYL
    • KMDWESTM23
  • Station Details: Vantage Vue>Davis USB Logger>MeteoBridge
Re: Querying the DataLogger??
« Reply #7 on: November 26, 2015, 01:11:34 AM »
Thank you everyone for all the detailed and well-documented information...very helpful!!  All of my questions have been answered except for the "last hour missing after downloading".  I received an email from the support team re: this and they suggest I verify the time and date on the console matches that of the computer.  A visual inspection shows that they do match...but I am going to do a test tonight and let WC sync the times at 12:15 AM.

Here is a step by step description of what I am doing and what I am seeing:  Start WC; it downloads from the logger; I view the data in the "Data View" tool; The last record (at the bottom) is an hour behind the current time (there might be one record with the current time; in which case the hour gap is just before it).

Update: Inspection of the .cat file with a text editor shows that those records are present...but that the gap is shifted!!  In this data file I see the gap between the end of the last live stream and the beginning of the logger download.

All the more reason to believe it might be a time differential problem.

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Might be this bug. (Re: Querying the DataLogger??)
« Reply #8 on: November 26, 2015, 10:17:50 PM »
Dear SalemBottomWeather and WeatherCat fans,

Thank you everyone for all the detailed and well-documented information...very helpful!!  All of my questions have been answered except for the "last hour missing after downloading". 

. . . .

All the more reason to believe it might be a time differential problem.

From your description, it sounds similar to a bug that was reported earlier this month.  Here is the thread about it:

http://athena.trixology.com/index.php?topic=1914.msg17264#msg17264

At the end of the thread, Stuart Ball (WeatherCat developer) reported that he had identified the issue and that it would be fixed in the next release of WeatherCat.

Cheers, Edouard

SalemBottomWeather

  • Calm
  • *
  • Posts: 5
    • N3OYL
    • KMDWESTM23
  • Station Details: Vantage Vue>Davis USB Logger>MeteoBridge
A time shift with each shutdown/re-start ---- Re: Querying the DataLogger??
« Reply #9 on: November 27, 2015, 04:35:01 AM »
I have verified that the time on my Davis console is the same as my computer and the missing hour problem continues to exist.

I’ve discovered by inspecting the .cat file with a text editor that the data records are actually there…its just that the time stamp on every record is being shifted backward by an hour with each shutdown/re-start/Logger-retrieval of WeatherCat.  This shifting is causing the hour gap to appear.

Here is a sample of 3 records.  Attached is a screenshot taken of the .cat file just after 8:30 pm.  Notice that the time stamps on records #158, 159 and 160 are 20:36, 20:37 and 20:38 respectively.

Following this, I then shutdown WeatherCat; waited about an hour; and started a new session around 9:45 pm.  WC updated from the logger upon startup (the console/PC times are synced).  Attached is another screenshot taken of the same records in the cat file after this session.  Please note that the time stamps on records #158, 159, and 160 are now 1 hour earlier than previous recorded.  The rest of the data elements on the records are exactly the same…just the time stamps have changed.

This time shift is happening on all records in the .cat file.

I have tried various scenarios and settings when shutting down/restarting/logger-retreival…even deleted the app, the data, and re-installed.   Every time I shut down WC and restart sometime later with a retrieval from the logger, the time shift occurs.  It has a cumulative effect from session to session…continuing to shift backward by an an hour with each shutdown/re-start/retrieval.

I am also attaching the two full snapshots of the .cat file taken from these two sessions.

Is there a setting somewhere that I have missed?

SalemBottomWeather

  • Calm
  • *
  • Posts: 5
    • N3OYL
    • KMDWESTM23
  • Station Details: Vantage Vue>Davis USB Logger>MeteoBridge
Re: Querying the DataLogger??
« Reply #10 on: November 27, 2015, 12:29:11 PM »
....just to be clear: The time shift always happens when i wait the sufficient amount of time after shutting down so that WC will retrieve from the logger upon restart.  The time on my Davis console, meteobridge, and computer are all in-sync.

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Try reposting in bugs. (Re: A time shift with each shutdown/re-start)
« Reply #11 on: November 27, 2015, 09:55:51 PM »
Dear SalemBottomWeather and WeatherCat troubleshooters,

I have verified that the time on my Davis console is the same as my computer and the missing hour problem continues to exist.

Unfortunately, I'm not aware of a problem, but this might be something that requires the attention of Stuart Ball, the WeatherCat developer.  Either repost this question in the WeatherCat bugs topic of the forum:

http://athena.trixology.com/index.php?board=7.0

or contact WeatherCat technical support directly by following the instructions on this page:

http://trixology.com/contact-us/

You might not get any reply until the weekend.  Stu is a busy guy.

Cheers, Edouard

WCDev

  • WeatherCat Developer
  • Administrator
  • Storm
  • *****
  • Posts: 2911
    • CW9739
    • ISCOTLAN25
    • Trixology
  • Station Details: Main Station: Vantage Pro-2, 24hr fars, solar, soil/leaf station, extra temp stations, no U.V. WeatherLink IP.
Re: Querying the DataLogger??
« Reply #12 on: November 29, 2015, 11:07:47 AM »
Hi SalemBottomWeather,
Thanks for emailing the details in, I'll take a look at it.