Author Topic: Additional Files Syncing/Timing  (Read 1646 times)

xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Additional Files Syncing/Timing
« on: November 02, 2017, 08:51:59 PM »
Earlier this week I had several "FTP Overdrive" warnings, 3 on one day. I stopped uploading the WC hourly video file around a week ago. The vid is still being created but no longer uploaded. I would have thought that this would have reduced the FTP load, at least once every hour.

I've changed the timings on the four Additional Files that use FTP. They now include odd fractions of a minute, so they should never require uploading at the same time as anything else.

But the question I have concerns the actual times these ftp operations occur.
  • Do the settings for Additional Files run at a time based on the start of a day similar to a cron job setting?
  • Does the 'clock' start when WC 'sees' a change in the Additional Files settings?
  • Something else

I'm not sure I understand the meaning of Note2 on page 137 of The Manual:
Quote
If Custom web is synchronised to weather sampling, additional files will not be processed whilst custom web is waiting for the sync (i.e. additional files are queued whilst custom web is waiting for sync).
I think that means if an Additional File would normally be uploaded and WC is fetching data just before that time, the file upload will be delayed until the fetching completes (plus any processing time for tags in that file). Other info that may be applicable:
  • Sampling Rate: 2 minutes (Adaptive selected)
  • Update Web Pages Every: 2 minutes (Sync: To Weather Fetch selected)
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Simple counter from WeatherCat launch (Re: Additional Files Syncing/Timing)
« Reply #1 on: November 03, 2017, 12:01:48 AM »
Dear X-Air and WeatherCat web spinners,

But the question I have concerns the actual times these ftp operations occur.
  • Do the settings for Additional Files run at a time based on the start of a day similar to a cron job setting?
  • Does the 'clock' start when WC 'sees' a change in the Additional Files settings?
  • Something else

Okay that's a good question and I'm not sure I know the answer.  We might have to ask Stu.  Looking at my WeatherCat log it appears that the additional files are simply processed at the interval you have chosen counting after WeatherCat launches and finally gets around to processing these files for the first time.

I'm not sure I understand the meaning of Note2 on page 137 of The Manual:
Quote
If Custom web is synchronised to weather sampling, additional files will not be processed whilst custom web is waiting for the sync (i.e. additional files are queued whilst custom web is waiting for sync).
I think that means if an Additional File would normally be uploaded and WC is fetching data just before that time, the file upload will be delayed until the fetching completes (plus any processing time for tags in that file). Other info that may be applicable:
  • Sampling Rate: 2 minutes (Adaptive selected)
  • Update Web Pages Every: 2 minutes (Sync: To Weather Fetch selected)

I'm not certain, but I believe what Stu is saying is if the processing time for an additional file happens while WeatherCat is processing the custom web, WeatherCat won't process that additional file in parallel.  Instead, the additional file is effectively added to the queue of items to process and only is finally processed after the custom web data has been generated and uploaded.  Since custom websites can be very large, this could delay the processing of the additional files by a noticeable amount.

I think "under the hood" all the FTP processing is using the same code in a single thread.  As a result, there isn't any practical way to try to process an additional file simultaneously as all the files of a custom web.  I really doubt it has any effect.  It means that for example services like AWEKAS might not get their data as promptly as they would like, but the world will go on.

Cheers, Edouard  [cheers1]

xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: Additional Files Syncing/Timing
« Reply #2 on: November 03, 2017, 12:57:52 AM »
Quote
...but the world will go on
In this day and age, that's a pretty bold statement!! [lol]
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: Additional Files Syncing/Timing
« Reply #3 on: November 03, 2017, 03:33:58 PM »
My questions result from several "FTP Overload" warnings a few days ago, 3 in one day, 2 on another. I've never seen so many even in a multi-month period. I had been adjusting the timings of the four Additional files and I had even stopped uploading the Daily(hourly)Movie. That should have reduced the FTP uploading bandwidth, although not the WC processing load. I have also re-adjusted the timings for those four files by including fractions of a minute, so as to avoid any chance of processing the normal two minute data gathering/updating and these four file uploads. Only been two days so far without 'overloads' and I suspect it makes not much difference to WC, anyway. ;)
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


Blicj11

  • Storm
  • *****
  • Posts: 3941
    • EW3808
    • KUTHEBER6
    • Timber Lakes Weather
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP Data Logger | iMac (2019), 3.6 GHz Intel Core i9, 40 GB RAM, macOS Ventura 13.6 | Sharx SCNC2900 Webcam | WeatherCat 3.3 | Supportive Wife
Re: Additional Files Syncing/Timing
« Reply #4 on: November 03, 2017, 08:50:09 PM »
I suspect it makes not much difference to WC, anyway. ;)

True. FTP overloads happen. You don't need to do anything unless they occur consistently and frequently. I get them sometimes and just ignore them.

If they really bother you, turn off the alerts and then you will be just as happy as you were before you knew about the FTP overload warning feature.
Blick


xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: Additional Files Syncing/Timing
« Reply #5 on: November 03, 2017, 09:59:13 PM »
Ah yes, the old "ignorance is bliss" theory! [lol]
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Dear X-Air, Blick, and WeatherCat types getting a little impatient for the second coming, . . .

Quote
...but the world will go on
In this day and age, that's a pretty bold statement!! [lol]

In a voice worthy of the protagonist and villain of  Edgar Allan Poe's short story "The Cask of Amontillado"

"True --true,"

As a teenager I had an audio-recording of this short story and the actor who read those lines was absolutely - perfect!

Cheers, Edouard  [cheers1]