Author Topic: WC background workload  (Read 5151 times)

xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
WC background workload
« on: August 27, 2015, 05:31:26 PM »
OK, just for testing purposes, I've changed the update intervals for most pages of my "in-progress" site to Prime numbers. Rarely updated pages such as Station info and Partners, use 61 minutes. The webcam movie uses 11 minutes.

Questions for Stu:
1. I assume the number of CUSTOMGRAPH$ and GAUGE$ affects the workload of WC. Would setting the update/uploading times for those two groups to different times (say, 2 and 3 minutes, or some other small Prime number) help 'spread the load'?

2. Does the uploading of the webcam movie usually take more time than it's 'construction'?

3. I assume reducing the number of movie 'frames' (individual camera jpeg images) will speed up the movie creation time.
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


awilltx

  • Guest
Re: WC background workload
« Reply #1 on: August 27, 2015, 09:15:49 PM »
I know you didn't ask this question of me, but here is what I know about the WC daily movie:


The file "WeatherCatDailyMovie" in your WeaterCatMedia/Movies/Today folder is constantly being updated with still images from your camera. When it gets to the top of the hour, the name is changed to WeatherCatDailyMovie_hour.mov. If you look in the WC log around the top of the hour, you will see a message telling you the name has been changed. The current file begins to be built anew.


Back when I was uploading the DailyMovie file to my site, I did not use the WC tag for uploading as it was taking too long. Instead, I wrote an Apple Script to ftp the file at 59 minutes after the hour every hour and set up cron to do it outside of WC Custom Web uploading. It is a pretty good way if you are having issues with upload times.


I use Transmit for ftp and can send you a copy of my Apple Script if you so desire. You could probably write one just as easily for Terminal.


I generate graphs for my site and have never had a problem with the upload times. Gauges I don't know about!


Hope this helps some,


Alan


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
You can monitor it. (Re: WC background workload)
« Reply #2 on: August 27, 2015, 11:49:30 PM »
Dear X-Air, Alan, and WeatherCat sys-admins,

OK, just for testing purposes, I've changed the update intervals for most pages of my "in-progress" site to Prime numbers. Rarely updated pages such as Station info and Partners, use 61 minutes. The webcam movie uses 11 minutes.

If you are curious, you can get a very good idea of how much system resources WeatherCat is using by running Activity Monitor while WeatherCat performs various activities.  You can see what WeatherCat is doing by keep the log window open.  So if you have log open and Activity Monitor going, you can see how much CPU is used to generate the custom web pages for example.

You might want to run something like XRG:

http://www.gauchosoft.com/Products/XRG/

There are plenty of little monitoring tools like this, so if you have another favorite of course that's fine.  XRG has one advantage it is completely free.  I keep the XRG window on the left side of every space I have on my computer.  So if I notice my Mac slowing down, I can immediately glance and see if I spot a problem.  Once XRG alerts me to a potential problem, I fire up Activity Monitor to diagnose the troublemaker.

Cheers, Edouard  [cheers1]

xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: WC background workload
« Reply #3 on: August 28, 2015, 02:33:12 AM »
Thanks for the info, Alan. Right now I'm just running on my main machine, so when I move to the Mac mini, WC will be the only app running. It's a slower machine, but not much going on, other than the OS.

AM and log reading are also good ideas. We'll just wait and see how things work on the mini.

Too much prep for the trip right now! ;)
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: WC background workload
« Reply #4 on: August 28, 2015, 06:34:38 PM »
Downloaded that monitor app. Quite impressive. Might be nice to have it as a Menu Bar icon...

Some interesting data in the WC log. Appears that WC is smart enough to delay generating all the (15) Gauges if the time for generating a different page pops up. It then continues the gauge generation function as needed. It seems to interweave page/tag/image generation functions to allow completing as many as possible at one time by delaying generation of gauges and, possibly graphs.
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Menu bar app also exists (Re: WC background workload)
« Reply #5 on: August 28, 2015, 10:37:13 PM »
Dear X-Air and WeatherCat utility gluttons,

Downloaded that monitor app. Quite impressive. Might be nice to have it as a Menu Bar icon...

Actually, somebody else has written just such an app:  iStat Menus:

https://bjango.com/mac/istatmenus/

However, if you want that extra convenience you'll have to pay for it.  I personally prefer having my system status in a separate window.  I just keep all my other windows moved over a bit so that XRG has a slice on the lower left corner of the monitor for every space.

Some interesting data in the WC log. Appears that WC is smart enough to delay generating all the (15) Gauges if the time for generating a different page pops up. It then continues the gauge generation function as needed. It seems to interweave page/tag/image generation functions to allow completing as many as possible at one time by delaying generation of gauges and, possibly graphs.

We are really lucky to have a first-class programmer like Stu interested in writing a program like WeatherCat.  I don't know how much Stu gets paid for his day job, but I would expect there are few programmers outside of Apple, Adobe, and other Mac software powerhouses that know the Mac and OS X as well as Stu does.  I wish I knew 1/10 of what Stu does!

Cheers, Edouard  [cheers1]

P.S. Thanks Stu!!  [tup]

Blicj11

  • Storm
  • *****
  • Posts: 3941
    • EW3808
    • KUTHEBER6
    • Timber Lakes Weather
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP Data Logger | iMac (2019), 3.6 GHz Intel Core i9, 40 GB RAM, macOS Ventura 13.6 | Sharx SCNC2900 Webcam | WeatherCat 3.3 | Supportive Wife
Re: Menu bar app also exists (Re: WC background workload)
« Reply #6 on: August 28, 2015, 10:45:05 PM »
Actually, somebody else has written just such an app:  iStat Menus:

I used to use iStat Menus and really liked it. Then one year,  when Apple came out with a new OS, iStat Menus required a payment to continue to use it and that's when I gave up menu convenience and saved myself a few bucks. I'm cheap, I admit.

We are really lucky to have a first-class programmer like Stu interested in writing a program like WeatherCat.
---snip---
Thanks Stu!!  [tup]

Amen! Well said, mate.
Blick


WCDev

  • WeatherCat Developer
  • Administrator
  • Storm
  • *****
  • Posts: 2911
    • CW9739
    • ISCOTLAN25
    • Trixology
  • Station Details: Main Station: Vantage Pro-2, 24hr fars, solar, soil/leaf station, extra temp stations, no U.V. WeatherLink IP.
Re: WC background workload
« Reply #7 on: August 29, 2015, 09:05:28 AM »


Questions for Stu:
1. I assume the number of CUSTOMGRAPH$ and GAUGE$ affects the workload of WC. Would setting the update/uploading times for those two groups to different times (say, 2 and 3 minutes, or some other small Prime number) help 'spread the load'?

2. Does the uploading of the webcam movie usually take more time than it's 'construction'?

3. I assume reducing the number of movie 'frames' (individual camera jpeg images) will speed up the movie creation time.

1. Not much.
2. There isn't anything to construct - it's already constructed - the MOVIE$ will only upload on the first upload past the hour - it'll upload the movie for the previous hour.
3. No, the movie is constructed as time moves forward (at the rate you set in the webcam preferences), so at the end of the hour, it's all done (and ready for upload by the MOVIE$ tag).

Here's a little background into how it all works.
WeatherCat is highly multithreaded - internally it is made up of a bunch of managers, each of these runs on its own thread - typically there'll be 30 or 40 whilst WeatherCat is running. There are managers for the webcam, movie creation, stats, data management, time, weather station hardware, plugins, custom web, simple web and so on.

(In addition, there are a bunch of plug-ins that plug into WeatherCat (via the plugin manager) - the station driver for your hardware, banner generator, on-line services (weather bug for example), live data view, SQL driver, current conditions calculator - again, these all run on their own threads)

When a manager (or plug-in) in WeatherCat wants some information (or to do something), it sends a message to another manager (or plugin) to get the data or perform the action - the reply will come in some time later (either with the data it was after or the result of the action it wanted carried out).

This has two benefits - 1: It spreads the workload across your processing cores, 2: it means there are no dependencies between these things - they have to have defined interfaces with defined messages, which makes programming easier for me :)

So in the case of processing your custom web templates, the custom web manager will start reading your files - as it sees a tag, it'll dispatch it to the right manager to get the data - for example a STAT$ tag is dispatched to the stats manager. Sometime later the result comes in - in this case it'll be very quick but the custom web processing doesn't wait for this result - it moves on to the next tag. As a side note, this is the reason why a tag like DOTSEPARATORS$ (forces decimal points to be a full stop character rather than localised) applies to the whole file - WeatherCat will look for these kind of tags before doing any other processing.

Anyway, once it has all the results, it does the replacement of the tags, saves the resulting file then moves on to the next file in your template set.

Some tags take longer than others to process - I think the longest one to process is any of the webcam image tags (WCAM$ for example). The custom web manager sends a message to the webcam manager saying get me the current image please. The webcam manager starts looking at the images coming in from the camera  - when they are consistent (i.e. for black level, colour balance,  contrast etc) it saves an image then replies back the custom web manager that the image is ready - the tag result is now available and the custom web manager will add the image to the upload queue.

This analysis of the images can take a long time - if the images are coming from a URL or a file on disk, then it doesn't take much time at all. If however it's a locally connected USB camera then it can take longer - USB webcams when not in use may go to sleep and when they wake up it can take a while for the image to stabilise - this is what the webcam manager is looking for before it grabs an image (it actually does this every time it grabs an image for whatever client - whether it be custom web, the movie manager, wunderground or just to save an image to disk (as set in your webcam preferences).

Clients of the webcam manager can also request the image is scaled in some way i.e. WCAM320$ - it'll do this processing after the above analysis and capture before returning the result to the client.

You can track the custom web processing in WeatherCat's log window - you'll see messages like:
Code: [Select]
08:27:56 29-Aug-2015:  CustomWeb: Generating web cam picture.
 08:27:56 29-Aug-2015:  CustomWeb: Waiting for web cam picture to stabilise.
 08:28:02 29-Aug-2015:  CustomWeb: Getting web cam image files.

Finally, MenuMeters is pretty good - http://www.ragingmenace.com/software/menumeters/ - everything here runs these (doesn't run on 10.11 however, possibly due to system integrity protection?).

Speaking of 10.11 - I wouldn't recommend it for WeatherCat at the moment - you will have issues. I have a beta build incoming which fixes all the problems as of 10.11 beta 7, which you should wait for if considering trying the 10.11 beta.

Hope that helps!

Blicj11

  • Storm
  • *****
  • Posts: 3941
    • EW3808
    • KUTHEBER6
    • Timber Lakes Weather
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP Data Logger | iMac (2019), 3.6 GHz Intel Core i9, 40 GB RAM, macOS Ventura 13.6 | Sharx SCNC2900 Webcam | WeatherCat 3.3 | Supportive Wife
Re: WC background workload
« Reply #8 on: August 29, 2015, 03:45:20 PM »
Stu, thanks for taking the time to provide us with the explanation of how WC processes tags. I found it quite interesting.

After I read your post, I read in the User Manual (gasp!) the Introduction and Concepts section that starts on Page 13. After having read your post here, the concepts explained in the User Manual make a lot more sense. Stu, I suggest you consider, for your next update of the User Manual, editing the Introduction & Concepts section to include parts of your more thorough posting here to explain the multithreading and tag processing concepts.
Blick


xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: WC background workload
« Reply #9 on: August 29, 2015, 04:24:26 PM »
Thanks, also, from me! You make it sound so simple, but I'm assuming you didn't just start out coding and then come up with the overall concept/scheme! It's a bit late for me to wrap my head around OOP, not to mention threads, managers, and plug-ins! My younger son does this for a living and can't seem to understand why his 'old man' can't create some app! "It's just typing, after all!" [computer]

Watching the log is very interesting. I've seen those plugInManager loading and a couple dozen station brand initialization steps plus initializing all the possible services we might be sharing with. At first I thought it would save some start up time by initializing only the items we have enabled, but the whole thing takes less than 30 seconds! Parsing the user defined list might not save anything and could even add startup time! It's details like that that show you've done so much work and thought into making this a premier app, in any field. Truly the mark of experience and talent [cheer] Your explanation, in easy to understand terms only confirms my respect and admiration! [tup]
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: WC background workload
« Reply #10 on: August 29, 2015, 04:32:37 PM »
OK, Stu, here's something that might indicate a bit of extra communications delay in my particular setup. I noticed the following item during the start up of WC: "***COULD NOT START WEATHER SERVER - THERE IS A WEATHERCAT ALREADY RUNNING ON THIS NETWORK***"

I assume this means that the instance of WC running on my iMac is using at least some kind of server (WEATHER SERVER) on the Mac mini also running WC for my 'live' site. I'm not sure, nor do I need to know what that server does, but it may be causing micro second delays in communications. Of course, it might also be reducing the cpu needs on the iMac instance! Just makes me aware that you've considered the consequences of running two (or more) instances of WC, even on different machines! ;)
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Thanks Stu! (Was: WC background workload)
« Reply #11 on: August 29, 2015, 10:51:39 PM »
Thanks Stu for the explanation!

I always thought WeatherCat was a very fancy piece of software, but it is even more slick than I imagined!

Cheers, Edouard

Blicj11

  • Storm
  • *****
  • Posts: 3941
    • EW3808
    • KUTHEBER6
    • Timber Lakes Weather
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP Data Logger | iMac (2019), 3.6 GHz Intel Core i9, 40 GB RAM, macOS Ventura 13.6 | Sharx SCNC2900 Webcam | WeatherCat 3.3 | Supportive Wife
Re: WC background workload
« Reply #12 on: August 31, 2015, 04:01:43 AM »
Xair:

The WeatherCat Server is the service that you run to provide data to WeatherCat Clients. If you are not running a WeatherCat Client you can turn off this service. See page 116 in the User Manual.
Blick


xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: WC background workload
« Reply #13 on: August 31, 2015, 02:42:49 PM »
Since WC has "clients", does it get any kind of renumeration? If so, does it directly to Stu, or do we need to pass it along? Probably a great thing that he checks for this kind of thing, otherwise, my "in-progress", dummy-data site would be dumping bogus info to those "clients" giving WC an unfair reputation! :o
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


wurzelmac

  • Storm
  • *****
  • Posts: 1361
    • ITIROLPR2
    • Wetterstation Pr?graten am Gro?venediger
  • Station Details: Davis Vantage PRO2 Plus (24h fan aspirated, wireless) with UV/Solar | Weather Envoy Data Logger | Mac mini 2023 M2 8/256 | 1x Canon EOS 1300D | macOS Sonoma 14.2
Re: WC background workload
« Reply #14 on: October 04, 2015, 09:20:26 AM »
Finally, MenuMeters is pretty good - http://www.ragingmenace.com/software/menumeters/ - everything here runs these (doesn't run on 10.11 however, possibly due to system integrity protection?).

I have done the workaround described here and it works for me!

Cheers,
Reinhard
Reinhard