Trixology
WeatherCat => WeatherCat General Discussion => Topic started by: xairbusdriver on August 27, 2015, 05:31:26 PM
-
OK, just for testing purposes, I've changed the update intervals for most pages of my "in-progress" site to Prime numbers. Rarely updated pages such as Station info and Partners, use 61 minutes. The webcam movie uses 11 minutes.
Questions for Stu:
1. I assume the number of CUSTOMGRAPH$ and GAUGE$ affects the workload of WC. Would setting the update/uploading times for those two groups to different times (say, 2 and 3 minutes, or some other small Prime number) help 'spread the load'?
2. Does the uploading of the webcam movie usually take more time than it's 'construction'?
3. I assume reducing the number of movie 'frames' (individual camera jpeg images) will speed up the movie creation time.
-
I know you didn't ask this question of me, but here is what I know about the WC daily movie:
The file "WeatherCatDailyMovie" in your WeaterCatMedia/Movies/Today folder is constantly being updated with still images from your camera. When it gets to the top of the hour, the name is changed to WeatherCatDailyMovie_hour.mov. If you look in the WC log around the top of the hour, you will see a message telling you the name has been changed. The current file begins to be built anew.
Back when I was uploading the DailyMovie file to my site, I did not use the WC tag for uploading as it was taking too long. Instead, I wrote an Apple Script to ftp the file at 59 minutes after the hour every hour and set up cron to do it outside of WC Custom Web uploading. It is a pretty good way if you are having issues with upload times.
I use Transmit for ftp and can send you a copy of my Apple Script if you so desire. You could probably write one just as easily for Terminal.
I generate graphs for my site and have never had a problem with the upload times. Gauges I don't know about!
Hope this helps some,
Alan
-
Dear X-Air, Alan, and WeatherCat sys-admins,
OK, just for testing purposes, I've changed the update intervals for most pages of my "in-progress" site to Prime numbers. Rarely updated pages such as Station info and Partners, use 61 minutes. The webcam movie uses 11 minutes.
If you are curious, you can get a very good idea of how much system resources WeatherCat is using by running Activity Monitor while WeatherCat performs various activities. You can see what WeatherCat is doing by keep the log window open. So if you have log open and Activity Monitor going, you can see how much CPU is used to generate the custom web pages for example.
You might want to run something like XRG:
http://www.gauchosoft.com/Products/XRG/ (http://www.gauchosoft.com/Products/XRG/)
There are plenty of little monitoring tools like this, so if you have another favorite of course that's fine. XRG has one advantage it is completely free. I keep the XRG window on the left side of every space I have on my computer. So if I notice my Mac slowing down, I can immediately glance and see if I spot a problem. Once XRG alerts me to a potential problem, I fire up Activity Monitor to diagnose the troublemaker.
Cheers, Edouard [cheers1]
-
Thanks for the info, Alan. Right now I'm just running on my main machine, so when I move to the Mac mini, WC will be the only app running. It's a slower machine, but not much going on, other than the OS.
AM and log reading are also good ideas. We'll just wait and see how things work on the mini.
Too much prep for the trip right now! ;)
-
Downloaded that monitor app. Quite impressive. Might be nice to have it as a Menu Bar icon...
Some interesting data in the WC log. Appears that WC is smart enough to delay generating all the (15) Gauges if the time for generating a different page pops up. It then continues the gauge generation function as needed. It seems to interweave page/tag/image generation functions to allow completing as many as possible at one time by delaying generation of gauges and, possibly graphs.
-
Dear X-Air and WeatherCat utility gluttons,
Downloaded that monitor app. Quite impressive. Might be nice to have it as a Menu Bar icon...
Actually, somebody else has written just such an app: iStat Menus:
https://bjango.com/mac/istatmenus/ (https://bjango.com/mac/istatmenus/)
However, if you want that extra convenience you'll have to pay for it. I personally prefer having my system status in a separate window. I just keep all my other windows moved over a bit so that XRG has a slice on the lower left corner of the monitor for every space.
Some interesting data in the WC log. Appears that WC is smart enough to delay generating all the (15) Gauges if the time for generating a different page pops up. It then continues the gauge generation function as needed. It seems to interweave page/tag/image generation functions to allow completing as many as possible at one time by delaying generation of gauges and, possibly graphs.
We are really lucky to have a first-class programmer like Stu interested in writing a program like WeatherCat. I don't know how much Stu gets paid for his day job, but I would expect there are few programmers outside of Apple, Adobe, and other Mac software powerhouses that know the Mac and OS X as well as Stu does. I wish I knew 1/10 of what Stu does!
Cheers, Edouard [cheers1]
P.S. Thanks Stu!! [tup]
-
Actually, somebody else has written just such an app: iStat Menus:
I used to use iStat Menus and really liked it. Then one year, when Apple came out with a new OS, iStat Menus required a payment to continue to use it and that's when I gave up menu convenience and saved myself a few bucks. I'm cheap, I admit.
We are really lucky to have a first-class programmer like Stu interested in writing a program like WeatherCat.
---snip---
Thanks Stu!! [tup]
Amen! Well said, mate.
-
Questions for Stu:
1. I assume the number of CUSTOMGRAPH$ and GAUGE$ affects the workload of WC. Would setting the update/uploading times for those two groups to different times (say, 2 and 3 minutes, or some other small Prime number) help 'spread the load'?
2. Does the uploading of the webcam movie usually take more time than it's 'construction'?
3. I assume reducing the number of movie 'frames' (individual camera jpeg images) will speed up the movie creation time.
1. Not much.
2. There isn't anything to construct - it's already constructed - the MOVIE$ will only upload on the first upload past the hour - it'll upload the movie for the previous hour.
3. No, the movie is constructed as time moves forward (at the rate you set in the webcam preferences), so at the end of the hour, it's all done (and ready for upload by the MOVIE$ tag).
Here's a little background into how it all works.
WeatherCat is highly multithreaded - internally it is made up of a bunch of managers, each of these runs on its own thread - typically there'll be 30 or 40 whilst WeatherCat is running. There are managers for the webcam, movie creation, stats, data management, time, weather station hardware, plugins, custom web, simple web and so on.
(In addition, there are a bunch of plug-ins that plug into WeatherCat (via the plugin manager) - the station driver for your hardware, banner generator, on-line services (weather bug for example), live data view, SQL driver, current conditions calculator - again, these all run on their own threads)
When a manager (or plug-in) in WeatherCat wants some information (or to do something), it sends a message to another manager (or plugin) to get the data or perform the action - the reply will come in some time later (either with the data it was after or the result of the action it wanted carried out).
This has two benefits - 1: It spreads the workload across your processing cores, 2: it means there are no dependencies between these things - they have to have defined interfaces with defined messages, which makes programming easier for me :)
So in the case of processing your custom web templates, the custom web manager will start reading your files - as it sees a tag, it'll dispatch it to the right manager to get the data - for example a STAT$ tag is dispatched to the stats manager. Sometime later the result comes in - in this case it'll be very quick but the custom web processing doesn't wait for this result - it moves on to the next tag. As a side note, this is the reason why a tag like DOTSEPARATORS$ (forces decimal points to be a full stop character rather than localised) applies to the whole file - WeatherCat will look for these kind of tags before doing any other processing.
Anyway, once it has all the results, it does the replacement of the tags, saves the resulting file then moves on to the next file in your template set.
Some tags take longer than others to process - I think the longest one to process is any of the webcam image tags (WCAM$ for example). The custom web manager sends a message to the webcam manager saying get me the current image please. The webcam manager starts looking at the images coming in from the camera - when they are consistent (i.e. for black level, colour balance, contrast etc) it saves an image then replies back the custom web manager that the image is ready - the tag result is now available and the custom web manager will add the image to the upload queue.
This analysis of the images can take a long time - if the images are coming from a URL or a file on disk, then it doesn't take much time at all. If however it's a locally connected USB camera then it can take longer - USB webcams when not in use may go to sleep and when they wake up it can take a while for the image to stabilise - this is what the webcam manager is looking for before it grabs an image (it actually does this every time it grabs an image for whatever client - whether it be custom web, the movie manager, wunderground or just to save an image to disk (as set in your webcam preferences).
Clients of the webcam manager can also request the image is scaled in some way i.e. WCAM320$ - it'll do this processing after the above analysis and capture before returning the result to the client.
You can track the custom web processing in WeatherCat's log window - you'll see messages like:
08:27:56 29-Aug-2015: CustomWeb: Generating web cam picture.
08:27:56 29-Aug-2015: CustomWeb: Waiting for web cam picture to stabilise.
08:28:02 29-Aug-2015: CustomWeb: Getting web cam image files.
Finally, MenuMeters is pretty good - http://www.ragingmenace.com/software/menumeters/ - everything here runs these (doesn't run on 10.11 however, possibly due to system integrity protection?).
Speaking of 10.11 - I wouldn't recommend it for WeatherCat at the moment - you will have issues. I have a beta build incoming which fixes all the problems as of 10.11 beta 7, which you should wait for if considering trying the 10.11 beta.
Hope that helps!
-
Stu, thanks for taking the time to provide us with the explanation of how WC processes tags. I found it quite interesting.
After I read your post, I read in the User Manual (gasp!) the Introduction and Concepts section that starts on Page 13. After having read your post here, the concepts explained in the User Manual make a lot more sense. Stu, I suggest you consider, for your next update of the User Manual, editing the Introduction & Concepts section to include parts of your more thorough posting here to explain the multithreading and tag processing concepts.
-
Thanks, also, from me! You make it sound so simple, but I'm assuming you didn't just start out coding and then come up with the overall concept/scheme! It's a bit late for me to wrap my head around OOP, not to mention threads, managers, and plug-ins! My younger son does this for a living and can't seem to understand why his 'old man' can't create some app! "It's just typing, after all!" [computer]
Watching the log is very interesting. I've seen those plugInManager loading and a couple dozen station brand initialization steps plus initializing all the possible services we might be sharing with. At first I thought it would save some start up time by initializing only the items we have enabled, but the whole thing takes less than 30 seconds! Parsing the user defined list might not save anything and could even add startup time! It's details like that that show you've done so much work and thought into making this a premier app, in any field. Truly the mark of experience and talent [cheer] Your explanation, in easy to understand terms only confirms my respect and admiration! [tup]
-
OK, Stu, here's something that might indicate a bit of extra communications delay in my particular setup. I noticed the following item during the start up of WC: "***COULD NOT START WEATHER SERVER - THERE IS A WEATHERCAT ALREADY RUNNING ON THIS NETWORK***"
I assume this means that the instance of WC running on my iMac is using at least some kind of server (WEATHER SERVER) on the Mac mini also running WC for my 'live' site. I'm not sure, nor do I need to know what that server does, but it may be causing micro second delays in communications. Of course, it might also be reducing the cpu needs on the iMac instance! Just makes me aware that you've considered the consequences of running two (or more) instances of WC, even on different machines! ;)
-
Thanks Stu for the explanation!
I always thought WeatherCat was a very fancy piece of software, but it is even more slick than I imagined! (http://www.canebas.org/WeatherCat/Forum_support_documents/Custom_emoticons/cool-emoticons-8.png)
Cheers, Edouard
-
Xair:
The WeatherCat Server is the service that you run to provide data to WeatherCat Clients. If you are not running a WeatherCat Client you can turn off this service. See page 116 in the User Manual.
-
Since WC has "clients", does it get any kind of renumeration? If so, does it directly to Stu, or do we need to pass it along? Probably a great thing that he checks for this kind of thing, otherwise, my "in-progress", dummy-data site would be dumping bogus info to those "clients" giving WC an unfair reputation! :o
-
Finally, MenuMeters is pretty good - http://www.ragingmenace.com/software/menumeters/ - everything here runs these (doesn't run on 10.11 however, possibly due to system integrity protection?).
I have done the workaround described here (http://member.ipmu.jp/yuji.tachikawa/MenuMetersElCapitan/) and it works for me!
Cheers,
Reinhard
-
Finally, MenuMeters is pretty good - http://www.ragingmenace.com/software/menumeters/ - everything here runs these (doesn't run on 10.11 however, possibly due to system integrity protection?).
I have done the workaround described here (http://member.ipmu.jp/yuji.tachikawa/MenuMetersElCapitan/) and it works for me!
Cheers,
Reinhard
Superb!
-
Found a something interesting after updating WC! While I was not seeing any errors on my site, I decided to see how things were going with Console. What greeted me, limited to WC messages, was about 50% error messages! :o
Things were scrolling by so fast I could barely grab fewer than 100 lines! I pasted them into a TextEdit window so I could keep from getting dizzy! [rolleyes2] What I saw was WC trying to access files from an incorrect directory!
After several attempts to correct things, mainly by changing the Custom Web -> Set HTML Source, did nothing to help. Console was still reporting that WC was looking at the wrong directory. I finally Quit WC, changed its plist name, and restarted.
The great news is that WC still knew about my Custom Gauges and Graphs! [cheer] Obviously, that info is stored elsewhere. Unfortunately, Console indicated that this had made absolutely no improvement in the error logging! [banghead]
By now I knew WC was not deciding, on its own, where to look for the files it wanted to modify/upload/use. I knew this because I had completely eliminated the directory that Console was reporting that WC was attempting to use! I had decided to "clean up" the paths on the drive by eliminating nearly duplicate folders. That's why I knew WC must have stored the path somewhere. You've all probably guessed where I had screwed up...
I finally figured out that I had failed (again?!) to realize what all the Preferences tabs were recording. When I finally looked at the Additional Files list, I could see that the incorrect path was still in those settings! No need to rename/move the plist, all I needed to do was edit the six files in the Additional Files list! DUH!! [rolleyes2] [computer]
All those ererz were caused by my own "house cleaning"!
Just as importantly to fixing this mistake, I also found that WC was wasting lots of cpu cycles and electrons by uploading some 30+ png's!
On my 'web cam' page, I also display two random images from a group stored on the server. But they are 'static' images, I may add/change them occasionally, but they don't need to be uploaded every two minutes! Apparently, WC looks at everything in the directory set in the "...HTML Source" pref. I have now removed those images and several php and html files that don't need updating regularly. This will, hopefully, very much lighten the workload on WC as well as drastically cut the bandwidth needed to update the site. How do I apologize to software for over-working it?! [lol] :P :( :-\
-
Hi xairbusdriver
Any time you trash a plist you must all so restart your computer it is still in cache and won't change if you don't restart your machine
cheer
[cheers1]
-
Well, I seldom "trash" any file for testing purposes. Who wants to use something that's been in the trash?! :o Simply renaming it suffices. And it's easier to see the new one created if you just change the end of the name a bit; the new one will usually show up right next to the old one.
However, I agree with your analysis, even though that was not a problem in this case. The plist was fine, I just was too slow in spotting what needed to be changed... a much too frequent problem in my life! [banghead] I couldn't see the trees for the forest! [blush]
-
While observing WC in Console for another purpose, I noticed that it was still uploading html files that are basically static. While they have images (gauges, graphs, and movie) in them, the html/php never changes. It's a small but needles waste of cpu cycles and electron movements. It probably makes less than .0001 mico-seconds difference to WC, but I need all the efficiency I can get due to other misteakes eye maik! [banghead]
-
Dear X-Air and WeatherCat web spinners,
While observing WC in Console for another purpose, I noticed that it was still uploading html files that are basically static. While they have images (gauges, graphs, and movie) in them, the html/php never changes. It's a small but needles waste of cpu cycles and electron movements. It probably makes less than .0001 mico-seconds difference to WC, but I need all the efficiency I can get due to other misteakes eye maik! [banghead]
You are correct that the load on your Mac is no problem. However, uploading the same files over and over eventually lead to a lot of uploaded data. Some web hosting providers have limited on the amount of data you can upload and in the old days even ISPs had such limits.
That is one of the best reasons to have a web template that dynamically updates instrument data instead of depending on uploading. My website is being uploaded far more often than really makes sense. That's why I want to also switch to some sort of Custom CGI setup on my web template like high-performance templates are doing. Oh well, perhaps if we get a really strong El Nin? I'll be stuck in the house long enough to take that project on.
Cheers, Edouard
-
uploading the same files over and over eventually lead to a lot of uploaded data.
In order to help Stu track down a minor "logged" error in Console, I edited out the file that I had created to get my Gauges created. The file is cleverly named "upload-gauges.html". It's a page that is not designed to ever be displayed, just an html file (could just as well be .txt. .php, etc.) containing the WC 'tags' for each gauge and 'encourages' WC to create and upload those images. I have similar files for the custom graphs and the movie. While my pages are already rather small since there are several embedded php scripts that insert html when the browser requests the page, I simply wanted to reduce the workload on WC by not uploading ("over and over") 2,000 characters when 200 would suffice.
By commenting out the html lines for half the gauges, I was attempting to stop WC from creating them. I even created/uploaded 'broken' images for the server so the browser didn't just display the normal 'broken' image icon.
I refreshed the 'gauges' page and reveled in seeing the 'broken' gauges. Much to my surprise, the next time WC updated things, I see that the actual gauges are back! Why? I double-checked the uploaded version of the "upload-gauges.html" file to confirm that the commented out line were still there. I had even 'crippled' the lines by changing the format/spelling of the tags I was trying to block.
Doing a multi-file search with BBEdit resulted in a new (for me) discovery! WC comes with a folder called "WeatherCat_all_tags" file. Inside the folder is an html file called "index". It has not been uploaded, fortunately, my "index.html" file is still on the server. I have thought this was simply a method of displaying the "WeatherCat Tags" list as a table.I don't believe this html file has anything to do with my question. The next paragraph seems to confirm that.
I have moved that folder out of the one that WC uses to access the "Additional Files" list. However, the images are still being created and placed in ~/Library/Application Support/WeatherCatCustomWeb, thus, they are uploaded to my server.
My question: How do I stop these gauges from being created without deleting them from my Tools->Custom Gauges area. I assume their details are in the <key>windowPositions1|2|3|4|5</key> dictionary, at least the ones with variations of the <string>276 182 177 122 0 0 1280 777 </string> entries. OTOH, those entries all have the same "122 0 0 1280 777" values, and the "177" is either that value or "377". [banghead]
The whole point of this exercise is to change the actions of WC to see how that change might affect the minor "logged" error.
-
I think I've discovered where the second upload of custom gauges was happening... sort of.
The partial "...HTML Source" setting in WC prefs is /WeatherCat 2.2/Web Templates/Mid-South. In that directory, until an hour ago were 7 files. 6 were files that needed updating constantly and they were all in the "Additional Files" list. However, there was a renamed file that was a copy of the gauges upload file; it was named "upload-gauges-original.html". The file it was a copy of had the last 6 WC gauge tags commented out. That, I thought would prevent those gauges from even being created, much less uploaded. I was wrong... again. [banghead]
As part of my investigation, I moved that duplicate, but renamed html file out of that path. WC promptly 'forgot' about it! The edited file, with the commented out tags, was now the only "gauges" used by WC, and it created/uploaded only the correct ones! [cheer]
Now for the kicker! After confirming this extra file was being used when I didn't want it, and not seeing any change in the "logged error" I was trying to figure out, I put that renamed, full of 15 WC gauge tags back into folder with the other 6 files. So far, WC has still ignored that file and is creating/uploading only the gauges in the file in the "Additional Files" list.
This behavior is exactly what I think should happen. But the question remains as to why WC continued to use a file that had not been specified in the "Additional Files" list, in the first place. [banghead] It appears that WC does not use the actual name of the file but a System (Unix) supplied code/number/identifier, instead. Even though I changed the name of the file, I made no other changes to it, nor did I delete the file with the original name from the "Additional Files" list. After I completely removed the file, WC could no longer 'see' it and was forced to re-inspect/re-build the list of files as set in the Prefs.
The conclusions above are strictly my own, and will probably be useless to anyone else, of course. [goofy] [runoff]
-
Dear X-Air and WeatherCat troubleshooters,
I definitely don't understand what you are observing there. I guess we'll have to wait for Stu to take a look and see if he can figure this one out.
Edouard
-
Have also discovered that files in the "Set HTML Source" path were what was causing me to see 'plain FTP' entries in Console. I mentioned removing a file from that path above. Since that time (~1:30pm), there have been only "Aux. FTP" entries in Console. Absolutely all the 'plain FTP' entries are before that time.
Apparently, WC uses the 'plain FTP' method with files in that path that are not specified in the "Additional Files" list. Any files in that list use the "Aux. FTP" method. At least, that's what I'm seeing here. If that's the case, at least I now have cut way back on the uploading needs. ;D