Trixology

WeatherCat => WeatherCat General Discussion => Topic started by: xairbusdriver on October 11, 2015, 10:38:30 PM

Title: FTP vs Aux. FTP - Console logs
Post by: xairbusdriver on October 11, 2015, 10:38:30 PM
I've been watching Console for various reasons lately. Nothing much on TV and the grass has about stopped growing. [cheer] Don't see much of anything here or in the Manual about these two things. What is the difference between them? Answering that will probably explain why both are needed and why they (at least the "Aux." version) exist and possibly how to eliminate the latter since it seems to be doing the same thing as the 'main' version. [interesting]to me, anyway...

Thanks! I gotta go watch the electric meter spin... [lol]
Title: Re: FTP vs Aux. FTP - Console logs
Post by: Blicj11 on October 12, 2015, 07:32:06 AM
Aux FTP identifies those files that need to be uploaded more frequently than the main FTP upload frequency, or are uploaded from a different directory than the Main FTP uploads. In either case, they are identified in the Custom Web Preferences as Additional Files.
Title: Re: FTP vs Aux. FTP - Console logs
Post by: xairbusdriver on October 12, 2015, 02:08:23 PM
Thanks, Blick!

Hhmm... That will require additional investigation. I'll double-check later, but I think they files are the same in both uploads. That seems a waste of time and micro-joules. I'm not understanding (among hundreds of other things!) where or why WC uploads files. There are only six actual "Additional Files", 15 gauges, 10 graphs, and one movie. Not seeing any problems, just like to know what, when, where, why, and whow (SP?)! [biggrin]
Title: What d'ya' mean stopped growin' !!(Was: FTP vs Aux. FTP - Console logs)
Post by: elagache on October 12, 2015, 09:21:22 PM
Dear X-Air, Blick, and WeatherCat yard and TV watchers, . . .  ;D

Quote from: xairbusdriver on October 11, 2015, 10:38:30 PM
I've been watching Console for various reasons lately. Nothing much on TV and the grass has about stopped growing.

 [wink] . . . What do ya' mean da' grass stopped growin'! ? ?? :o  Our grass just started! . . . .  [biggrin]

Of course that might have something to do with the ground fault interrupter tripping and taking down both the sprinkler control and the power for the weather station.  I found this out da' hard way. When I plugged in my super-deluxe PC case fan - nothing happened!  (http://www.canebas.org/WeatherCat/Forum_support_documents/Custom_emoticons/eek2.gif)  It too me a few minutes of troubleshooting before I realized what happened. (http://www.canebas.org/WeatherCat/Forum_support_documents/Custom_emoticons/thinking_idea.gif)

As to television, your problem is that you didn't buy a TiVo (https://www.tivo.com/) years ago and stocked it with all sorts of weird and wonderful goodies.  We are going back to the first year of The curse of Oak Island (http://www.history.com/shows/the-curse-of-oak-island).  It is always fun to watch suckers spend their money and the weirdest part of this story is that somehow it has to be true and very strange!  :o

Quote from: xairbusdriver on October 11, 2015, 10:38:30 PM
Thanks! I gotta go watch the electric meter spin... [lol]

Did you know you can get devices that attach to your electric meter so that you can watch it spin from the comfort of your own computer? . . .  [rolleyes2]

But get the TiVo first, much more entertainment value! (http://www.canebas.org/WeatherCat/Forum_support_documents/Custom_emoticons/tune_in_TV_emoticon.gif)

Cheers, Edouard  [cheers1]
Title: Re: FTP vs Aux. FTP - Console logs
Post by: xairbusdriver on October 12, 2015, 11:12:06 PM
No TiVo required here. PBS still has new programs that are worth watching, repeats are also available with a little searching. I'm sure there are worthwhile programs on commercial TV, just not something I'm interested in. Then there are these forums... [cheer]
Title: As exciting as . . . . (Was: FTP vs Aux. FTP - Console logs)
Post by: elagache on October 13, 2015, 10:13:43 PM
Dear X-Air and WeatherCat connoisseurs of fine entertainment . . .

Quote from: xairbusdriver on October 12, 2015, 11:12:06 PM
I'm sure there are worthwhile programs on commercial TV, just not something I'm interested in. Then there are these forums... [cheer]

 [wink] . . . . Well now, if you are looking for some real entertainment, maybe you should water your lawn!

You know what they say about sailboat racing . . . . .  it is about as exacting as watching the grass grow! . . .  [lol2]

Cheers, Edouard  [cheers1]
Title: Re: FTP vs Aux. FTP - Console logs
Post by: xairbusdriver on October 13, 2015, 11:21:27 PM
Not to discourage your "entertainment" activities...

I've been watching Console all afternoon, specifically the WeatherCat logs. Since ~1:30pm when I moved a file out of the "Set HTML Source" path. I posted details about it in the '<WC workload (http://athena.trixology.com/index.php?topic=1817.msg16790#msg16790)>' thread. Apparently, WC uses the 'plain FTP' method with files in that path that are not specified in the "Additional Files" list. Any files in that list use the "Aux. FTP" method. At least, that's what I'm seeing here. If that's the case, at least I now have cut way back on the uploading needs. [tup]
Title: Re: FTP vs Aux. FTP - Console logs
Post by: xairbusdriver on October 18, 2015, 12:02:03 AM
Quote
Aux FTP identifies those files that need to be uploaded more frequently than the main FTP upload frequency, or are uploaded from a different directory than the Main FTP uploads. In either case, they are identified in the Custom Web Preferences as Additional Files.
OK, I've now earned a Boy Scout Merit Badge in Console Monitoring! [cheer] Unfortunately, my membership expired several decades ago...

I was still perplexed/confused about why WC was basically using "Aux. FTP" and plain "FTP" both for practically the whole site! When I finally re-read Edouard's post, the "different directory" part reminded me that I have intentionally moved everything out of the "Set HTML Source" path in the prefs, except for four html and two php files. Any entry/text/tag used in any of the them is unique to each file, none of those entries/texts/tags are used in any other of the six files. So, why the doubles up FTP reports?! Within 35 seconds I saw ~180MB uploads with "Aux. FTP" and then plain "FTP". Numbers were not exactly the same but ? 1MB!

Here is a snapshot of the files structure on the drive. Please tell me how/why WC is uploading the CUSTOMstuff$ so often. The "CustomWebUploads.plist" has exactly the same six files (as mentioned above) mentioned above and listed in the "Additional Files" prefs.

(http://i1327.photobucket.com/albums/u666/xAirbusDriver/Local-File-Structure_zpsycc497o7.jpg)
Title: Re: FTP vs Aux. FTP - Console logs
Post by: WCDev on October 18, 2015, 11:26:47 AM
180MB sounds like a video being uploaded?
Title: Re: FTP vs Aux. FTP - Console logs
Post by: xairbusdriver on October 18, 2015, 09:41:50 PM
Quote
180MB sounds like a video being uploaded?
I'm not sure. The movie is ~30MB, but the 15 gauges average ~49KB for a total of ~746KB. The 10 graphs  total of~592KB. But they are being uploaded twice. I can see an entry just after each hour when the actual movie is being uploaded.

Here's the console entries just before the last movie upload, but I don't see a figure corresponding to the total bytes that would include all the gauges, graphs and the movie. [rolleyes2]
Code: [Select]
?10/18/15 3:00:07.167 PM WeatherCat[5376]: DailyMovieController: Renamed /Users/xairbusguy/Library/Application Support/WeatherCatMedia/Movies/Today/WeatherCatDailyMovie.mov to /Users/jameschaffin/Library/Application Support/WeatherCatMedia/Movies/Today/WeatherCatDailyMovie_14_00.mov
 10/18/15 3:00:08.748 PM WeatherCat[5376]: CWOP: Starting connection...
 10/18/15 3:00:08.769 PM WeatherCat[5376]: CWOP: Trying to connect to cwop.aprs.net.
 10/18/15 3:00:08.854 PM WeatherCat[5376]: CWOP: Connecting to cwop.aprs.net.
 10/18/15 3:00:12.857 PM WeatherCat[5376]: CWOP: Sending data to cwop.aprs.net. Success.
 10/18/15 3:00:15.862 PM WeatherCat[5376]: CWOP: Finished.
 10/18/15 3:00:15.883 PM WeatherCat[5376]: CWOP: Next transmission in 10 minutes and 23 seconds.
 10/18/15 3:00:23.538 PM WeatherCat[5376]: CustomWeb: Primed; waiting for weather sync.
 10/18/15 3:00:29.934 PM WeatherCat[5376]: Fetching data from weather station...
 10/18/15 3:00:30.571 PM WeatherCat[5376]: Temperature: 66?F  Dewpoint: 32?F  Windchill: 66?F  Pressure: 30 inHg  Av. Windspeed: 3 MPH from 52.6?  Current Conditions: Clear
 10/18/15 3:00:32.949 PM WeatherCat[5376]: CustomWeb: Processing custom web pages...
 10/18/15 3:00:36.267 PM WeatherCat[5376]: CustomWeb: Generating gauges.
 10/18/15 3:00:40.171 PM WeatherCat[5376]: CustomWeb: Finished parsing.
 10/18/15 3:00:40.193 PM WeatherCat[5376]: CustomWeb: Generating web cam picture.
 10/18/15 3:00:40.216 PM WeatherCat[5376]: CustomWeb: Waiting for web cam picture to stabilise.
 10/18/15 3:00:40.237 PM WeatherCat[5376]: FTP: FTP upload starting.
 10/18/15 3:00:40.257 PM WeatherCat[5376]: FTP: Uploading index.html. (31 files left to upload.)
 10/18/15 3:00:40.278 PM WeatherCat[5376]: FTP: Uploading statistics.html. (30 files left to upload.)
 10/18/15 3:00:40.300 PM WeatherCat[5376]: FTP: Uploading customgauge1.png. (29 files left to upload.)
 10/18/15 3:00:40.325 PM WeatherCat[5376]: FTP: Uploading customgauge2.png. (28 files left to upload.)
 10/18/15 3:00:40.346 PM WeatherCat[5376]: FTP: Uploading customgauge3.png. (27 files left to upload.)
 10/18/15 3:00:40.368 PM WeatherCat[5376]: FTP: Uploading customgauge4.png. (26 files left to upload.)
 10/18/15 3:00:40.389 PM WeatherCat[5376]: FTP: Uploading customgauge5.png. (25 files left to upload.)
 10/18/15 3:00:40.410 PM WeatherCat[5376]: FTP: Uploading customgauge6.png. (24 files left to upload.)
 10/18/15 3:00:40.431 PM WeatherCat[5376]: FTP: Uploading customgauge7.png. (23 files left to upload.)
 10/18/15 3:00:40.452 PM WeatherCat[5376]: FTP: Uploading customgauge8.png. (22 files left to upload.)
 10/18/15 3:00:40.477 PM WeatherCat[5376]: FTP: Uploading customgauge9.png. (21 files left to upload.)
 10/18/15 3:00:40.499 PM WeatherCat[5376]: FTP: Uploading customgauge10.png. (20 files left to upload.)
 10/18/15 3:00:40.522 PM WeatherCat[5376]: FTP: Uploading customgauge11.png. (19 files left to upload.)
 10/18/15 3:00:40.547 PM WeatherCat[5376]: FTP: Uploading customgauge12.png. (18 files left to upload.)
 10/18/15 3:00:40.569 PM WeatherCat[5376]: FTP: Uploading customgauge13.png. (17 files left to upload.)
 10/18/15 3:00:40.592 PM WeatherCat[5376]: FTP: Uploading customgauge14.png. (16 files left to upload.)
 10/18/15 3:00:40.618 PM WeatherCat[5376]: FTP: Uploading customgauge15.png. (15 files left to upload.)
 10/18/15 3:00:40.640 PM WeatherCat[5376]: FTP: Uploading upload-gauges.html. (14 files left to upload.)
 10/18/15 3:00:40.666 PM WeatherCat[5376]: FTP: Uploading customgraph1.jpg. (13 files left to upload.)
 10/18/15 3:00:40.688 PM WeatherCat[5376]: FTP: Uploading customgraph2.jpg. (12 files left to upload.)
 10/18/15 3:00:45.781 PM WeatherCat[5376]: FTP: Uploading customgraph3.jpg. (11 files left to upload.)
 10/18/15 3:00:45.803 PM WeatherCat[5376]: FTP: Uploading customgraph4.jpg. (10 files left to upload.)
 10/18/15 3:00:45.826 PM WeatherCat[5376]: FTP: Uploading customgraph5.jpg. (9 files left to upload.)
 10/18/15 3:00:45.849 PM WeatherCat[5376]: FTP: Uploading customgraph6.jpg. (8 files left to upload.)
 10/18/15 3:00:45.872 PM WeatherCat[5376]: FTP: Uploading customgraph7.jpg. (7 files left to upload.)
 10/18/15 3:00:45.897 PM WeatherCat[5376]: FTP: Uploading customgraph8.jpg. (6 files left to upload.)
 10/18/15 3:00:45.922 PM WeatherCat[5376]: FTP: Uploading customgraph9.jpg. (5 files left to upload.)
 10/18/15 3:00:45.947 PM WeatherCat[5376]: FTP: Uploading customgraph10.jpg. (4 files left to upload.)
 10/18/15 3:00:45.971 PM WeatherCat[5376]: FTP: Uploading upload-graphs.html. (3 files left to upload.)
?10/18/15 3:00:45.998 PM WeatherCat[5376]: FTP: Uploading WeatherCatDailyMovie.mov. (2 files left to upload.)
 10/18/15 3:00:46.023 PM WeatherCat[5376]: FTP: Uploading upload-movies.html. (1 files left to upload.)
 10/18/15 3:00:46.046 PM WeatherCat[5376]: FTP: Uploading WX-date-time-String.php.
 10/18/15 3:01:00.301 PM WeatherCat[5376]: CustomWeb: Getting web cam image files.
 10/18/15 3:01:00.372 PM WeatherCat[5376]: CustomWeb: Finished processing custom web pages.
 10/18/15 3:01:00.660 PM WeatherCat[5376]: WeatherBug:  dateutc=2015-10-18%2020%3A01%3A00&winddir=61&windspeedmph=1&windgustmph=11.0&humidity=29&tempf=65.8&rainin=0.00&dailyrainin=0.000&baromin=30.417&monthlyrainin=0.04&Yearlyrainin=23.95
 10/18/15 3:01:05.169 PM WeatherCat[5376]: CustomWeb: Processing additional custom web file: /Applications/WeatherCat 2.2/Web Templates/Mid-South/index.html
 10/18/15 3:01:08.449 PM WeatherCat[5376]: CustomWeb: Finished parsing.
 10/18/15 3:01:08.473 PM WeatherCat[5376]: CustomWeb: Finished processing custom web pages.
 10/18/15 3:01:08.671 PM WeatherCat[5376]: CustomWeb: Processing additional custom web file:   /Applications/WeatherCat 2.2/Web Templates/Mid-South/WX-date-time-String.php
 10/18/15 3:01:08.902 PM WeatherCat[5376]: Aux. FTP: FTP upload starting.
 10/18/15 3:01:08.929 PM WeatherCat[5376]: Aux. FTP: Uploading index.html.
?10/18/15 3:01:11.648 PM WeatherCat[5376]: Aux. FTP: Finished. Uploaded 5224 bytes in 3 seconds at 1394 bytes per second.

The "upload-movies.html" file is nothing more than this:
Code: [Select]
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8" />
    <meta name="generator" content="BBEdit 10.1.1" />
    <title>Upload WeatherCatDailyMovie.mov and the 640px backyard image</title>
</head>
<body>
MOVIE$
WCAM640$
</body>
</html>
According to a BBEdit Multi-file search, of the entire "WeatherCat 2.2" directory (which is displayed in post #7, the tag appears exactly twice: In the "upload-movies.html" file and in Documentation/WeatherCat_all_tags/index.html. That latter file is not being uploaded and certainly not the entire catalog of tags listed in that file. ;)

From the com.trixology.weathercat.plist:
Code: [Select]
<key>CustomWebPageHTMLPath</key>
    <string>/Applications/WeatherCat 2.2/Web Templates/Mid-South</string>That seems correct to me.

And the list of six files as seen in the "CustomWebUploads.plist" is correct. The entry for the "upload-movies.html also looks correct (31 minutes, not 2):
Code: [Select]
<dict>
    <key>FTP</key>
    <true/>
    <key>enabled</key>
    <true/>
    <key>path</key>
    <string>/Applications/WeatherCat 2.2/Web Templates/Mid-South/upload-movies.html</string>
?   <key>periodMins</key>
?   <real>31</real>
</dict>
Thia post is getting too long and complicated.  [blush] Perhaps I should simply Shut down the mini and see if things work more as I think they should?
Title: Re: FTP vs Aux. FTP - Console logs
Post by: xairbusdriver on October 22, 2015, 12:00:35 AM
Anyone interested in seeing some Console logs? ;)
Am I the only one seeing this 'double-uploading'? :o
I am concerned that I am wasting bandwidth with this behavior which I simply do not understand. Nor can I find any info about why it is occurring. [banghead]

Thanks!
Title: Re: FTP vs Aux. FTP - Console logs
Post by: awilltx on October 22, 2015, 02:44:51 PM
Hello xair,


Would you post screenshots of your WC pref Custom Web "Main" and "Additional files" panes and then zip up those files so I could take a look.


Thanks,


Alan
Title: Re: FTP vs Aux. FTP - Console logs
Post by: xairbusdriver on October 22, 2015, 03:31:42 PM
The "Additional Files" list:
(http://i1327.photobucket.com/albums/u666/xAirbusDriver/Custom20Web20Files_zps5rl4fakt.jpg)
The files in the actual HTML Source path:The HTML Source path info:
(http://i1327.photobucket.com/albums/u666/xAirbusDriver/Custom%20Web%20Main_zpsmsqxhvvx.jpg)

Hope this helps. I note that the correct html & php files are in the WeatherCatCustomWeb directory. Also the newly created custom gauges and graphs. It appears that WC also creates the basic images for things such as "all-dials", "all_dialsc", "barometer", etc., but they are not being uploaded, which is correct, I think.
Title: Re: FTP vs Aux. FTP - Console logs
Post by: xairbusdriver on October 22, 2015, 03:47:55 PM
Here are the three html files that "request" the custom gauges, graphs, and the QuickTime movie. Perhaps WC is 'confused' by these files? I can only assume that something I am doing or the way I have things set up is causing this behavior. [banghead]
Title: Re: FTP vs Aux. FTP - Console logs
Post by: awilltx on October 22, 2015, 04:11:27 PM
Thanks for the info. Based on your "main" screenshot, you are processing ALL of the files in your Mid-South directory every two minutes, then your additional files (the same files!) are being processed every two minutes (except for your movie file, which is set to every 51 minutes). Basically, you are uploading your (big, I presume) movie file every two minutes as well as every 51 minutes. Additionally, you are double uploading all of the others at two minute intervals.


To rectify:


1. Remove all of your "additional files"


2. Move your upload-movies.html to another folder (other than "Mid-South") then add this one file to the "additional files" pane at 51 minutes.


Hope this helps!


Alan
Title: Re: FTP vs Aux. FTP - Console logs
Post by: Blicj11 on October 22, 2015, 04:29:34 PM
Alan is correct, your Main processes all of of your files in the directory. You cannot use the Aux file setup to "slow down" anything that is in the Main.  Use Aux only for the purpose of processing those files than need to be uploaded more frequently than the setting in the Main and for those files you are uploading from a separate directory than the Main.
Title: Re: FTP vs Aux. FTP - Console logs
Post by: xairbusdriver on October 22, 2015, 04:35:41 PM
Thanks! That makes sense! I'm sure I was confusing everyone with my stupidity! (That happens quite often!) I thought the "Additional Files" were supposed to be in the HTML Source path! [banghead] I like to keep things as complicated as possible... [blush] [rolleyes2]

The "2 minute Upload Web Pages Every:" setting in the "Main" Custom Web prefs takes care of any files in the "HTML Source" directory. Period. "Additional Files" are only for Additional Files (not in the source path)! What a concept! [cheer]

The HTML Source path remains the same, but I moved the movie upload file out of there. Deleted all the "Additional Files" except the movie upload file and changed the path to it.

Thanks again, Alan and Blick! [tup]
Title: Re: FTP vs Aux. FTP - Console logs
Post by: xairbusdriver on October 22, 2015, 04:40:01 PM
Quote
You cannot use the Aux file setup to "slow down" anything that is in the Main.  Use Aux only for the purpose of processing those files than need to be uploaded more frequently than the setting in the Main and for those files you are uploading from a separate directory than the Main.
The key word there is "use". I was not understanding how anything was "using" Aux, much less why. I was not understanding that I had any control/influence in causing/using the "Aux." method. That was my original request for what the difference is and what it was for. Sorry I didn't word that request properly. I now "see the light"! 8)
Title: Re: FTP vs Aux. FTP - Console logs
Post by: awilltx on October 22, 2015, 09:02:40 PM
I hope you enjoy your newly found bandwidth!


Alan
Title: Re: FTP vs Aux. FTP - Console logs
Post by: WCDev on October 25, 2015, 04:23:14 PM
Yes, generally you want everything processed by the normal custom web processing - only use the additional files for time sensitive stuff - current data/daily high/lows for gauges for example or parts of your site you want updated more frequently than the normal uploads.

Additional files are FTP'd via the aux FTP queue (which is an additional FTP path to your site running in parallel to the main queue), so aux. uploads won't be held up by any large uploads going up from the main custom web processing (a movie upload for example can take some time).
Title: Re: FTP vs Aux. FTP - Console logs
Post by: xairbusdriver on October 26, 2015, 12:18:05 AM
Quote
...only use the additional files for time sensitive stuff - current data/daily high/lows for gauges for example or parts of your site you want updated more frequently than the normal uploads.
OK, it seems I'm still learning! Although I knew/know that the movie is not uploaded until it is ready, it still left it in the "Additional Files" list unnecessarily. I wonder, however, if that list could also be used for pages that need updating on less than the normal (2 minutes, in my case) time frame. That was my initial reason for leaving the movie file in the "List". I now see that this is pointless, in that case, of course. [banghead]
Title: Re: FTP vs Aux. FTP - Console logs
Post by: xairbusdriver on October 27, 2015, 02:17:34 PM
I think I have the Aux. FTP usage down to the minimum... almost. Seems to be used only twice an hour. I think I can get that down to zero, however.

I have searched the Manual, without success, to find an explanation of when WC uploads the DailyMovie. Searching the forums discloses that WC is 'smart' enough to only upload the movies just after it is created, once an hour. I think have confirmed my theory that I do not need to have the MOVIE$ tag in the Additional Files list. WC will not [banghead] upload the large DailyMovie file even if that tag is in a file that gets processed every two minutes (my 'normal' data update interval). The uploading process will be rather busy just after the new hour when every single tag will be getting processed and all the WC generated images get created and uploaded. Thankfully, 'threading' (and Stu's knowledge!) takes care of this massive input.

I would appreciate being pointed to a page in the Manual where this 'once-an-hour' rule is discussed/explained. I'm probably using the wrong search terms(s) to find it. I have found several posts by him that make the 'once-an-hour' statement, however.
I shall now remove the "upload-movie.txt" file from the Additional Files list. I'll add the MOVIE$ tag to one of the few files that actually need updating every few minutes. Lastly, I will stop 'bugging' everyone about my slowness in understanding the basics of this great app! (I had to be careful not to make a Freudian slip and call myself a "great ape"!). [blush]
Title: Re: FTP vs Aux. FTP - Console logs
Post by: WCDev on October 31, 2015, 04:31:21 PM
Just checked it, it isn't in the manual. But it's an implementation detail - it would be more accurate to say that the movie will be uploaded the next time the MOVIE$ tag is processed once the movie is complete; currently they are hourly and finish at the top of the hour, so it'll be uploaded on the first custom web run after the hour.
Title: Re: FTP vs Aux. FTP - Console logs
Post by: xairbusdriver on October 31, 2015, 09:00:53 PM
Quote
more accurate to say that the movie will be uploaded the next time the MOVIE$ tag is processed once the movie is complete
Yeah. But it's starting to sound like legalese! :P Maybe when you have time to edit the Manual. I'm sure that is one of your favorite tasks! [lol2] Obviously, everyone else understood the process, so I'd say just forget the 'edit'. I've confirmed the 'intelligence' of WC, with your expert tutelage; it operates exactly as you say. [cheer] I just stuck the MOVIE$ tag in a file that gets processed every two minutes, but the movie actually only uploads once an hour. Have had no "Aux. FTP" traffic after making that change. [removed link to the now castly photobucket.com site]

BTW, have had no more "FATAL ERROR" messages from Apple since correcting my Custom Graph setup erer some weeks ago. [blush]