Author Topic: Something do deal with FTP slowdowns?  (Read 2654 times)

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Something do deal with FTP slowdowns?
« on: March 01, 2012, 10:45:07 PM »
Dear WeatherCat fans,

This morning my web hosting provider had "issuez" and WeatherCat started to be unable to upload all the files associated with my weather website in the 15 minute interval between updates.  WeatherCat is amazingly robust.  It just kept adding files to the queue and when I got back from the morning shopping errands there was quite a queue.

This problem seems quite rare in general and perhaps I'll have to change provider (shucks, I thought GoDaddy was better than this! )  However, the Internet seems to be getting more and more congested, so I thought I should toss the problem out as perhaps a "put on the back burner" issue.

One possible thing that could be done is to have WeatherCat check if it has successfully uploaded all the files from the last custom web generation before starting the next.  I don't know if its easy enough to  implement to be worth the effort.

Another possibility would be to add some AppleScripting capability.  So that an AppleScript could to stop the custom web generation.  Then AppleScript could be used as a test bed to explore different ways to deal with the backlog.

I had another idea that may or may not be a good.  I ultimately turned off custom web generation until WeatherCat managed to empty the queue.  Having to go back to the preferences to do that seemed a little awkward.  Would folks like to have an "on-off" button next to each service provided by WeatherCat?  That way if you see a problem with any service, you can quickly disable it?

Obviously, no clear ideas here.  I thought I would toss out my experiences for everyone to ponder a bit.  Perhaps this won't be a common problem so it is a non-issue.  On the other hand, if Internet congestion does get worse, all of us will have had a chance to "stew the matter a bit."

Comments from the WeatherCat faithful most welcome!!

Cheers, Edouard  [cheers1]

P.S. Obviously one thing I should try to do is try to reduce the size of what I am uploading.  I would like to see what Server-Side Includes could do to only upload what has actually changed.  However, I'm behind on the AppleScripts, so don't know when I could take that project on.

WCDev

  • WeatherCat Developer
  • Administrator
  • Storm
  • *****
  • Posts: 2911
    • CW9739
    • ISCOTLAN25
    • Trixology
  • Station Details: Main Station: Vantage Pro-2, 24hr fars, solar, soil/leaf station, extra temp stations, no U.V. WeatherLink IP.
Re: Something do deal with FTP slowdowns?
« Reply #1 on: March 01, 2012, 11:04:37 PM »
If things get too bad Edouard, WeatherCat will dump the whole queue (the Status window will indicate this), then it'll start again, dumping as necessary until things improve..

Cheers,
Stu.

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
I'm drowning!! (Re: FTP slowdowns?)
« Reply #2 on: March 02, 2012, 03:11:13 AM »
Howdy Stu and Weather web site administrators . . . .

If things get too bad Edouard, WeatherCat will dump the whole queue (the Status window will indicate this), then it'll start again, dumping as necessary until things improve..

 ;) Oh, I'm so glad WeatherCat will stop and wait until things improve . . . .  Cuz, there are weather graphs, and html pages, and AWEKAS uploads, and webcam images . . . . desperately flooding into my room from the overflow . . . . . . I'm drowning!!!!!  :-\

Good thing I'm a certified scuba diver!!


Uh seriously, not sure what GoDaddy is doing, but my FTP service is nearly non-existent.  My WeatherCat log is filled with errors like this:

Code: [Select]
6:53:24 PM 01-Mar-2012:  FTP1: curl: (7) couldn't connect to host
curl: (7) couldn't connect to host

 6:53:24 PM 01-Mar-2012:  FTP1: FTP ERROR (7) - Couldn't connect to host. Server down?
 6:53:24 PM 01-Mar-2012:  FTP1: FTP ERROR 911: Could not upload file://localhost/Users/elagache/WeatherCat%20Web%20Sites/WC%20Custom%20Web/WeatherCatBanner.jpg - Aborting upload.
 6:53:25 PM 01-Mar-2012:  FTP1: Finished. Uploaded 128767 bytes in 380 seconds at 338 bytes per second.
 6:53:53 PM 01-Mar-2012:  FTP2: curl: (7) couldn't connect to host
curl: (7) couldn't connect to host

 6:53:53 PM 01-Mar-2012:  FTP2: FTP ERROR (7) - Couldn't connect to host. Server down?
 6:53:53 PM 01-Mar-2012:  FTP2: FTP ERROR 910: Failed to start upload file://localhost/Users/elagache/WeatherCat%20Web%20Sites/WC%20Custom%20Web/all_dials.jpg - retrying in 1 second.
 6:53:54 PM 01-Mar-2012:  FTP2: Uploading all_dials.jpg & windspeed.jpg. (21 files left to upload.)
 6:54:06 PM 01-Mar-2012: 
 6:54:06 PM 01-Mar-2012:  Fetching data from weather station...
 6:54:06 PM 01-Mar-2012:  Temperature: 46?F  Dewpoint: 43?F  Windchill: 46?F Pressure: 30 inHg  Av. Windspeed: 0 MPH from WNW. Current Conditions: N/A
 6:54:58 PM 01-Mar-2012:  CustomWeb: Processing additional custom web file: /Users/elagache/WeatherCat Web Sites/WC Custom Upload Templates/awekas_wl.html
 6:54:59 PM 01-Mar-2012:  CWOP: Starting connection...
 6:54:59 PM 01-Mar-2012:  CWOP: Trying to connect to cwop.aprs.net.
 6:54:59 PM 01-Mar-2012:  CWOP: Connecting to cwop.aprs.net.
 6:55:03 PM 01-Mar-2012:  CWOP: Sending data to cwop.aprs.net. Success.
 6:55:05 PM 01-Mar-2012:  CustomWeb: Finished parsing.
 6:55:05 PM 01-Mar-2012:  CustomWeb: Finished processing custom web pages.
 6:55:06 PM 01-Mar-2012:  CWOP: Finished.
 6:55:06 PM 01-Mar-2012:  CWOP: Next transmission in 9 minutes and 43 seconds.
 6:55:06 PM 01-Mar-2012:  FTP1: FTP upload starting.
 6:55:06 PM 01-Mar-2012:  FTP1: Uploading awekas_wl.html.
 6:55:11 PM 01-Mar-2012:  FTP1: Finished. Uploaded 331 bytes in 6 seconds at 54 bytes per second.
 6:55:13 PM 01-Mar-2012:  FTP2: Uploading cloudbase.jpg & tempout.jpg. (19 files left to upload.)
 6:55:19 PM 01-Mar-2012:  FTP2: Uploading windchill.jpg & humout.jpg. (17 files left to upload.)
 6:55:25 PM 01-Mar-2012:  FTP2: Uploading humin.jpg & rain24h.jpg. (15 files left to upload.)
 6:55:29 PM 01-Mar-2012: 
 6:55:29 PM 01-Mar-2012:  Fetching data from weather station...
 6:55:29 PM 01-Mar-2012:  Temperature: 46?F  Dewpoint: 43?F  Windchill: 46?F Pressure: 30 inHg  Av. Windspeed: 0 MPH from NW by W. Current Conditions: N/A
 6:55:31 PM 01-Mar-2012:  FTP2: Uploading windspeed1.jpg & temperature1.jpg. (13 files left to upload.)
 6:55:43 PM 01-Mar-2012:  FTP2: Uploading windchill1.jpg & cloudbase1.jpg. (11 files left to upload.)
 6:55:50 PM 01-Mar-2012:  FTP2: Uploading uv1.jpg & tempin1.jpg. (9 files left to upload.)
 6:55:56 PM 01-Mar-2012:  FTP2: Uploading precipitation1.jpg & customgraph1.jpg. (7 files left to upload.)
 6:56:08 PM 01-Mar-2012:  FTP2: Uploading customgraph3.jpg & customgraph5.jpg. (5 files left to upload.)
 6:56:20 PM 01-Mar-2012:  FTP2: Uploading customgraph7.jpg & WeatherCat_all_tags.html. (3 files left to upload.)
 6:56:28 PM 01-Mar-2012:  FTP2: Uploading webcam320.jpg. (1 file left to upload.)
 6:56:38 PM 01-Mar-2012:  FTP2: Uploading image.jpg.
 6:56:55 PM 01-Mar-2012:  FTP2: Finished. Uploaded 1078723 bytes in 332 seconds at 3240 bytes per second.
 6:57:26 PM 01-Mar-2012:  FTP2: FTP upload starting.
 6:57:26 PM 01-Mar-2012:  FTP2: Uploading WeatherCatBanner.jpg.
 6:57:31 PM 01-Mar-2012:  FTP2: Finished. Uploaded 32083 bytes in 6 seconds at 5031 bytes per second.
 6:57:58 PM 01-Mar-2012:  Wunderground: Generating image for Wunderground.
 6:57:58 PM 01-Mar-2012:  Wunderground: Waiting for web cam picture to stabilise.
 6:58:04 PM 01-Mar-2012: 
 6:58:04 PM 01-Mar-2012:  Fetching data from weather station...
 6:58:04 PM 01-Mar-2012:  Temperature: 46?F  Dewpoint: 42?F  Windchill: 46?F Pressure: 30 inHg  Av. Windspeed: 0 MPH from NW. Current Conditions: N/A
 6:58:08 PM 01-Mar-2012:  Wunderground: Getting web cam picture file.
 6:58:08 PM 01-Mar-2012:  Wunderground: Wunderground image ready for FTP - added to FTP queue.
 6:58:09 PM 01-Mar-2012:  FTP1: FTP upload starting.
 6:58:09 PM 01-Mar-2012:  FTP1: Uploading image.jpg.
 6:58:14 PM 01-Mar-2012:  FTP1: Finished. Uploaded 48073 bytes in 6 seconds at 7980 bytes per second.
 6:59:06 PM 01-Mar-2012: 
 6:59:06 PM 01-Mar-2012:  Fetching data from weather station...
 6:59:06 PM 01-Mar-2012:  Temperature: 46?F  Dewpoint: 42?F  Windchill: 46?F Pressure: 30 inHg  Av. Windspeed: 0 MPH from NW. Current Conditions: N/A
 6:59:46 PM 01-Mar-2012:  CustomWeb: Processing additional custom web file: /Users/elagache/WeatherCat Web Sites/WC Custom Upload Templates/awekas_wl.html
 6:59:53 PM 01-Mar-2012:  CustomWeb: Finished parsing.
 6:59:53 PM 01-Mar-2012:  CustomWeb: Finished processing custom web pages.
 6:59:53 PM 01-Mar-2012:  FTP2: FTP upload starting.
 6:59:54 PM 01-Mar-2012:  FTP2: Uploading awekas_wl.html.
 6:59:58 PM 01-Mar-2012:  FTP2: Finished. Uploaded 332 bytes in 5 seconds at 58 bytes per second.

So the problem in my case is that some upload speeds as fast as reasonable: ~5000 bytes per second - to as bad as . . . . ~ 50 bytes per second.  Stu, what would you recommend WeatherCat users do in a muddy situation like this?  Should I just leave my custom web settings where they are and let da' Cat cope as best it can with the poor performance?

Sorry to ask, but I hate to see a cat  [cat] suffer!!

Cheers, Edouard  [cheers1]

P.S. Given that it really doesn't matter too much, I compromised and nudged the website update rate back once an hour.  Hopefully nothing too catastrophic can happen under those circumstances.