Author Topic: Troubles with WC3  (Read 342 times)

xairbusdriver

  • Storm
  • *****
  • Posts: 2042
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.13.4/WC 3.0
Troubles with WC3
« on: April 07, 2018, 09:53:15 PM »
Tracking down why WC3 keeps quitting, I find the "Status" window "Custom Web:" value confusing. It either shows ""Waiting For Weather Sync." or "Idle". The "Last Run:" value always shows "N/A". However, my web page seems to be updating as usual, most pages every 2 minutes. Station Comms indicate 100.00%, no errors. "Webcam: Running (89 frames)" What is the time frame for that? I'm requesting an image capture every 2 minutes, is "89" the number of 'frames' captured since WC was restarted?

I get don't see any change in the "Status" window, even if I use the Tools->Update Weather Now.

WC is quitting intermittently. Earlier today I got the following emails:
Quote
WeatherCat watchdog: WeatherCat appears to have hung; rebooting as a precautionary measure.
WeatherCat TimeStamp: 11:13:59 07-Apr-18
----------
WeatherCat: Wunderground driver appears to have hung. Restarting it. Result to follow...
WeatherCat TimeStamp: 12:44:40 07-Apr-18
----------
WeatherCat: Custom alerts thread appears to have hung. Restarting it. Result to follow...
WeatherCat TimeStamp: 12:44:42 07-Apr-18
----------
WeatherCat: Wunderground driver running again.
WeatherCat TimeStamp: 12:45:11 07-Apr-18
----------
WeatherCat: Custom alerts running again.
WeatherCat TimeStamp: 12:45:12 07-Apr-18
Console log just before those messages:
Quote
2018/04/07 10:54:05:951  Error in StatsManager:findTotalInDateRange - hit default
2018/04/07 10:54:06:042  Error in StatsManager:findTotalInDateRange - hit default
2018/04/07 11:01:10:915  Deleting old historical images
2018/04/07 11:01:10:929  Creating daily movie
2018/04/07 11:01:14:375  DEBUG: Create daily movie - File webcam_11_06_27.jpg not now day
2018/04/07 11:01:14:416  Finished writing daily timelapse movie...checking completion status...
2018/04/07 11:01:14:416  Video writing succeeded.
2018/04/07 11:01:14:416  Video writing done
2018/04/07 11:01:14:538  Creating hourly timelapse movie...
2018/04/07 11:01:15:053  Finished writing hourly timelapse movie...checking completion status...
2018/04/07 11:01:15:053  Video writing succeeded.
2018/04/07 11:01:15:053  Video writing done
2018/04/07 11:05:58:045  Error in StatsManager:findTotalInDateRange - hit default
2018/04/07 11:05:58:136  Error in StatsManager:findTotalInDateRange - hit default
2018/04/07 11:13:59:054  WeatherCat: watchdog - WeatherCat appears to have hung, trying to send admin email.
2018/04/07 11:14:01:106  WeatherCat: watchdog - sent admin email. Terminating WeatherCat.
2018/04/07 11:14:01:106  WeatherCat: *****watchdog - rebooting WeatherCat, it appears to have hung*****
2018/04/07 11:14:17:028 

Saving log to disk at /Users/jameschaffin/Library/Logs/WeatherCat
2018/04/07 11:14:17:028
I see entries for several days from "WCKicker":
Quote
2018-04-02 19:20:44,681 - WCKicker - INFO ---------------------------
2018-04-02 19:20:44,681 - WCKicker - INFO -WeatherCat kicker launched
2018-04-02 21:27:24,266 - WCKicker - INFO ---------------------------
2018-04-02 21:27:24,275 - WCKicker - INFO -WeatherCat kicker launched
2018-04-02 22:12:26,869 - WCKicker - INFO -reboot_weathercat - rebooting program WeatherCat on Tue, 03 Apr 2018 03:12:26 +0000
2018-04-02 22:59:07,101 - WCKicker - INFO ---------------------------
2018-04-02 22:59:07,101 - WCKicker - INFO -WeatherCat kicker launched
2018-04-03 14:51:11,397 - WCKicker - INFO ---------------------------
2018-04-03 14:51:11,407 - WCKicker - INFO -WeatherCat kicker launched
2018-04-03 14:53:29,309 - WCKicker - INFO ---------------------------
2018-04-03 14:53:29,309 - WCKicker - INFO -WeatherCat kicker launched
2018-04-03 15:26:33,164 - WCKicker - INFO ---------------------------
2018-04-03 15:26:33,164 - WCKicker - INFO -WeatherCat kicker launched
2018-04-06 11:53:56,235 - WCKicker - INFO ---------------------------
2018-04-06 11:53:56,245 - WCKicker - INFO -WeatherCat kicker launched
2018-04-06 11:55:53,575 - WCKicker - INFO ---------------------------
2018-04-06 11:55:53,575 - WCKicker - INFO -WeatherCat kicker launched
2018-04-06 20:56:00,999 - WCKicker - INFO -reboot_weathercat - rebooting program WeatherCat on Sat, 07 Apr 2018 01:56:00 +0000
2018-04-06 21:43:57,260 - WCKicker - INFO ---------------------------
2018-04-06 21:43:57,261 - WCKicker - INFO -WeatherCat kicker launched
2018-04-07 11:14:16,769 - WCKicker - INFO -reboot_weathercat - rebooting program WeatherCat on Sat, 07 Apr 2018 16:14:16 +0000
2018-04-07 12:43:04,720 - WCKicker - INFO ---------------------------
2018-04-07 12:43:04,722 - WCKicker - INFO -WeatherCat kicker launched
I am assuming that each manual Quit/Run cycle also generates one of these entries which would account for most of the ones on 4-2,3 & 6 (did not work on the app on 4-4 or 4-5)

Any other info I can grab/supply?
WC3 running on Mac mini (late 2017), OS 10.13.4, also running Amphetamine, have Prevent App Nap checked (app's Info), UN-checked Enable Power Nap(System Prefs).

xairbusdriver

  • Storm
  • *****
  • Posts: 2042
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.13.4/WC 3.0
Re: Troubles with WC3
« Reply #1 on: April 07, 2018, 10:11:42 PM »
To The Dev: Video uploading question
Do I need to place a file with the new webcam tags in the Additional Files list with a timing of anything other than "60.0" minutes? Does it really need to run more than once after WC starts?
EDITED 4/8/18 7:58pm CST It would seem logical (to me, anyway) that once WC 'sees' the tags, it would set a variable (bool?) that would cause it to ftp the video(s) as needed until it quits or crashes(!). Dear boy, the only way WC "sees" those tags is when it parses the files in the "Set HTML Source path or in the Additional Files list. [rolleyes2] As for having the file that requests uploading the video, once an hour should suffice, probably as soon after the new hour as possible, but after enough time to allow the 'WC1hrMovie' to finish uploading? I'm going to add that file to the Additional Files list at about 5 past the next hour hoping that it will run at approximately every hour + 5 minutes.

BTW, thanks, again, for using the newer format! And I think the day-long vid will be great, as long as my camera keeps working... [banghead]

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 4775
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, MacBook Pro (Early 2011)
Take a sample of the hung process? (Re: Troubles with WC3)
« Reply #2 on: April 07, 2018, 10:54:30 PM »
Dear X-Air and WeatherCat troubleshooters,

Tracking down why WC3 keeps quitting, I find the "Status" window "Custom Web:" value confusing.
. . . . .

I have no idea what is happening to you.  It is extremely strange that you are getting any errors from the Stats Manager.  I've forgotten, which webcam camera are you using?

To help Stu figure out what is going on, turn off the watchdog process temporarily and allow WeatherCat to get into this hung state.  Then get a sample of the process using these instructions:

http://athena.trixology.com/index.php?topic=1775.0

Email the sample to Trixology technical support:

https://trixology.com/contact-us/

Unfortunately, Stu's day job is being particularly taxing at the moment, so he might be a bit slow to get back you.  However, Stu should be able to figure out exactly what is going on from the sample of the WeatherCat process.

Sorry, very strange . . . .

Cheers, Edouard

xairbusdriver

  • Storm
  • *****
  • Posts: 2042
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.13.4/WC 3.0
Re: Troubles with WC3
« Reply #3 on: April 08, 2018, 01:11:38 AM »
WILL COmply.

I left the house for a couple of hours and found it 'dead' again. As s0on as I open File Sharing WC starts downloading the Historical Data. I may be seeing the mini refusing to run without a full-time monitor. I had a chip/dongle that was plugged into the old one that 'tricked' the computer into thinking there was a monitor attached. I am using the same dongle on the new one, but the newer OS may be smarter than I am! I'll do some searching to see if I can find any info on that. [rolleyes2] Frankly, that makes more sense that blaming WC. [banghead]
Later: My dongle (fit Headless) actually shows up in the System Prefs/Display as the attached monitor. As far as I can tell, it's not much more than a ~75ohm resister.

Once again, the errors are "Error in StatsManager:findTotalInDateRange - hit default". It seems to occur twice, within micro seconds, but with several minutes between attempts:
Quote
2018/04/07 17:04:39:287  Error in StatsManager:findTotalInDateRange - hit default
2018/04/07 17:04:39:381  Error in StatsManager:findTotalInDateRange - hit default
2018/04/07 17:16:31:328  Error in StatsManager:findTotalInDateRange - hit default
2018/04/07 17:16:31:420  Error in StatsManager:findTotalInDateRange - hit default
2018/04/07 17:28:23:380  Error in StatsManager:findTotalInDateRange - hit default
2018/04/07 17:28:23:472  Error in StatsManager:findTotalInDateRange - hit default
2018/04/07 17:40:15:484  Error in StatsManager:findTotalInDateRange - hit default
2018/04/07 17:40:15:575  Error in StatsManager:findTotalInDateRange - hit default
2018/04/07 17:52:08:131  Error in StatsManager:findTotalInDateRange - hit default
2018/04/07 17:52:08:223  Error in StatsManager:findTotalInDateRange - hit default
2018/04/07 17:58:05:343  WeatherCat: watchdog - WeatherCat appears to have hung, trying to send admin email.
2018/04/07 17:58:07:975  WeatherCat: watchdog - sent admin email. Terminating WeatherCat.
2018/04/07 17:58:24:764 

Saving log to disk at /Users/jameschaffin/Library/Logs/WeatherCat
2018/04/07 17:58:24:764 

----------LAUNCH------------

2018/04/07 18:28:49:907  WeatherCat V3.00  Build 2224. 19:53:40 Mar 15 2018 loading...
2018/04/07 18:28:49:909  ***Good afternoon and welcome to WeatherCat!***
2018/04/07 18:28:49:922  Launching WeatherCat kicker
I'll surmise that ----------LAUNCH------------ occurred when I first enabled File Sharing.

xairbusdriver

  • Storm
  • *****
  • Posts: 2042
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.13.4/WC 3.0
Re: Troubles with WC3
« Reply #4 on: April 08, 2018, 01:38:35 AM »
1. OK, I've looked through every WC Pref page at least twice, even the ones that don't pertain to anything I need; no sign of "Watchdog".
2. I've found 5 or 6 posts asking/saying folks have ENabled or DISabled Watchdog, but none with instructions for a simpleton to actually do either.
3. Found no hits for "watch", "watchdog" nor "dog" with the Search function here.
4. Also found no entry in The Manual searching for the same terms.
5. I even found some hits on the web but they only referred to posts I'd already looked at here. [rolleyes2]

I know I saw and enabled that function when setting up WC3. [banghead] Was it in the "Configure Station Comms..." where we are warned that doing that again will cause golf ball sized hail, if not brimstones to fall upon thee?

I'll just find some old tin roofing and give that a try... back soon, I hope! [rolleyes2] Nope, not there, either.

The absolute only place I can find the word "Watchdog" is in the Status window, which doesn't have any user capable controls/buttons/text fields that I can see.

Does anyone have access to the secret page that has a link to the secure site that holds a clue as to where the probably huge, brightly colored and lonely "Watchdog" switch is?!!! [rolleyes2] [banghead] :o

Alan Rowley

  • Strong Breeze
  • ***
  • Posts: 143
  • It is far better to wear out than to rust away.
    • D3696
    • INOTTING15
    • Selston Weather
  • Station Details: Weather station: Davis Vantage Pro2. Software: WeatherCat. Computer: Mac Mini, High Sierra OS
Re: Troubles with WC3
« Reply #5 on: April 08, 2018, 07:51:24 AM »
Isn't watchdog the box in Preferences / Units/Misc1 called 'Monitor & Re-launch WeatherCat'?

xairbusdriver

  • Storm
  • *****
  • Posts: 2042
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.13.4/WC 3.0
Re: Troubles with WC3
« Reply #6 on: April 08, 2018, 02:36:26 PM »
Yes, there is that label and a checkbox. Unfortunately, it nor it's tooltip use the word "watchdog" nor is that word in the Manual. I'll un-check it and see how it goes later today. Right now, I've simply turned off using the webcam and removed the file that uploads the MP4 files.

WC is not actually crashing. It is simply 'hanging'. Simply sharing the mini's screen causes it to "download historical data". The last three times, the dialog has used "512" as the number of 'data point(?) or records(?), no matter how long it had been 'hung'. That is suspicious in itself but also strange that is is a power of 2 which is basic to all binary computer systems. It may just indicate the maximum size of something in the data logger.

WCDev

  • WeatherCat Developer
  • Administrator
  • Storm
  • *****
  • Posts: 2772
    • CW9739
    • ISCOTLAN25
    • Trixology
  • Station Details: Main Station: Vantage Pro-2, 24hr fars, solar, soil/leaf station, extra temp stations, no U.V. WeatherLink IP.
Re: Troubles with WC3
« Reply #7 on: April 08, 2018, 06:47:18 PM »
From your logs:
Code: [Select]
WeatherCat watchdog: WeatherCat appears to have hung; rebooting as a precautionary measure.
WeatherCat TimeStamp: 11:13:59 07-Apr-18
----------
WeatherCat: Wunderground driver appears to have hung. Restarting it. Result to follow...
WeatherCat TimeStamp: 12:44:40 07-Apr-18
----------
WeatherCat: Custom alerts thread appears to have hung. Restarting it. Result to follow...
WeatherCat TimeStamp: 12:44:42 07-Apr-18
----------
WeatherCat: Wunderground driver running again.
WeatherCat TimeStamp: 12:45:11 07-Apr-18
----------
WeatherCat: Custom alerts running again.
WeatherCat TimeStamp: 12:45:12 07-Apr-18

This is an indication you are being power managed - the threads are being put to sleep which is being picked up by the watchdog and then restarted. It also appears that the main thread is also being put to sleep, which again the watchdog will pick up and in this case attempt to restart WeatherCat.

It also appear in this case that it doesn't actually restart until you screen share, whereupon it starts running again. Again looks like power management.

I would suggest as a first step, recreating the WeatherCat preferences file - whenever strange stuff starts happening this is an easy step to take to make sure the problem isn't down to a corrupt preferences file (which does happen).


To do this, first make sure WeatherCat isn't running and then access your preferences folder by using the Finder's Go->Go To Folder... menu command and enter:
~/Library/Preferences/

Scroll down the list of preferences files until you find the one named:
com.trixology.weathercat.plist

Delete (or rename) this file then reboot the machine and launch WeatherCat. The reboot after removing the old preferences file is very important - without this, the old settings will be cached in RAM and reused when WeatherCat relaunches.

Also note that many of your settings will be reset to defaults/cleared so make a note of any settings you’ll need to re-enter/change when the settings have been reset (such as your latitude/longitude).

With regard to the 'Error in StatsManager:findTotalInDateRange - hit default' message - this is a warning that somewhere on your system you are making a query for a channel that doesn't exist on your system - this could be in a web template file, custom graph, custom gauge, custom text window, banner generator or similar where there is a STAT$ type tag for a data type that doesn't exist on your system - it isn't anything to worry about, just an indication that the data requested couldn't be provided.

Finally, the 'Monitor & Relaunch' option controls the watchdog - turning this off turns it off.

Hope that helps!

xairbusdriver

  • Storm
  • *****
  • Posts: 2042
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.13.4/WC 3.0
Re: Troubles with WC3
« Reply #8 on: April 08, 2018, 07:23:41 PM »
Thanks so much. I really appreciate your working on your day off!

I've done enough pref deletes that I know to make screen shots of each page or a text file with all the values before renaming the file. [lol]

I might as well go through the files to confirm the old tags match up with the new ones, at least those that changed. I'll just remove the Gauges page since Apple has yet to fix the image creation problem. They are on a separate page as the Graphs which seem to be fine. Note to self: Verify they are updating correctly.

I will probably wait a few hours on the plist change to see if the hanging happens again. That way I can grab a sample from Activity Monitor. I turned off the video feed earlier today and it did not stop the hang. I'm trying to make changes one at a time to see what might help. I was even considering deleting the whole install and starting over, the plist change will, of course, be a lot easier. [tup]

Thanks, again, for the new version! Please enjoy the rest of your weekend! [cheer] Go play with your kids, assuming it's not too dark/snowy! [WCSmall]

Dave

  • Gentle Breeze
  • **
  • Posts: 21
    • CW7890
    • KTNPARSO2
    • ParsonsWeather.com
  • Station Details: Davis VP2
Re: Troubles with WC3
« Reply #9 on: April 08, 2018, 09:11:58 PM »
xair,

If you decide to clear everything and start over, you should consider changing the OS on your Mac Mini to Sierra. The current Mac Mini (Late 2014) originally shipped with Yosemite so it can run that OS, El Capitan or Sierra. That is unless Apple is now doing something to the firmware on the currently available Late 2014 Mac Mini to prevent it.

With all the issues reported here that are caused by High Sierra, I have kept the Mac Minis on both of my stations on 10.12.6. WC3 runs smoothly on Sierra.

I can post instructions on how to change your Mac Mini to Sierra if you are interested.



xairbusdriver

  • Storm
  • *****
  • Posts: 2042
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.13.4/WC 3.0
Re: Troubles with WC3
« Reply #10 on: April 08, 2018, 10:10:45 PM »
Unfortunately, changing the OS on a "new" mini may be a bit more complicated. Even though it was built some time in 2017, I think it has had 10.13 on it since it was accepted by the quality control folks. [rolleyes2]

I've collected a couple of WC3 Activity Samples and sent them on to the Dev. I probably should have open AM before this. When I did it today, after I found WC 'hung' again, I found a process called UserEventAgent listed as not responding. I captured a sample of it and sent it to Stu also, mainly because when I Force Quit that agent, I noticed that WC 'woke up' and started uploading data to the web. WC also did NOT seem to need to download any historical data on this Screen Sharing access.

I'm going to not make any more changes and see what AM shows if WC again 'hangs'. "watchdog" is off, page with Gauge tags is removed from the file which normally causes WC to create them, the page displaying them was also removed from the update pages. I still need to check on which STAT$ tags I might be asking WC to handle. Have not yet replaced the plist, either. One day at a time...

Dave

  • Gentle Breeze
  • **
  • Posts: 21
    • CW7890
    • KTNPARSO2
    • ParsonsWeather.com
  • Station Details: Davis VP2
Re: Troubles with WC3
« Reply #11 on: April 08, 2018, 10:44:35 PM »
Here are the steps to change the OS to Sierra on your mini if you decide you want to try it:

Backup your system to an external drive in case you need to go back to 10.13. I use Carbon Copy Cloner, it is faster and much more reliable than Time Machine.

Download Sierra from the App store, it does not appear when you search for it so use this link: http://appstore.com/mac/macossierra

After the download finishes, it will launch but you will get a message stating it can’t install, simply close it.

Insert a USB flash drive, at least 8gb in size, that has nothing you need to keep. Rename the USB drive “SierraBootInstall”.

Open terminal, copy/paste the following command and hit enter:

sudo /Applications/Install\ macOS\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/SierraBootInstall --applicationpath /Applications/Install\ macOS\ Sierra.app --nointeraction &&say Done

Enter the admin password if requested and hit enter. It can take up to 30 minutes to complete depending on the speed of the USB drive. You will see “Done” when it is finished.

Reboot the Mac and hold down the Option key. Choose “Install macOS Sierra” from the boot menu options.

Before you start the install, run Disk Utility from the menu and erase the Mac’s hard drive so you will have a clean install of Sierra. Exit Disk Utility when it is finished and run the Sierra installer.



xairbusdriver

  • Storm
  • *****
  • Posts: 2042
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.13.4/WC 3.0
Re: Troubles with WC3
« Reply #12 on: April 08, 2018, 11:16:24 PM »
Thanks. I actually have a USB drive with 10.11.6 on it so I could run DiskWarrior on the internal drive of the old mini. Might be worth a try! :thinking: At least I could back my WC Gauges back!

xairbusdriver

  • Storm
  • *****
  • Posts: 2042
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.13.4/WC 3.0
Re: Troubles with WC3
« Reply #13 on: April 09, 2018, 01:57:49 AM »
Still running after several hours. I suspect the problem may have been that 'not responding' UserEventAgent. I think that may be running any time the OS is.

But, not satisfied with things working smoothly: [banghead] [computer] I'm going to add that file to the Additional Files list at about 5 past the next hour hoping that it will run at approximately every hour + 5 minutes.

xairbusdriver

  • Storm
  • *****
  • Posts: 2042
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.13.4/WC 3.0
Re: Troubles with WC3
« Reply #14 on: April 09, 2018, 03:51:31 PM »
Still working smoothly for 12+ hours! [cheer] Thanks for all the helpful suggestions and ideas! What a community! ThU5:-) Wish I could say the same for my old iMac.

I believe the core problem was the 'not responding' UserEventAgent. Of course, I don't know what caused that, but I will check Activity Monitor first if I detect a problem on the mini.