Author Topic: MQTT Into WeatherCat Synthetic Channel  (Read 5135 times)

mikegf

  • Gentle Breeze
  • **
  • Posts: 14
    • FW4257
    • INSWYASS5
  • Station Details: Davis VP2 Pro, Macstudio M2U, MacOS 14.0
MQTT Into WeatherCat Synthetic Channel
« on: May 07, 2020, 05:14:28 AM »
Another obscure AppleScript integration I'd like to share in case anyone has the same requirement (doubtful!) or wishes to bring in MQTT data from another device.  Output from Weathercat to MQTT was done well way back (2012) by mangrovemike, but getting MQTT data into Weathercat wasn't so simple! 

My use case - I have bought a cheap Chinese diesel RV heater on eBay in order to heat my 6'x10' greenhouse during winter (hopefully to make better use of it). It was an "all-in-one" with 6L tank and cabinet.  I used this as I don't have mains (but do have a spare SUV battery and solar) and they are very efficient (about 250-500ml/hr for 5kW out).  The supplied controller is pretty basic but Ray Jones here in AUS (http://afterburner.mrjones.id.au) makes a brilliant replacement controller (with WiFi/BT, external temp probes etc) that can (amongst other things) be monitored/controlled via MQTT (not a lot on the site but subscribe to his Facebook group via the link at the bottom for lots of info on this).  The controller is in active development so is being added to all the time. 

While I can get the greenhouse temp via my Davis Temp/Hum station, I did want to tap into the MQTT messages to get the fuel used, put it on a custom gauge and alert me if it was getting low (rather than having to access the controller webpage or front panel all the time).  This is the resultant script.  It may also be useful as a basis for getting data from other external sensors that are MQTT capable (or you have made capable via an arduino/Esp32 etc) but not able to be integrated into your station (as a Davis user I know all about this!). The Afterburner sends data in JSON format

As mentioned, subscribing to MQTT data (ie data in) is a bit more fiddly than publishing (ie data out).  The reason is several.  Firstly the Synthetic channel AppleScript uses the data that is returned and so the script needs to complete (and is triggered periodically).  The second issue is that although the MQTT subscription via mosquitto writes to stdout, you can't get to the output directly via AppleScript while it is running.  Even using the -E (ie run until everything is read then exit) option I had issues catching and working the results on exit.  The "standard" way to access this type of script data is via redirecting outputting to a file and then reading the file back in, which is what I have done.  Also helps in debug.  The third issue is that Afterburner only sends a full JSON string periodically (with only changed items in-between) so if no other processes (such as MQTT webpage) subscribe (which also triggers a full output) then it could be some time until you get your data.  Esp an issue with the fuel burned as if the heater is not running it is not burning fuel and hence no parameter output!  So with the AppleScript exiting each time (and I could not find an obvious way to read the synthetic channel's previous value), you also need to save older data to file to recover it for the next AppleScript cycle.

The script is assuming that you are using the mosquitto broker (like mangrovemike's setup) and have installed JSONHelper (free from Mac Store).  It could be modified for other brokers as it is done via shell script anyway.  You don't need JSONHelper, but the native code then required is awful unless your are a Mac dev!

In essence I have a file (inFile) to capture the subscription output from mosquitto_sub() and a file to hold the last fuel usage value.  After I read the old value back in, I run the mosquito_sub() script to get the latest MQTT data.  There are several command line options here which are important:

1. I need to maintain a persistent session between the individual AppleScript runs.  So I need the '--disable-clean-session' to make it persistent and '-i MosqTestS' to manually set a session name to what I want (in this case MosqTestS).
2. In order to store all (normally) "non-retained" data on the broker until the next subscription request the QOS must be 1 or 2 (Afterburner makes the data non-retained explicitly and I have no intent on forking/modifying the codebase!)
3. The '-E' causes the script to exit once all available MQTT subscription items have been read (rather than crudely killing the process!)

I then try to read the lines of subscription info in inFile, run them through JSONHelper to decode the JSON text into an AppleScript list, and if the parameter I need (PumpCount) is present then I assign it to a variable to use.  Note that once I have exited the repeat loop I will only have the latest value in newPumpCount.  I then convert it to fuel used, save it to lastFile and return the value.  Separately I have an alert message to tell me if the fuel used is over 5000ml. 

The fuel usage is manually reset (via either the web interface or controller) after you refill the tank.

Hopefully this may be of use to someone


xairbusdriver

  • Storm
  • *****
  • Posts: 3127
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #1 on: May 07, 2020, 05:01:58 PM »
Thanks, Mike! I assume you don't have a helpful wife to run out and read the fuel gauge for you? [lol]

BTW, you might be prepared to get some questions about AS and it's use in Catalina macOS 10.15! ;D
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


Blicj11

  • Storm
  • *****
  • Posts: 3945
    • EW3808
    • KUTHEBER6
    • Timber Lakes Weather
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP Data Logger | iMac (2019), 3.6 GHz Intel Core i9, 40 GB RAM, macOS Ventura 13.6 | Sharx SCNC2900 Webcam | WeatherCat 3.3 | Supportive Wife
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #2 on: May 07, 2020, 06:53:03 PM »
Mike, thank you so much for taking the time to explain how you solved the fuel monitoring issue. This is really quite fascinating stuff and I appreciate you sharing it with us.
Blick


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Thanks for sharing! (Re: MQTT Into WeatherCat Synthetic Channel)
« Reply #3 on: May 07, 2020, 11:44:38 PM »
Dear Mike, X-Air, Blick, and WeatherCat Scripters,

Indeed thanks for sharing!  I wasn't aware of JSON Helper, it does look interesting!  I have had a long interest in doing something somewhat similar.  Our garage is under 2 of our bedrooms and when we use a cars late in the day, that warms up the garage and floor under the bedrooms.  It seemed like a simple enough task to mount a Davis temperature probe in the garage and then operate a fan to cool the garage when the outside air was cold enough to actually be able to cool the garage.

One small problem in this scheme, I've never had the time to figure out how to set up the fan! . . . . .

Thanks again for sharing!  Unlike mine, your script is a success story!  [tup]

Cheers Edouard  [cheers1]

mikegf

  • Gentle Breeze
  • **
  • Posts: 14
    • FW4257
    • INSWYASS5
  • Station Details: Davis VP2 Pro, Macstudio M2U, MacOS 14.0
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #4 on: May 08, 2020, 02:22:48 AM »
Hi Edouard,

During my "adventure" getting this done I also picked up what others had been doing with the Afterburner controller.  A number of them have used Home Assistant and Mosquitto MQTT running on a Pi to integrate their IoT stuff in their RVs - one has even been able to control his Afterburner heater via Alexa (which I also assume he uses for music, lights etc).

With a number of integrations available for Home Assistant (see https://www.home-assistant.io/integrations/) I suspect you could quite simply implement mangrovemike's weathercat-MQTT bridge script to send temp to a Pi,  then control a fan via one of the IoT plugins (ie simple on/off mains) that they use for lamps etc (like this https://www.amazon.com/Z-Wave-Power-Switch-ZEN15-Humidifiers/dp/B07578W7KY - just from a quick google so no endorsement).  Without having used Home Assistant, I suspect that you may even be able to send the relevant commands for the z-wave directly from Weathercat (ie rather than send temp, you would construct the relevant "on" or "off" command).   JSON helper also encodes as well if you need it.

One of my options here was to send the Afterburner Heater on/of commands using my last empty synthetic channel, based on the Davis Temp/Hum sensor in the greenhouse (that I paid a lot for and probably don't really need now!).  But given that the Afterburner setup is pretty sophisticated I decided to let it run independently (KISS) and just use the script monitor the fuel.

Mike

mikegf

  • Gentle Breeze
  • **
  • Posts: 14
    • FW4257
    • INSWYASS5
  • Station Details: Davis VP2 Pro, Macstudio M2U, MacOS 14.0
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #5 on: May 08, 2020, 02:33:26 AM »
Thanks, Mike! I assume you don't have a helpful wife to run out and read the fuel gauge for you? [lol]

BTW, you might be prepared to get some questions about AS and it's use in Catalina macOS 10.15! ;D

Hi X-Air,

Have the wife but fat chance of that.  In fact I know I am also hopeless remembering to do routine tasks so need to automate as much as possible (eg veg gardens, fruit trees, greenhouse all controlled by an Hunter internet irrigation controller, Afterburner to heat the greenhouse, water level monitoring stations on all my home tanks, Grainfather home brew fermenters with Tilt/TiltPi hydrometers [beer] etc).

And don't think I'm an AS guru - wouldn't even rate myself as competent.  Just did a lot of computer science stuff in my engineering degrees and most of my engineering career was in operations (where you always seem to have inadequate tech info, spares etc so need to be able to research/learn very quickly!!).  But happy to help if I can (I just don't spend too much time on here consistently)

Mike

The Grand Poohbah

  • Gale
  • ****
  • Posts: 385
  • Developer of WeatherCat for iOS, tvOS, and watchOS
    • EW6355 KCANEVAD43
    • Hopeful Hill Ranch
  • Station Details: Vantage Pro 2, aspirated, solar radiation, uv, soil temp and moisture
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #6 on: May 08, 2020, 08:00:44 PM »
Mike,

Very impressive work. I've been having a lot of fun using MQTT to connect the nodeMCU/esp8266 with a variety of cloud services, but didn't realize that AppleScript was the way to connect the nodeMCU/esp8266 to WeatherCat. Now all I have to do is learn AppleScript...*sigh*

Please consider writing a "How to" WeatherCat Wiki describing how to connect WeatherCat to external IoT devices. It would help open new vistas of sensor data gathering and reporting.

--grand

xairbusdriver

  • Storm
  • *****
  • Posts: 3127
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #7 on: May 08, 2020, 09:06:36 PM »
AppleScript is easy! After all, it's just typing! cmu:-)
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


mikegf

  • Gentle Breeze
  • **
  • Posts: 14
    • FW4257
    • INSWYASS5
  • Station Details: Davis VP2 Pro, Macstudio M2U, MacOS 14.0
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #8 on: May 09, 2020, 02:19:57 AM »
Mike,

Very impressive work. I've been having a lot of fun using MQTT to connect the nodeMCU/esp8266 with a variety of cloud services, but didn't realize that AppleScript was the way to connect the nodeMCU/esp8266 to WeatherCat. Now all I have to do is learn AppleScript...*sigh*

Please consider writing a "How to" WeatherCat Wiki describing how to connect WeatherCat to external IoT devices. It would help open new vistas of sensor data gathering and reporting.

--grand

I'll see how I go time-wise.

You should be able to quickly modify my script to get your MQTT data in if it is JSON format without a lot of AS knowledge using your core programming skills (I don't have much AS knowledge and I am hardly what I would call a professional programmer!) - just install mosquitto (via homebew) and JSONHelper from AppStore.  Key is making sure your MQTT data is QOS1 (so non-retained data is held for you by the broker until you ask).  If not then you may need to read/alter the data on your node and re-publish.  You can do all the testing with the Script Editor (in Utilities) rather than WeatherCat.  If you show the log you can see replies to help with debug. 

While you can suck out multiple data items from the MQTT messages (in the try section of the JSONHelper loop), of course you can only return a numerical value to the syn channel result.  If you use multiple syn channels then make sure you set the mosquitto_sub() session names and lastFile file name different for each!  I was going to suck out the pumpCal variable as well from the MQTT stream, but given that every single pump (OEM and Chinese copy) is the same displacement, hard coding was easier!

I have solved my problem (which was poor AS syntax) with getting the correct result from the mosquitto_sub() call without using a file so have updated the code (also added a little more error tolerance) and have attached the new version.

If anyone knows how to get the last value of the current Syn channel please let me know (then I can remove the remaining file).  It is not accessible from the dropdown and I even tried getting it via an AS call to WC itself (STAT$SYN4 in my case) but it would appear that the live value of the syn channel info is cleared on entry to the AS script (?).  I could probably do it via a second syn channel but for a wider use (say getting in external data from non-weather station channels like grand wants) that would chew up the channels and effectively limit to 2.

Mike

mangrovemike

  • Calm
  • *
  • Posts: 9
    • DW6277
    • INSWMERE2
    • Merewether Weather
  • Station Details: Davis Station Vantage Vue Pro
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #9 on: August 24, 2020, 07:22:39 AM »
Awesome job Mike!

8 years after I wrote the script to put WeatherCat data into MQTT, I was curious if it was possible to go the other way. Was thinking along the lines that you had outlined here. Brilliant and thank you.

Mangrove
(the other Mike ;-) )

mikegf

  • Gentle Breeze
  • **
  • Posts: 14
    • FW4257
    • INSWYASS5
  • Station Details: Davis VP2 Pro, Macstudio M2U, MacOS 14.0
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #10 on: August 25, 2020, 01:40:26 AM »
Glad I could repay for the work you did  [cheers1]

I am also using your script to get data out of Weathercat and into Home Assistance via MQTT.  I am setting it up now to take soil moisture data from my garden (via Davis probe and Weathercat) and (hopefully) allow it to trigger additional retic actions (I am using Hunter Hydrawise) if it gets too dry this upcoming summer (I have gone to hydroponic vertical gardens with coco-coir which obviously dry out much quicker than soil)

I did notice that your script (with the main repeat loop) effectively "froze" the app and so you needed to force quit to exit.  I forked the code and updated it to use an idle loop instead.  That means that if you export as an executable and check "stay open after run handler" it will re-run after the set idle time (which is the return value of the script) without taking up too much processor time.  I then add it to my startup items.  If you are interested in the change I'll send you a pull request in GitHub.

Mike

mangrovemike

  • Calm
  • *
  • Posts: 9
    • DW6277
    • INSWMERE2
    • Merewether Weather
  • Station Details: Davis Station Vantage Vue Pro
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #11 on: October 24, 2020, 03:35:19 AM »
I did notice that your script (with the main repeat loop) effectively "froze" the app and so you needed to force quit to exit.  I forked the code and updated it to use an idle loop instead.  That means that if you export as an executable and check "stay open after run handler" it will re-run after the set idle time (which is the return value of the script) without taking up too much processor time.  I then add it to my startup items.  If you are interested in the change I'll send you a pull request in GitHub.

Mike

That would be great for the pull request. Thanks.

mikegf

  • Gentle Breeze
  • **
  • Posts: 14
    • FW4257
    • INSWYASS5
  • Station Details: Davis VP2 Pro, Macstudio M2U, MacOS 14.0
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #12 on: October 24, 2020, 07:00:19 AM »
Pull request done. 

Been running this pretty much full time since Jul (feeding Home Assistant) so it is pretty stable.   Occasionally you may get an error if it fails to talk to WeatherCat.  Only had it 2 or 3 times in that time - can't say whether its the script, WeatherCat or just a Mac Issue.

Mike

mikegf

  • Gentle Breeze
  • **
  • Posts: 14
    • FW4257
    • INSWYASS5
  • Station Details: Davis VP2 Pro, Macstudio M2U, MacOS 14.0
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #13 on: August 30, 2021, 11:25:03 PM »
Just an update for those who might stumble upon this - I have modified the script to make it more stable.  I have put in pull requests to mangrove mike but my forked repo is at https://github.com/mikethefarmer/WeatherCatMosquittoBridge

Blicj11

  • Storm
  • *****
  • Posts: 3945
    • EW3808
    • KUTHEBER6
    • Timber Lakes Weather
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP Data Logger | iMac (2019), 3.6 GHz Intel Core i9, 40 GB RAM, macOS Ventura 13.6 | Sharx SCNC2900 Webcam | WeatherCat 3.3 | Supportive Wife
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #14 on: September 01, 2021, 11:28:02 PM »
Mike:

Thanks for keeping us in the loop on this labor of love. Well done.
Blick