Author Topic: MQTT Into WeatherCat Synthetic Channel  (Read 530 times)

mikegf

  • Calm
  • *
  • Posts: 7
    • FW4257
    • INSWYASS5
  • Station Details: Davis VP2 Pro, iMac 17,1 MacOS 10.15
MQTT Into WeatherCat Synthetic Channel
« on: May 07, 2020, 05:14:28 am »
Another obscure AppleScript integration I'd like to share in case anyone has the same requirement (doubtful!) or wishes to bring in MQTT data from another device.  Output from Weathercat to MQTT was done well way back (2012) by mangrovemike, but getting MQTT data into Weathercat wasn't so simple! 

My use case - I have bought a cheap Chinese diesel RV heater on eBay in order to heat my 6'x10' greenhouse during winter (hopefully to make better use of it). It was an "all-in-one" with 6L tank and cabinet.  I used this as I don't have mains (but do have a spare SUV battery and solar) and they are very efficient (about 250-500ml/hr for 5kW out).  The supplied controller is pretty basic but Ray Jones here in AUS (http://afterburner.mrjones.id.au) makes a brilliant replacement controller (with WiFi/BT, external temp probes etc) that can (amongst other things) be monitored/controlled via MQTT (not a lot on the site but subscribe to his Facebook group via the link at the bottom for lots of info on this).  The controller is in active development so is being added to all the time. 

While I can get the greenhouse temp via my Davis Temp/Hum station, I did want to tap into the MQTT messages to get the fuel used, put it on a custom gauge and alert me if it was getting low (rather than having to access the controller webpage or front panel all the time).  This is the resultant script.  It may also be useful as a basis for getting data from other external sensors that are MQTT capable (or you have made capable via an arduino/Esp32 etc) but not able to be integrated into your station (as a Davis user I know all about this!). The Afterburner sends data in JSON format

As mentioned, subscribing to MQTT data (ie data in) is a bit more fiddly than publishing (ie data out).  The reason is several.  Firstly the Synthetic channel AppleScript uses the data that is returned and so the script needs to complete (and is triggered periodically).  The second issue is that although the MQTT subscription via mosquitto writes to stdout, you can't get to the output directly via AppleScript while it is running.  Even using the -E (ie run until everything is read then exit) option I had issues catching and working the results on exit.  The "standard" way to access this type of script data is via redirecting outputting to a file and then reading the file back in, which is what I have done.  Also helps in debug.  The third issue is that Afterburner only sends a full JSON string periodically (with only changed items in-between) so if no other processes (such as MQTT webpage) subscribe (which also triggers a full output) then it could be some time until you get your data.  Esp an issue with the fuel burned as if the heater is not running it is not burning fuel and hence no parameter output!  So with the AppleScript exiting each time (and I could not find an obvious way to read the synthetic channel's previous value), you also need to save older data to file to recover it for the next AppleScript cycle.

The script is assuming that you are using the mosquitto broker (like mangrovemike's setup) and have installed JSONHelper (free from Mac Store).  It could be modified for other brokers as it is done via shell script anyway.  You don't need JSONHelper, but the native code then required is awful unless your are a Mac dev!

In essence I have a file (inFile) to capture the subscription output from mosquitto_sub() and a file to hold the last fuel usage value.  After I read the old value back in, I run the mosquito_sub() script to get the latest MQTT data.  There are several command line options here which are important:

1. I need to maintain a persistent session between the individual AppleScript runs.  So I need the '--disable-clean-session' to make it persistent and '-i MosqTestS' to manually set a session name to what I want (in this case MosqTestS).
2. In order to store all (normally) "non-retained" data on the broker until the next subscription request the QOS must be 1 or 2 (Afterburner makes the data non-retained explicitly and I have no intent on forking/modifying the codebase!)
3. The '-E' causes the script to exit once all available MQTT subscription items have been read (rather than crudely killing the process!)

I then try to read the lines of subscription info in inFile, run them through JSONHelper to decode the JSON text into an AppleScript list, and if the parameter I need (PumpCount) is present then I assign it to a variable to use.  Note that once I have exited the repeat loop I will only have the latest value in newPumpCount.  I then convert it to fuel used, save it to lastFile and return the value.  Separately I have an alert message to tell me if the fuel used is over 5000ml. 

The fuel usage is manually reset (via either the web interface or controller) after you refill the tank.

Hopefully this may be of use to someone


xairbusdriver

  • Storm
  • *****
  • Posts: 2467
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.14.4/WC 3.0.4
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #1 on: May 07, 2020, 05:01:58 pm »
Thanks, Mike! I assume you don't have a helpful wife to run out and read the fuel gauge for you? [lol]

BTW, you might be prepared to get some questions about AS and it's use in Catalina macOS 10.15! ;D

Blicj11

  • Storm
  • *****
  • Posts: 3382
    • EW3808
    • KUTHEBER6
    • Timber Lakes Utah
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP Data Logger | iMac (2019), 3.6 GHz Intel Core i9, 40 GB RAM, macOS Mojave | Sharx SCNC2900 Webcam | Supportive Wife
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #2 on: May 07, 2020, 06:53:03 pm »
Mike, thank you so much for taking the time to explain how you solved the fuel monitoring issue. This is really quite fascinating stuff and I appreciate you sharing it with us.
Blick


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 5487
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Thanks for sharing! (Re: MQTT Into WeatherCat Synthetic Channel)
« Reply #3 on: May 07, 2020, 11:44:38 pm »
Dear Mike, X-Air, Blick, and WeatherCat Scripters,

Indeed thanks for sharing!  I wasn't aware of JSON Helper, it does look interesting!  I have had a long interest in doing something somewhat similar.  Our garage is under 2 of our bedrooms and when we use a cars late in the day, that warms up the garage and floor under the bedrooms.  It seemed like a simple enough task to mount a Davis temperature probe in the garage and then operate a fan to cool the garage when the outside air was cold enough to actually be able to cool the garage.

One small problem in this scheme, I've never had the time to figure out how to set up the fan! . . . . .

Thanks again for sharing!  Unlike mine, your script is a success story!  [tup]

Cheers Edouard  [cheers1]

mikegf

  • Calm
  • *
  • Posts: 7
    • FW4257
    • INSWYASS5
  • Station Details: Davis VP2 Pro, iMac 17,1 MacOS 10.15
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #4 on: May 08, 2020, 02:22:48 am »
Hi Edouard,

During my "adventure" getting this done I also picked up what others had been doing with the Afterburner controller.  A number of them have used Home Assistant and Mosquitto MQTT running on a Pi to integrate their IoT stuff in their RVs - one has even been able to control his Afterburner heater via Alexa (which I also assume he uses for music, lights etc).

With a number of integrations available for Home Assistant (see https://www.home-assistant.io/integrations/) I suspect you could quite simply implement mangrovemike's weathercat-MQTT bridge script to send temp to a Pi,  then control a fan via one of the IoT plugins (ie simple on/off mains) that they use for lamps etc (like this https://www.amazon.com/Z-Wave-Power-Switch-ZEN15-Humidifiers/dp/B07578W7KY - just from a quick google so no endorsement).  Without having used Home Assistant, I suspect that you may even be able to send the relevant commands for the z-wave directly from Weathercat (ie rather than send temp, you would construct the relevant "on" or "off" command).   JSON helper also encodes as well if you need it.

One of my options here was to send the Afterburner Heater on/of commands using my last empty synthetic channel, based on the Davis Temp/Hum sensor in the greenhouse (that I paid a lot for and probably don't really need now!).  But given that the Afterburner setup is pretty sophisticated I decided to let it run independently (KISS) and just use the script monitor the fuel.

Mike

mikegf

  • Calm
  • *
  • Posts: 7
    • FW4257
    • INSWYASS5
  • Station Details: Davis VP2 Pro, iMac 17,1 MacOS 10.15
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #5 on: May 08, 2020, 02:33:26 am »
Thanks, Mike! I assume you don't have a helpful wife to run out and read the fuel gauge for you? [lol]

BTW, you might be prepared to get some questions about AS and it's use in Catalina macOS 10.15! ;D

Hi X-Air,

Have the wife but fat chance of that.  In fact I know I am also hopeless remembering to do routine tasks so need to automate as much as possible (eg veg gardens, fruit trees, greenhouse all controlled by an Hunter internet irrigation controller, Afterburner to heat the greenhouse, water level monitoring stations on all my home tanks, Grainfather home brew fermenters with Tilt/TiltPi hydrometers [beer] etc).

And don't think I'm an AS guru - wouldn't even rate myself as competent.  Just did a lot of computer science stuff in my engineering degrees and most of my engineering career was in operations (where you always seem to have inadequate tech info, spares etc so need to be able to research/learn very quickly!!).  But happy to help if I can (I just don't spend too much time on here consistently)

Mike

The Grand Poohbah

  • Gale
  • ****
  • Posts: 312
  • Developer of WeatherCat for iOS, tvOS, and watchOS
    • EW6355 KCANEVAD43
    • Hopeful Hill Ranch
  • Station Details: Vantage Pro 2, aspirated, solar radiation, uv, soil temp and moisture
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #6 on: May 08, 2020, 08:00:44 pm »
Mike,

Very impressive work. I've been having a lot of fun using MQTT to connect the nodeMCU/esp8266 with a variety of cloud services, but didn't realize that AppleScript was the way to connect the nodeMCU/esp8266 to WeatherCat. Now all I have to do is learn AppleScript...*sigh*

Please consider writing a "How to" WeatherCat Wiki describing how to connect WeatherCat to external IoT devices. It would help open new vistas of sensor data gathering and reporting.

--grand

xairbusdriver

  • Storm
  • *****
  • Posts: 2467
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.14.4/WC 3.0.4
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #7 on: May 08, 2020, 09:06:36 pm »
AppleScript is easy! After all, it's just typing! cmu:-)

mikegf

  • Calm
  • *
  • Posts: 7
    • FW4257
    • INSWYASS5
  • Station Details: Davis VP2 Pro, iMac 17,1 MacOS 10.15
Re: MQTT Into WeatherCat Synthetic Channel
« Reply #8 on: May 09, 2020, 02:19:57 am »
Mike,

Very impressive work. I've been having a lot of fun using MQTT to connect the nodeMCU/esp8266 with a variety of cloud services, but didn't realize that AppleScript was the way to connect the nodeMCU/esp8266 to WeatherCat. Now all I have to do is learn AppleScript...*sigh*

Please consider writing a "How to" WeatherCat Wiki describing how to connect WeatherCat to external IoT devices. It would help open new vistas of sensor data gathering and reporting.

--grand

I'll see how I go time-wise.

You should be able to quickly modify my script to get your MQTT data in if it is JSON format without a lot of AS knowledge using your core programming skills (I don't have much AS knowledge and I am hardly what I would call a professional programmer!) - just install mosquitto (via homebew) and JSONHelper from AppStore.  Key is making sure your MQTT data is QOS1 (so non-retained data is held for you by the broker until you ask).  If not then you may need to read/alter the data on your node and re-publish.  You can do all the testing with the Script Editor (in Utilities) rather than WeatherCat.  If you show the log you can see replies to help with debug. 

While you can suck out multiple data items from the MQTT messages (in the try section of the JSONHelper loop), of course you can only return a numerical value to the syn channel result.  If you use multiple syn channels then make sure you set the mosquitto_sub() session names and lastFile file name different for each!  I was going to suck out the pumpCal variable as well from the MQTT stream, but given that every single pump (OEM and Chinese copy) is the same displacement, hard coding was easier!

I have solved my problem (which was poor AS syntax) with getting the correct result from the mosquitto_sub() call without using a file so have updated the code (also added a little more error tolerance) and have attached the new version.

If anyone knows how to get the last value of the current Syn channel please let me know (then I can remove the remaining file).  It is not accessible from the dropdown and I even tried getting it via an AS call to WC itself (STAT$SYN4 in my case) but it would appear that the live value of the syn channel info is cleared on entry to the AS script (?).  I could probably do it via a second syn channel but for a wider use (say getting in external data from non-weather station channels like grand wants) that would chew up the channels and effectively limit to 2.

Mike