Generic XML and Meteobridge Success

[rpmik]

Hi all,
I'm currently trialing WeatherCat. To do so, I needed to get WeatherCat to interrogate Meteobridge on the LAN. Fortunately, Meteobridge has a basic URL encoded template system and WeatherCat, since version 2.1, has a Generic XML driver. It is also fortunate that the IP address field had no character length limits for this particular to setup that Generic XML Driver. This means that Meteobridge, with possibly any weather station, can provide basic XML data to WeatherCat. In my case, I have a basic Ambient Wx station and this method has been very reliable for the last 24 hours.

In the WeatherCat Station Communications dialog, for Station Type select Generic XML (HTTP) and then enter your Meteobridge’s IP address appended with a URL encoded XML template below.

You may need to change out the sensor tags for your setup. The tags are in square brackets such as [th0temp-act]

Code: [Select]

user:password@192.168.x.y/cgi-bin/template=%3Cvalue%3E%3Cname%3E%0A%0AAirTemp_C%0A%0A%3C%2Fname%3E%3Cdata%3E%0A%0A%5Bth0temp-act%5D%0A%0A%3C%2Fdata%3E%3C%2Fvalue%3E%0A%0A%0A%3Cvalue%3E%3Cname%3E%0A%0AIntTemp_C%0A%0A%3C%2Fname%3E%3Cdata%3E%0A%0A%5Bthb0temp-act%5D%0A%0A%3C%2Fdata%3E%3C%2Fvalue%3E%0A%0A%3Cvalue%3E%3Cname%3E%0A%0ASolar%0A%0A%3C%2Fname%3E%3Cdata%3E%0A%0A%5Bsol0rad-act%5D%0A%0A%3C%2Fdata%3E%3C%2Fvalue%3E%0A%0A%3Cvalue%3E%3Cname%3E%0A%0AUVIndex%0A%0A%3C%2Fname%3E%3Cdata%3E%0A%0A%5Buv0index-act%5D%0A%0A%3C%2Fdata%3E%3C%2Fvalue%3E%0A%0A%3Cvalue%3E%3Cname%3E%0A%0ABarometer_hPa%0A%0A%3C%2Fname%3E%3Cdata%3E%0A%0A%5Bthb0press-act%5D%0A%0A%3C%2Fdata%3E%3C%2Fvalue%3E%0A%0A%3Cvalue%3E%3Cname%3E%0A%0AWindSpeed_ms%0A%0A%3C%2Fname%3E%3Cdata%3E%0A%0A%5Bwind0wind-act%5D%0A%0A%3C%2Fdata%3E%3C%2Fvalue%3E%0A%3Cvalue%3E%3Cname%3E%0A%0AWindDirect_deg%0A%0A%3C%2Fname%3E%3Cdata%3E%0A%0A%5Bwind0dir-act%5D%0A%0A%3C%2Fdata%3E%3C%2Fvalue%3E%0A%3Cvalue%3E%3Cname%3E%0A%0ARain_mm%0A%0A%3C%2Fname%3E%3Cdata%3E%0A%0A%5Brain0total-act%5D%0A%0A%3C%2Fdata%3E%3C%2Fvalue%3E%0A%3Cvalue%3E%3Cname%3E%0A%0ARH%0A%0A%3C%2Fname%3E%3Cdata%3E%0A%0A%5Bth0hum-act%5D%0A%0A%3C%2Fdata%3E%3C%2Fvalue%3E%0A%3Cvalue%3E%3Cname%3E%0A%0AIntRH%0A%0A%3C%2Fname%3E%3Cdata%3E%0A%0A%5Bthb0hum-act%5D%0A%0A%3C%2Fdata%3E%3C%2Fvalue%3E

For more details on WeatherCat's generic XML, see the change log for V2.1.0:
https://wiki.trixology.com/index.php?title=Release_Notes#V2.1.0_-_3rd_April_2015

For more details on the Meteobridge template system:
https://meteobridge.com/wiki/index.php/Templates

Hope this helps someone other than me!

[dfw_pilot]

That's one way to do it - I learn something new every day.

I've had a LAN connection between Meteobridge and WC for about 7 years now. I just tell WC to connect via WeatherLink IP and input the Meteobridge IP address.

 

[rpmik]

That's one way to do it - I learn something new every day.

I've had a LAN connection between Meteobridge and WC for about 7 years now. I just tell WC to connect via WeatherLink IP and input the Meteobridge IP address.

 

Unfortunately, Meteobridge only passes through data from Davis loggers. My intention is to eventually upgrade to Davis or similar. If there's some way to get WeatherCat to work without using Meteobridge's template.cgi, I'd love to know where that info is. In any case, that template.cgi is very powerful and I've used it to integrate with iOS Shortcuts -- definitely worth checking out!

[elagache]

Thank you rpmik for sharing this obviously little known feature of WeatherCat!

You are the first person that I'm aware of to take advantage of this feature.  Hopefully this will attract even more users to WeatherCat since it expands the possible weather stations you can use.

Thanks again,

Edouard

[Blicj11]

Wow, this is interesting. Thanks for sharing.

[rpmik]

So I've cut out Meteobridge from the hardware stack. Now I fetch data directly from my Ambient ObserverIP LAN-connected unit. I have localhost-only Apache2 running on my Mac with Python CGI to, on demand, 1) scrape ObserverIP http://192.168.x.x/livedata.htm, and 2) output, without writing to disk, the expected WeatherCat XML.

One less piece of aging hardware dangling behind a desk!

Setting up Apache2 on Sonoma is very easy since Apache2 comes with Sonoma still, even if the following instructions were meant for Ventura (be sure to add .py as a handler):
https://discussions.apple.com/docs/DOC-250006086

Some info to parse for enabling CGI .py files:
https://stackoverflow.com/a/9145915

Note that my "shebang" (#!), so that Python modules I've installed via HomeBrew import properly, is:

Code: [Select]

#!/opt/homebrew/bin/python3
If you use macOS system Python3, you'll probably get an internal server error when Python3 fails to find Python modules. And you'll get yelled at if you try to use pip to install Python modules as root (FINE). So use HomeBrew Python3 and install any needed modules using pip :-)

And, finally, I modified Python ObserverIP scraping code from
https://forums.indigodomo.com/viewtopic.php?f=249&t=22348#p176353
Which I think originally came from the WeeWx ObserverIP driver? Or the above became this:
https://github.com/poblabs/weewx-ObserverIP/blob/master/observerip.py

Since it's fairly short, I'll post my modified Python script here.

Code: [Select]

#!/opt/homebrew/bin/python3
import cgi, cgitb
from lxml import html
import requests

cgitb.enable() # for better troubleshooting, apparently?

station_ip = "192.168.7.117" # Change this to your ObserverIP address

# Get the latest live data from the ObserverIP, then create a tree from the content we can parse
page = requests.get("http://%s/livedata.htm" % station_ip, timeout=5) #5 second time out; Request docs say this is important.
page.raise_for_status() # catch status errors
tree = html.fromstring(page.content)

# Screen scrape the data. Help from:
# http://docs.python-guide.org/en/latest/scenarios/scrape/
# http://stackoverflow.com/a/22469878/1177153
inBattery = tree.xpath('//input[@name="inBattSta"]')[0].value
outBattery = tree.xpath('//input[@name="outBattSta1"]')[0].value
inTemp = tree.xpath('//input[@name="inTemp"]')[0].value
inHumid = tree.xpath('//input[@name="inHumi"]')[0].value
absPressure = tree.xpath('//input[@name="AbsPress"]')[0].value
relPressure = tree.xpath('//input[@name="RelPress"]')[0].value
outTemp = tree.xpath('//input[@name="outTemp"]')[0].value
outHumid = tree.xpath('//input[@name="outHumi"]')[0].value
windDir = tree.xpath('//input[@name="windir"]')[0].value
windSpeed = tree.xpath('//input[@name="avgwind"]')[0].value
windGust = tree.xpath('//input[@name="gustspeed"]')[0].value
solarRadiation = tree.xpath('//input[@name="solarrad"]')[0].value
# below is not available on my ObserverIP
#uv = tree.xpath('//input[@name="uv"]')[0].value
uvi = tree.xpath('//input[@name="uvi"]')[0].value
# WeatherCat expects daily accumulated rain
rainDaily = tree.xpath('//input[@name="rainofdaily"]')[0].value

# Convert from F to C
XML = "<value><name>AirTemp_C</name><data>" + str((float(outTemp)-32)*(5/9)) + "</data></value>"
XML += "<value><name>IntTemp_C</name><data>" + str((float(inTemp)-32)*(5/9)) + "</data></value>"
XML += "<value><name>RH</name><data>" + str(outHumid) + "</data></value>"
XML += "<value><name>IntRH</name><data>" + str(inHumid) + "</data></value>"
XML += "<value><name>Solar</name><data>" + str(solarRadiation) + "</data></value>"
XML += "<value><name>UVIndex</name><data>" + str(uvi) + "</data></value>"
# Convert from inHG to hPA
XML += "<value><name>Barometer_hPa</name><data>" + str(float(absPressure)*33.864) + "</data></value>"
# Convert from MPH to m/s
XML += "<value><name>WindSpeed_ms</name><data>" + str(float(windSpeed)*0.44704) + "</data></value>"
XML += "<value><name>WindDirect_deg</name><data>" + str(windDir) + "</data></value>"
XML += "<value><name>Rain_mm</name><data>" + str(float(rainDaily)*25.4) + "</data></value>"

print("Content-Type: text/plain\r\n\r\n")  # this is required or get an error (could be text/xml?)
print(XML)


[elagache]

Dear Ryan and WeatherCat tinkerers,

Thanks for sharing your upgrade!  It is obvious that you like to tinker and aren't afraid to get under the hood of MacOS.

. . . . .
If you use macOS system Python3, you'll probably get an internal server error when Python3 fails to find Python modules. And you'll get yelled at if you try to use pip to install Python modules as root (FINE). So use HomeBrew Python3 and install any needed modules using pip :-) . . . .

Your difficulty in simply using the MacOS supplied Python suggests that Apple is starting withhold some of the UNIX tools for whatever reason.  This might be why WeatherCat continues to not support the more common SFTP protocol.  WeatherCat uses curl to perform the actual FTP upload.  Last time our fearless leader (Stuart Ball, WeatherCat developer) informed us of the issue, he reported that MacOS was behind on the curl versions and one supplied didn't support SFTP.  That last report was a few years ago though.

Thanks again,

Edouard

[rpmik]

Hi all. I've got a quick update to my Python3 bridge for Ambient ObserverIP to WeatherCat. This update adds a bridge for PurpleAir. It relies on WeatherCat's Generic XML Driver's auxiliary temperatures to store AQI pm 2.5 channel As and Bs values. It uses Aux Temp 1 and Aux Temp 3. Why aux temp 1 & 3 and not 1 & 2? Because I couldn't get data to save into aux temp 2 (or 4).

Anyway, field names don't matter much to me, so I don't mind using aux temp 1 and 3 for storing AQI data. Yes, WeatherCat expects those values to be in centigrade. So here in the U.S., WC will dutifully display those aux temp 1 and 3 values in Fahrenheit. Well, that's where Synthetic Channels are useful. Simply use Synthetic Channels to convert the Fahrenheit values back to centigrade for the original AQI value and make sure the value is unitless.

Here's the simple Python3 code that reads the PurpleAir's local web interface and reports out on the AQI pm2.5 channels A and B into WeatherCat's Generic XML driver format. Add this code to the ObserverIP code above or use it as standalone:

Code: [Select]

#!/opt/homebrew/bin/python3
import cgi, cgitb
import json
import requests

cgitb.enable() # Doesn't seem to work, though.

# Change this IP address to your PurpleAir's
purple_ip = "192.168.x.x"

# load the PurpleAir JSON page
page_purple = requests.get("http://%s/json" % purple_ip, timeout=7)
page_purple.raise_for_status()

#Put the JSON content into a dictionary
purple_json = json.loads(page_purple.content)

XML = "<value><name>T1_C</name><data>" + str(float(purple_json["pm2.5_aqi"])) + "</data></value>\r"
XML += "<value><name>T3_C</name><data>" + str(float(purple_json["pm2.5_aqi_b"])) + "</data></value>\r"

print("Content-Type: text/plain\r\n\r\n")
print(XML)


That's it for the Python side of things.

Then set up a Synthetic channel or two (AQI Channel A and then B) with the following if you need to convert back to "centigrade" and set the output to unitless:

Code: [Select]

return ((Param1-32)*5)/9

Where Param1, in this case, is Aux Temp 1 (AQI pm2.5 Chan A) or Aux Temp 3 (AQI pm2.5 Chan B).

Then hope that the AQIs are indeed recorded into Aux Temp 1 and Aux Temp 3 so that the synthetic channels work so that you can create a custom chart (see attached) and set Email triggers for good and/or bad air quality.

Hope this all makes sense. I'm all about bending software to my will ;-)  I'm very happy this was possible.

Ryan

[elagache]

Thanks Ryan for your contributed solutions!

Cheers, Edouard

[staze]

Update: Not sure HOW I fixed it, might have been adding contenttype to the url. So it's now: "/cgi-bin/template.cgi?templatefile=weathercat.txt&contenttype=application/xml"

----

Previous post

@rpmik: Thanks for this, elagache linked me from my thread asking about this. What I'm not sure about is I'm trying to use a template file, and WC refuses to say it's valid.

Have you had this issue, and if so, how did you get around it? I can't tell if WC is expecting xml headers, or...?

fwiw, I'm using a template file, which you can put in templates, and call as /cgi-bin/template.cgi?templatefile=weathercat.txt (in my case).

The template is just

Code: [Select]

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<weathercat>
<value>
<name>AirTemp_C</name>
<data>[th0temp-act]</data>
</value>
<value>
<name>IntTemp_C</name>
<data>[thb0temp-act]</data>
</value>
<value>
<name>RH</name>
<data>[th0hum-act]</data>
</value>
<value>
<name>IntRH</name>
<data>[thb0hum-act]</data>
</value>
<value>
<name>Barometer_hPa</name>
<data>[thb0press-act]</data>
</value>
<value>
<name>WindSpeed_ms</name>
<data>[wind0wind-act]</data>
</value>
<value>
<name>WindDirect_deg</name>
<data>[wind0dir-act]</data>
</value>
<value>
<name>Rain_mm</name>
<data>[rain0total-act]</data>
</value>
</weathercat>
This validates as valid xml, and the fields populate correctly from MB, but WC just refuses to think it's valid... =(

fwiw, my connection string just looks like

meteobridge:password@10.0.6.22/cgi-bin/template.cgi?templatefile=weathercat.txt

I've tried with and without the http:// at the start, and that doesn't matter. =(

Also not sure about your example, it seems to have random 52's in it? It decodes to

Code: [Select]

user:password@192.168.x.y/cgi-bin/template.cgi?<value><name>AirTemp_C</name><data>52</[th0temp-act]></value><value><name>IntTemp_C</name><data>52</[thb0temp-act]></value><value><name>Solar</name><data>52</[sol0rad-act]></value><value><name>UVIndex</name><data>52</[uv0index-act]></value><value><name>Barometer_hPa</name><data>52</[thb0press-act]></value><value><name>WindSpeed_ms</name><data>52</[wind0wind-act]></value><value><name>WindDirect_deg</name><data>52</[wind0dir-act]></value><value><name>Rain_mm</name><data>52</[rain0total-act]></value><value><name>RH</name><data>52</[th0hum-act]></value><value><name>IntRH</name><data>52</[thb0hum-act]></value>


[staze]

Update: Yes, I had to add Soloar and UVIndex fields, and just leave them blank. It now doesn't say they're sensor errors.

----

Related, any idea why it would be showing every query being sensor errors?

Station Comms: Bad - Check Installation (0.00%).
Comms Errors: 0. Good Data: 3161. Sensor Errors: 3161. But it's got data!

Log shows it fetching data.

All the channels that I'm providing show status Valid. Feels like it's either expecting some data that I'm not providing (so it's a sensor error), or... My station (Vantage Vue, feeding via Meteobridge) doesn't provide UV, or Solar Radiation, or Aux Temp's, or Synth channels... do I need to just put those in the XML output but blank?

[elagache]

Dear staze and WeatherCat tinkers,

You might try sending a private message to Ryan (rpmik) to get his attention.  Unfortunately, he would be the only person who is likely to be able to help you.

Edouard

[rpmik]

Hi Staze,
Sorry I've not been around here too much. Glad you got some headway, though. I have no experience using template files since I had only an original Meteobridge without that feature. So I wouldn't have been able to help there. It'd strange to me that you needed to use perfectly constructed XML though since I did not ned the header or the <weathercat> element. Not sure if any of that may be causing the odd sensor error problem with Weathercat reading the XML. But I guess if it works now, why not? :-)

That said, hopefully I can help now.

Also not sure about your example, it seems to have random 52's in it? It decodes to

I'm not sure how that happened. I've fixed it. But don't think that'll help you at this point. Sorry about that!

Update: Yes, I had to add Soloar and UVIndex fields, and just leave them blank. It now doesn't say they're sensor errors.

----

Related, any idea why it would be showing every query being sensor errors?

Station Comms: Bad - Check Installation (0.00%).
Comms Errors: 0. Good Data: 3161. Sensor Errors: 3161. But it's got data!

Log shows it fetching data.

All the channels that I'm providing show status Valid. Feels like it's either expecting some data that I'm not providing (so it's a sensor error), or... My station (Vantage Vue, feeding via Meteobridge) doesn't provide UV, or Solar Radiation, or Aux Temp's, or Synth channels... do I need to just put those in the XML output but blank?

You should use 0 (zero) instead of blank values to avoid sensor errors. Strange meteobridge cannot report solar and UV from Davis.

[staze]

The <weathercat> piece is so it would validate as XML. =) without it (no parent element) it wouldn't validate.

Good to know about 0 vs null. I'll see about that. Right now it seems to be interpreting null and 0 as the same, and the errors have gone away and those channels now validate as valid. Meteobridge CAN report that info, but the Vantage Vue (my weather station) doesn't have UV/Solar. =)

The last piece, and what I came back to ask, is about rain. Right now I have it just providing a rain year to date value, which is currently 24". WC seems to be interpreting that fine, but it's a little funny looking and seeing that number. Should I flip that to rain today ([rain0total-daysum])? I think this is how WC previously handled it when it was talking directly to my davis rather than through meteobridge.

Thanks!

[rpmik]

The last piece, and what I came back to ask, is about rain. Right now I have it just providing a rain year to date value, which is currently 24". WC seems to be interpreting that fine, but it's a little funny looking and seeing that number. Should I flip that to rain today ([rain0total-daysum])? I think this is how WC previously handled it when it was talking directly to my davis rather than through meteobridge.

I don’t know. WC hypothetically could detect a daily rain measurement, but if it’s working now, then I would just leave it alone. That XML seems pretty rigid. Let WC handle the gory details  ;-)