Author Topic: Calling all curl geniuses  (Read 325 times)

xairbusdriver

  • Storm
  • *****
  • Posts: 2305
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.14.4/WC 3.0.4
Calling all curl geniuses
« on: February 08, 2019, 03:47:26 AM »
A couple of days ago I noticed my METAR displays were broken. Those are coded text messages pilots can use to get weather at most civilian airports. I use a PHP function to request the messages from NOAA. I check and read the texts about twice an hour. I then then write them to files on my server so I can get them any time I want.

At first, I thought the problem was a short power outage that occurred just before I noticed the messages were not displaying properly. Checking the files on the server I noticed they all said the same thing! They were basically just very small html files pointing to where the actual files could be found. And that's when I noticed that NOAA had "jumped" on to the httpS 'band wagon'! ;)

I changed the file that had the base url, adding the super-secret-S, but that has not changed the result: I am still not able to access the NOAA files. I have added a curl option CURLOPT_SSL_VERIFYPEER to false. But that hasn't shown any improvement. This option is not safe for transferring sensitive info, but the wx messages are certainly not even confidential, and I'm not sending any info from my end.

I don't know if any of the Template authors are providing this particular info, but I am hoping that they or others can point me to a simple solution. BTW, the new url for these messages is https://tgftp.nws.noaa.gov/data/observations/metar/stations/icao_Station_ID.TXT That ID is a four character code for the airport (KDFW, KLAX, EGLL (Heathrow), VHHH (Hong Kong), etc.).

Thanks for any constructive suggestions/links!

wurzelmac

  • Storm
  • *****
  • Posts: 1225
    • ITIROLPR2
    • Wetterstation Prägraten am Großvenediger
  • Station Details: Davis Vantage PRO2 Plus (24h fan aspirated, wireless) with UV/Solar | Weather Envoy Data Logger | Mac mini Mid 2011 2,7 GHz Intel Core i7 with 12 GB of RAM | 2x Canon EOS 1100D | macOS 10.13.6 HighSierra
Re: Call all curl geniuses
« Reply #1 on: February 08, 2019, 06:02:08 AM »
I don't know if any of the Template authors are providing this particular info, but I am hoping that they or others can point me to a simple solution.

Hello xair,
for sure no help but I can tell that 'Meteotemplate' is providing this info. I do not use this function on my website, but the demo of Meteotemplate shows it! At the bottom of the linked page you can select every airport, I think.
Cheers,
Reinhard


wvdkuil

  • Strong Breeze
  • ***
  • Posts: 136
    • DW8113 (D8113)
    • IVLAAMSG47
    • Weatherstation Wilsele/Leuven/Belgium
  • Station Details: VantagePro2+ UV,Solar - Macbook/Mini
Re: Call all curl geniuses
« Reply #2 on: February 08, 2019, 02:01:20 PM »
With the ongoing problems with their  server-moving I had to add
Code: [Select]
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

Also check  https://tgftp.nws.noaa.gov/data/observations/metar/stations/  You get all known 4 letter codes. Sometimes it is just a typo when it seems not to work.

There are providers where you can get a "partly translated" METAR for free:
https://aviationweather.gov/adds/dataserver_current/httpparam?dataSource=metars&requestType=retrieve&format=xml&hoursBeforeNow=3&mostRecent=true&stationString=EBBR
Code: [Select]
<?xml version="1.0" encoding="UTF-8"?>
<response xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XML-Schema-instance" version="1.2" xsi:noNamespaceSchemaLocation="http://aviationweather.gov/adds/schema/metar1_2.xsd">
  <request_index>34775715</request_index>
  <data_source name="metars" />
  <request type="retrieve" />
  <errors />
  <warnings />
  <time_taken_ms>5</time_taken_ms>
  <data num_results="1">
    <METAR>
      <raw_text>EBBR 081320Z 20018G29KT 9999 BKN021 09/05 Q1007 TEMPO 20020G35KT</raw_text>
      <station_id>EBBR</station_id>
      <observation_time>2019-02-08T13:20:00Z</observation_time>
      <latitude>50.9</latitude>
      <longitude>4.5</longitude>
      <temp_c>9.0</temp_c>
      <dewpoint_c>5.0</dewpoint_c>
      <wind_dir_degrees>200</wind_dir_degrees>
      <wind_speed_kt>18</wind_speed_kt>
      <wind_gust_kt>29</wind_gust_kt>
      <visibility_statute_mi>6.21</visibility_statute_mi>
      <altim_in_hg>29.734253</altim_in_hg>
      <sky_condition sky_cover="BKN" cloud_base_ft_agl="2100" />
      <flight_category>MVFR</flight_category>
      <minT_c>2.0</minT_c>
      <metar_type>METAR</metar_type>
      <elevation_m>58.0</elevation_m>
    </METAR>
  </data>
</response>
At some you register and get a free API-key: https://api.checkwx.com/metar/CYYC/decoded  this is a json file you then receive
Code: [Select]
{"results":1,
"data":[{"icao":"CYYC",
"name":"Calgary International",
"observed":"08-02-2019 @ 13:00Z",
"raw_text":"CYYC 081300Z 02011G17KT 2 1\/2SM IC BR SCT015 OVC039 M19\/M22 A3014 RMK SN2SC2SC4 SLP305",
"barometer":{"hg":30.14,"kpa":102.06,"mb":1030.5},
"ceiling":{"code":"OVC","text":"Overcast","feet_agl":3900,"meters_agl":1188.72},
"clouds":[{"code":"SCT","text":"Scattered","base_feet_agl":1500,"base_meters_agl":457.2},
          {"code":"OVC","text":"Overcast","base_feet_agl":3900,"base_meters_agl":1188.72}],
"conditions":[{"code":"IC","text":"Ice crystals"},
              {"code":"BR","text":"Mist"}],
"dewpoint":{"celsius":-22,"fahrenheit":-8},
"elevation":{"feet":3556.43,"meters":1084},
"flight_category":"IFR",
"humidity_percent":77,
"temperature":{"celsius":-19,"fahrenheit":-2},
"visibility":{"miles":"2 1\/2","miles_float":2.5,"meters":"4,023","meters_float":4023.36},
"wind":{"degrees":20,"speed_kts":11,"speed_mph":13,"speed_mps":6,"gust_kts":17,"gust_mph":20,"gust_mps":9}
}]}

Success, Wim
Development and maintenance of Leuven-Template&Scripts  at: https://support.leuven-template.eu/
Support for the HWS-template for WD http://wd34.weather-template.com/  Demo http://wd34.weather-template.com/pwsWD/
My PWS at home https://weer.sluispark.be/
And at my daughters house https://www.weerstation-herent.be/

xairbusdriver

  • Storm
  • *****
  • Posts: 2305
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.14.4/WC 3.0.4
Re: Calling all curl geniuses
« Reply #3 on: February 08, 2019, 03:17:30 PM »
Thanks for the extra options, Wim! ThU32:-)

Most of my time in building my site was in writing the PHP to handle and parse the text. That seems to work, at least in all the cases I've seen. Even the local NOAA folks couldn't fathom why some observer used some text last year. There is a list of approved codes/abbreviations, but there is also a section for human comments in special situations. My "solution" involves multiple arrays that more than covers 98% of a report. Plus a few images/symbols showing the trends of the pressure. It is all an exercise in learning PHP/html/CSS/etc. That's why I used "enjoy" in describing why I don't use complete Templates anymore.

After flying for my entire career, I know a lot of the ICAO codes! Most I never flew to, of course! But, believe me, I am aware of the dangerz of typoes!! [blush] cmu:-)

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 5239
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
macOS version of curl badly out of date. (Re: Calling all curl geniuses )
« Reply #4 on: February 08, 2019, 10:49:32 PM »
Dear X-Air, Reinhard, Wim, and WeatherCat web spinners,

This may or may not be relevant but the version of curl that is provided with macOS is years behind the current version.  This is one of the reasons Stu hasn't been able to support secure FTP for WeatherCat.  If your only problem is getting a current version of curl, you can do this through MacPorts:

https://www.macports.org/

In order to use the current version, you would need to provide a execution path to the MacPorts version to make sure you aren't using the obsolete version supplied with macOS.

Cheers, Edouard  [cheers1]

xairbusdriver

  • Storm
  • *****
  • Posts: 2305
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.14.4/WC 3.0.4
Re: Calling all curl geniuses
« Reply #5 on: February 08, 2019, 11:26:07 PM »
Thanks for the info, but we don't need any cURL updates on the computer, it's supplied on the hosted server along with their PHP install. Nor does my wife want any cURL, anyway. She was just complaining about that yesterday morning when it was almost 75 F with humidity at ~99.99%. If you made too loud a noise, drops of rain would fall nearby! cmu:-)

Still no joy even with the additional options offered. I may need to check that the hosting company hasn't made some "improvements" without telling anyone... [rolleyes2]

wvdkuil

  • Strong Breeze
  • ***
  • Posts: 136
    • DW8113 (D8113)
    • IVLAAMSG47
    • Weatherstation Wilsele/Leuven/Belgium
  • Station Details: VantagePro2+ UV,Solar - Macbook/Mini
Re: Calling all curl geniuses
« Reply #6 on: February 09, 2019, 08:08:23 AM »
Thanks for the info, but we don't need any cURL updates on the computer, it's supplied on the hosted server along with their PHP install. Nor does my wife want any cURL, anyway. She was just complaining about that yesterday morning when it was almost 75 F with humidity at ~99.99%. If you made too loud a noise, drops of rain would fall nearby! cmu:-)

Still no joy even with the additional options offered. I may need to check that the hosting company hasn't made some "improvements" without telling anyone... [rolleyes2]
In a topic on WXforum https://www.wxforum.net/index.php?topic=34378.msg370323#msg370323 there where problems with the DNS server of some webhoster.

I attach a test-script to display most errors which can occur.
Put it in the same folder as your script and run it.  .../test_url.php
leave it as is, it will try to load the METAR for AABP If that works OK you could modify the link and test your METAR.

Success, Wim
Development and maintenance of Leuven-Template&Scripts  at: https://support.leuven-template.eu/
Support for the HWS-template for WD http://wd34.weather-template.com/  Demo http://wd34.weather-template.com/pwsWD/
My PWS at home https://weer.sluispark.be/
And at my daughters house https://www.weerstation-herent.be/

xairbusdriver

  • Storm
  • *****
  • Posts: 2305
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.14.4/WC 3.0.4
Re: Calling all curl geniuses
« Reply #7 on: February 09, 2019, 03:46:07 PM »
Thanks so much Wim! That script looks very similar to the one I created last night. I think I messed up my previously working script while making changes and planned on restoring the old version from TimeMachine today and double-checking the steps. But your file includes the ability to catch and display errors which I failed to include in the originals... my bad, I know. [blush] [rolleyes2]

Just ran your script and it worked perfectly... of course! [cheer] ThU5:-)

Next I'll try mine... Working again!
Naturally, the process took longer than needed because of my lack of skills... Also learned the difference between $rawData = curl_exec ($ch) and $info = curl_getinfo($ch);
Thanks for explaining how to get and display errors, also!

I plan on putting all but one of the CURLOPT settings in an array. Mainly to make the main code block more compact. I run that block in a loop for the five airports I want. That means I need to change the curl_setopt($curl, CURLOPT_URL, $raw_METAR[$airport]); as a separate option inside the loop. All the others will be set with a curl_setopt_array($curl, $options );

Previously, I was closing the connection after the loop ended. I will look up the specifications and do some searching as to whether which method is best:
  • open/collect data/close in a loop
  • open once/collect data five times/close when finished
It probably makes no speed difference, but one way may be preferred by servers/professionals.

DoinitMyWay! Thanks to the expert help of Mr. van der Kuil! [tup]