Author Topic: Meteotemplate  (Read 60273 times)

jachym

  • Strong Breeze
  • ***
  • Posts: 210
    • Meteotemplate
  • Station Details: WH1080
Re: Meteotemplate
« Reply #255 on: November 03, 2016, 12:58:48 PM »
OK, I need your help now :D

I need someone to provide me a link to their realtimeMT.txt, which also includes the soil temperature etc. data, I need something to test it.

Steve

  • Global Moderator
  • Storm
  • *****
  • Posts: 1344
    • DW8454
    • KOHAVON11
    • Avon Weather
  • Station Details: Davis Wireless VP2 Plus w/24 hr FARS, Leaf & Soil Station, 2017 27" iMac 3.8 GHz i5, 40GB RAM, Mac OS 10.14, WeatherCat 3
Re: Meteotemplate
« Reply #256 on: November 03, 2016, 01:12:29 PM »
Steve - Avon, Ohio, USA


CWOP: DW8454 • WU: KOHAVON11 • AWEKAS • CoCoRaHS • Skywarn •  UK Met Office
MWWN •PWSweather • WeatherBug • WeatherCloud • Twitter • Facebook

jachym

  • Strong Breeze
  • ***
  • Posts: 210
    • Meteotemplate
  • Station Details: WH1080
Re: Meteotemplate
« Reply #257 on: November 03, 2016, 01:26:50 PM »
Thanks, but the problem is that this file has no data for the soil temperature, soil moisture etc. If you look at the fields, they are all 0 (or in case of temperature 32.0F, which is 0C)

The fields are ST1 (soil T), SM1 (soil moisture) and LW1 (leaf wetness)

Do you measure this? The tag was not replaced by WCat

Steve

  • Global Moderator
  • Storm
  • *****
  • Posts: 1344
    • DW8454
    • KOHAVON11
    • Avon Weather
  • Station Details: Davis Wireless VP2 Plus w/24 hr FARS, Leaf & Soil Station, 2017 27" iMac 3.8 GHz i5, 40GB RAM, Mac OS 10.14, WeatherCat 3
Re: Meteotemplate
« Reply #258 on: November 03, 2016, 02:38:54 PM »
Yea, that's just poor timing. My Soil Station transmitter and Davis Console periodically lose communication with each other. Sometimes it'll be weeks, between, others a day apart. I haven't been able to track down the problem. Right now it is not communicating. It should show correctly in a couple of hours. Sorry for the confusion.



I have sensors for:
Soil Temp 1,2, and 4
Soil Moisture 1,2,and 4
Leaf Wetness 1
Leaf Temperature 1 is calculated; there is no sensor for that.
Steve - Avon, Ohio, USA


CWOP: DW8454 • WU: KOHAVON11 • AWEKAS • CoCoRaHS • Skywarn •  UK Met Office
MWWN •PWSweather • WeatherBug • WeatherCloud • Twitter • Facebook

jachym

  • Strong Breeze
  • ***
  • Posts: 210
    • Meteotemplate
  • Station Details: WH1080
Re: Meteotemplate
« Reply #259 on: November 03, 2016, 02:40:34 PM »
The plugin shows soil T, soil moisture and leaf wetness and it will take values from your sensors "1"

Steve

  • Global Moderator
  • Storm
  • *****
  • Posts: 1344
    • DW8454
    • KOHAVON11
    • Avon Weather
  • Station Details: Davis Wireless VP2 Plus w/24 hr FARS, Leaf & Soil Station, 2017 27" iMac 3.8 GHz i5, 40GB RAM, Mac OS 10.14, WeatherCat 3
Re: Meteotemplate
« Reply #260 on: November 03, 2016, 02:56:46 PM »
Can you please tell me what exactly was the problem when trying to import the data, what the format of the date is etc. Or just post a link to one of these CSV files I will try it on my test server

The import problem was with the time format in the WeatherCat data. It is shown as 2016/11/02T16:33:00 when it expects to see 2016-11-02 16:33 using format: Y/m/d H:i

Testing the input results in the following:

Quote
Testing CSV/text file...

Errors will be highlighted in red...

Loading CSV/text file from 2011.csv

File loaded successfully.
Header row in the file, skipping first line.

The tested file has 33397 data sets. This import test will load only the first line.



Starting to parse data.

Loading line 1...
Separating fields...
Decimal separator is a period, no need to do anything, can be parsed by PHP.


Parsing date and time...Date and time in a single field.
MySQL accepts date and time in format YYYY-MM-DD HH:MM, the parsed date and time is:



Parsing variables...

Temperature
Raw value: 64.3
Database temperature units: F
File temperature units: F
Same units, no need to do any conversion.
Final temperature value: 64.3 F

Humidity
Final humidity value: 61 %

Pressure
Raw value: 29.293
Database pressure units: inhg
File pressure units: inhg
Same units, no need to do any conversion.
Final pressure value: 29.29 inhg

Wind speed
Raw value: 1.0
Database wind speed units: mph
File wind speed units: mph
Same units, no need to do any conversion.
Final wind speed value: 1.0 mph

Wind gust
Raw value: 11.0
Database wind speed units: mph
File wind gust units: mph
Same units, no need to do any conversion.
Final wind speed value: 11.0 mph

Precipitation
Raw value: 0.00
Database precipitation units: in
File precipitation units: in
Same units, no need to do any conversion.
Final precipitation value: 0.00 in

Rain rate
Raw value: 0.00
Database precipitation units: in/h
Rain rate is saved in the database in units per hour.
File rain rate units: in
Same units, no need to do any conversion.
Final rain rate value: 0.00 in/h

Wind direction
Final wind direction value: 48 degrees

Solar sensor enabled in main settings, reading data...
Solar radiation
Final solar radiation value: 0 W/m2


Parsing line 1 finished.
Checking validity of data based on your limits in Main settings...
Checking date and time...
There seems to be a problem with the date and time. Either it has not been parsed correctly or it is a date/time in the future! Check the format you specified and also the actual values.
Minimum temperature allowed: -40 F
Maximum temperature allowed: 120 F
Temperature OK.
Minimum humidity allowed: 0 %
Maximum humidity allowed: 100 %
Humidity OK.
Minimum pressure allowed: 28 inhg
Maximum pressure allowed: 32 inhg
Pressure OK.
Minimum wind speed allowed: 0 mph
Maximum wind speed allowed: 60 mph
Wind speed OK.
Minimum wind gust allowed: 0 mph
Maximum wind gust allowed: 60 mph
Wind gust OK.
Minimum daily precipitation allowed: 0 in
Maximum daily precipitation allowed: 200 in
Daily precipitation OK.
Minimum rain rate allowed: 0 in/h
Maximum rain rate allowed: 500 in/h
Rain rate OK.
Minimum wind direction allowed: 0 degrees
Maximum wind direction allowed: 360 degrees
Wind direction OK.
Minimum solar radiation allowed: 0 W/m2
Maximum solar radiation allowed: 2000 W/m2
Solar radiation OK.

Calculating dew point based on parsed values...
Temperature and humidity both valid... converting and calculating dew point....
Dew point calculated: 50.54 F
Calculating apparent temperature based on parsed values...
Temperature, humidity, and wind speed all valid... converting and calculating apparent temperature....
Apparent temperature calculated: 64.04 F

Summary
There is a problem with the date and time, nothing would be imported. Check the date/time format you specified and the actual values in the file.




--- END ----

The file is located at http://www.avon-weather.com/template/admin/2011.csv
Steve - Avon, Ohio, USA


CWOP: DW8454 • WU: KOHAVON11 • AWEKAS • CoCoRaHS • Skywarn •  UK Met Office
MWWN •PWSweather • WeatherBug • WeatherCloud • Twitter • Facebook

Steve

  • Global Moderator
  • Storm
  • *****
  • Posts: 1344
    • DW8454
    • KOHAVON11
    • Avon Weather
  • Station Details: Davis Wireless VP2 Plus w/24 hr FARS, Leaf & Soil Station, 2017 27" iMac 3.8 GHz i5, 40GB RAM, Mac OS 10.14, WeatherCat 3
Re: Meteotemplate
« Reply #261 on: November 03, 2016, 03:09:32 PM »
I'm still catching up from yesterday's comments, so bear with my slowness.


UV is included in some of the blocks etc. When I was creating MT over a yr ago I didnt know some stations measure it and my station does not even measure solar so it is problematic for me to test this. There are other reasons for not including UV in the db as well (eg. it is not included in most log files by other SW etc.). Right now it would mean rewriting all the scripts, so it just is only in the current blocks.

I did include the soil T etc. in the WCat plugin so that it can be later on included in the blocks you mentioned, just a matter of time, but it is planned

Okay, I wrongfully assumed that you were not bringing in UV and soil data. When I was mapping the fields for importing my six years of data, it did not ask for UV or any soil data, so I assumed you were not using it. That was my error. You may be storing new data (brought in by the weatherCat.php plugin) in the database, but not bringing in any historical data. I haven't looked in the database to see what you are storing (or even know how to look in there.)

Since I'm just creating this new database, I was hoping to bring in six years of historical data, including the soil info. A lot of the information is just transient (displayed and then forgotten) such as UV, current temp, etc. Soil data (including ET) needs to have historical data for trends, year over year comparisons, etc. If I end up replacing my main weather web page with MeteoTemplate, I can simply create text blocks with WeatherCats's graphs and statistics.

Thanks for the clarification. My web site has been static for a few years, and I'm enjoying the challenge to learn something new. Your patience is greatly appreciated!
Steve - Avon, Ohio, USA


CWOP: DW8454 • WU: KOHAVON11 • AWEKAS • CoCoRaHS • Skywarn •  UK Met Office
MWWN •PWSweather • WeatherBug • WeatherCloud • Twitter • Facebook

Steve

  • Global Moderator
  • Storm
  • *****
  • Posts: 1344
    • DW8454
    • KOHAVON11
    • Avon Weather
  • Station Details: Davis Wireless VP2 Plus w/24 hr FARS, Leaf & Soil Station, 2017 27" iMac 3.8 GHz i5, 40GB RAM, Mac OS 10.14, WeatherCat 3
Re: Meteotemplate
« Reply #262 on: November 03, 2016, 03:37:18 PM »
furl_open is now enabled! Metar and WU access now work. Not importing from WU yet, unless the WeatherCat historic data time problem can't be solved. I'll see what else I can break now... ;)

Slowly making progress. :)
Steve - Avon, Ohio, USA


CWOP: DW8454 • WU: KOHAVON11 • AWEKAS • CoCoRaHS • Skywarn •  UK Met Office
MWWN •PWSweather • WeatherBug • WeatherCloud • Twitter • Facebook

jachym

  • Strong Breeze
  • ***
  • Posts: 210
    • Meteotemplate
  • Station Details: WH1080
Re: Meteotemplate
« Reply #263 on: November 03, 2016, 03:57:53 PM »
OK and what if you try for the date, specifying this format:

Y/m/d\TH:i:s

that should work

Steve

  • Global Moderator
  • Storm
  • *****
  • Posts: 1344
    • DW8454
    • KOHAVON11
    • Avon Weather
  • Station Details: Davis Wireless VP2 Plus w/24 hr FARS, Leaf & Soil Station, 2017 27" iMac 3.8 GHz i5, 40GB RAM, Mac OS 10.14, WeatherCat 3
Re: Meteotemplate
« Reply #264 on: November 03, 2016, 04:11:09 PM »
Nope, same errors. I'd tried with the "T" in there before, but not the seconds. According to your linked reference, T is for time zone, and it is then expecting a different format.

Maybe I'll have to use the WU import after all.

Wait: Let me try something else. BRB...
Steve - Avon, Ohio, USA


CWOP: DW8454 • WU: KOHAVON11 • AWEKAS • CoCoRaHS • Skywarn •  UK Met Office
MWWN •PWSweather • WeatherBug • WeatherCloud • Twitter • Facebook

jachym

  • Strong Breeze
  • ***
  • Posts: 210
    • Meteotemplate
  • Station Details: WH1080
Re: Meteotemplate
« Reply #265 on: November 03, 2016, 04:14:20 PM »
Yes, but make sure that you put the backslash in front of the T, that tells the script it should ignore the "meaning of T" and take it literally as the letter T

Steve

  • Global Moderator
  • Storm
  • *****
  • Posts: 1344
    • DW8454
    • KOHAVON11
    • Avon Weather
  • Station Details: Davis Wireless VP2 Plus w/24 hr FARS, Leaf & Soil Station, 2017 27" iMac 3.8 GHz i5, 40GB RAM, Mac OS 10.14, WeatherCat 3
Re: Meteotemplate
« Reply #266 on: November 03, 2016, 04:33:17 PM »
I'd copy/pasted your format, so I had the backslash. WeatherCat's time has hyphens, so I tried Y-m-d\TH:i:s and Y-m-d\TH:i without the seconds and neither worked.

My idea was to open the file in Excel and save as a .CSV, which stripped the "T" and added a space. But it also did something else, because when I pushed the "Show Field Numbers" button, I got this:

Quote
Loading file...

Loading CSV/text file from 2011m.csv

File loaded successfully.
Header row in the file, skipping first line, using for field labels.


Starting to parse data....


Fatal error: Allowed memory size of 100663296 bytes exhausted (tried to allocate 8388608 bytes) in /home/smorris/public_html/avon-weather.com/template/admin/showFields.php on line 53

Even if I skipped showing the field numbers, because they were already set from the other attempt, it still had the same time errors.
Steve - Avon, Ohio, USA


CWOP: DW8454 • WU: KOHAVON11 • AWEKAS • CoCoRaHS • Skywarn •  UK Met Office
MWWN •PWSweather • WeatherBug • WeatherCloud • Twitter • Facebook

jachym

  • Strong Breeze
  • ***
  • Posts: 210
    • Meteotemplate
  • Station Details: WH1080
Re: Meteotemplate
« Reply #267 on: November 03, 2016, 04:37:42 PM »
OK, then the best thing would be if you could post here the URL of the file on your server and I will look myself

Steve

  • Global Moderator
  • Storm
  • *****
  • Posts: 1344
    • DW8454
    • KOHAVON11
    • Avon Weather
  • Station Details: Davis Wireless VP2 Plus w/24 hr FARS, Leaf & Soil Station, 2017 27" iMac 3.8 GHz i5, 40GB RAM, Mac OS 10.14, WeatherCat 3
Re: Meteotemplate
« Reply #268 on: November 03, 2016, 05:06:34 PM »
Yep, the file is located at http://www.avon-weather.com/template/admin/2011.csv

And the Excel modified one is located at http://www.avon-weather.com/template/admin/2011m.csv



You may not have seen the previous post, but I got the furl_open enabled. Thanks!

And my comments on UV and soil data in post #261, so that's cleared up.
Steve - Avon, Ohio, USA


CWOP: DW8454 • WU: KOHAVON11 • AWEKAS • CoCoRaHS • Skywarn •  UK Met Office
MWWN •PWSweather • WeatherBug • WeatherCloud • Twitter • Facebook

jachym

  • Strong Breeze
  • ***
  • Posts: 210
    • Meteotemplate
  • Station Details: WH1080
Re: Meteotemplate
« Reply #269 on: November 03, 2016, 05:12:07 PM »
OK, I looked at the first link, there is no "T", remember that you must open it in some text editor, not Excel, the raw format of the first two rows is:

Code: [Select]
Date, Conditions, Ext. Temperature,Dew Point,Wind Chill,Pressure,Wind Speed (10 Min. Average),Average Wind Direction,Precipitation Per Hour,Precipitation Total,Ext. Humidity,Int. Humidity,Int. Temperature,Wind Gust (10 Mins),Solar Radiation,UV Index,Leaf Wetness 1,Leaf Wetness 4,Soil Moisture 1,Soil Moisture 2,Soil Moisture 4,Leaf Temperature 1,Leaf Temperature 2,Soil Temperature 1,Soil Temperature 2,Soil Temperature 4,Monthly Rain,Annual Rain,Daily E.T.,Monthly E.T.,Annual E.T.
2011-09-06 22:00:00,Light Clouds,64.3,50.6,64.3,29.293,1.0,48,0.00,0.00,61,44.0,80.6,11.0,0,0.0,0,0,0,0,0,0.0,0.0,32.0,32.0,32.0,0.00,0.00,0.00,0.00,0.00,

So the date time should work as "one field" and then

Code: [Select]
Y-m-d H:i:s