Author Topic: Meteotemplate  (Read 60287 times)

Steve

  • Global Moderator
  • Storm
  • *****
  • Posts: 1344
    • DW8454
    • KOHAVON11
    • Avon Weather
  • Station Details: Davis Wireless VP2 Plus w/24 hr FARS, Leaf & Soil Station, 2017 27" iMac 3.8 GHz i5, 40GB RAM, Mac OS 10.14, WeatherCat 3
Re: Meteotemplate
« Reply #270 on: November 03, 2016, 05:15:48 PM »
I've done that, and just did again on the Excel modified file. Same error. Maybe it doesn't like the seconds?


Quote
Starting to parse data.

Loading line 1...
Separating fields...
Decimal separator is a period, no need to do anything, can be parsed by PHP.


Parsing date and time...Date and time in a single field.
MySQL accepts date and time in format YYYY-MM-DD HH:MM, the parsed date and time is:



Parsing variables...

Temperature
Raw value:
Database temperature units: F
File temperature units: F
Same units, no need to do any conversion.
Final temperature value: 0.0 F

Humidity
Final humidity value: %

Pressure
Raw value:
Database pressure units: inhg
File pressure units: inhg
Same units, no need to do any conversion.
Final pressure value: 0.00 inhg

Wind speed
Raw value:
Database wind speed units: mph
File wind speed units: mph
Same units, no need to do any conversion.
Final wind speed value: 0.0 mph

Wind gust
Raw value:
Database wind speed units: mph
File wind gust units: mph
Same units, no need to do any conversion.
Final wind speed value: 0.0 mph

Precipitation
Raw value:
Database precipitation units: in
File precipitation units: in
Same units, no need to do any conversion.
Final precipitation value: 0.00 in

Rain rate
Raw value:
Database precipitation units: in/h
Rain rate is saved in the database in units per hour.
File rain rate units: in
Same units, no need to do any conversion.
Final rain rate value: 0.00 in/h

Wind direction
Final wind direction value: degrees

Solar sensor enabled in main settings, reading data...
Solar radiation
Final solar radiation value: W/m2


Parsing line 1 finished.
Checking validity of data based on your limits in Main settings...
Checking date and time...
There seems to be a problem with the date and time. Either it has not been parsed correctly or it is a date/time in the future! Check the format you specified and also the actual values.
Steve - Avon, Ohio, USA


CWOP: DW8454 • WU: KOHAVON11 • AWEKAS • CoCoRaHS • Skywarn •  UK Met Office
MWWN •PWSweather • WeatherBug • WeatherCloud • Twitter • Facebook

jachym

  • Strong Breeze
  • ***
  • Posts: 210
    • Meteotemplate
  • Station Details: WH1080
Re: Meteotemplate
« Reply #271 on: November 03, 2016, 05:17:30 PM »
OK, just a sec, Im gonna try it on my test server and see what is hapenning

jachym

  • Strong Breeze
  • ***
  • Posts: 210
    • Meteotemplate
  • Station Details: WH1080
Re: Meteotemplate
« Reply #272 on: November 03, 2016, 05:21:04 PM »
First link works fine for me:

Code: [Select]
Testing CSV/text file...

Errors will be highlighted in red...

Loading CSV/text file from http://www.avon-weather.com/template/admin/2011.csv

File loaded successfully.
Header row in the file, skipping first line.

The tested file has 33397 data sets. This import test will load only the first line.



Starting to parse data.

Loading line 1...
Separating fields...
Decimal separator is a period, no need to do anything, can be parsed by PHP.


Parsing date and time...Date and time in a single field.
MySQL accepts date and time in format YYYY-MM-DD HH:MM, the parsed date and time is: 2011-09-06 22:00



Parsing variables...

Check the date...

Look at the screenshot and use exactly the same thing

Steve

  • Global Moderator
  • Storm
  • *****
  • Posts: 1344
    • DW8454
    • KOHAVON11
    • Avon Weather
  • Station Details: Davis Wireless VP2 Plus w/24 hr FARS, Leaf & Soil Station, 2017 27" iMac 3.8 GHz i5, 40GB RAM, Mac OS 10.14, WeatherCat 3
Re: Meteotemplate
« Reply #273 on: November 03, 2016, 05:33:39 PM »
Wait, that worked, and is exactly in the format I tried before! Only difference was I had overwrite instead of skip!

Weird!!!

Thank you, and sorry for the seemingly endless confusion on my part. I'll import these now one year at a time, and then turn my CRON job back on.
Steve - Avon, Ohio, USA


CWOP: DW8454 • WU: KOHAVON11 • AWEKAS • CoCoRaHS • Skywarn •  UK Met Office
MWWN •PWSweather • WeatherBug • WeatherCloud • Twitter • Facebook

Steve

  • Global Moderator
  • Storm
  • *****
  • Posts: 1344
    • DW8454
    • KOHAVON11
    • Avon Weather
  • Station Details: Davis Wireless VP2 Plus w/24 hr FARS, Leaf & Soil Station, 2017 27" iMac 3.8 GHz i5, 40GB RAM, Mac OS 10.14, WeatherCat 3
Re: Meteotemplate
« Reply #274 on: November 03, 2016, 05:50:14 PM »
Grrrrrr! Stupid me!  [banghead] [banghead] [banghead] [banghead]

The 2011.csv file on my web space was not as exported by WeatherCat. After the problems yesterday, I had deleted it from the server, and then this morning, replaced it with the local copy.

Except the local copy had been modified, and I forgot when I uploaded it. Yesterday, I tried an experiment, where I opened the .csv file in TextWrangler, and did a find/replace for all of the "T" in the file. Then spent about an hour putting back the Ts that were supposed to be there, such as in the header, and several conditions.

Since we know that the modified file worked, I can do it to the other years. But since the import tool doesn't use "conditions", I can skip fixing them, and copy/paste a fixed header. I'll do that after lunch.

Every time I make progress, I do something to set myself behind...
Steve - Avon, Ohio, USA


CWOP: DW8454 • WU: KOHAVON11 • AWEKAS • CoCoRaHS • Skywarn •  UK Met Office
MWWN •PWSweather • WeatherBug • WeatherCloud • Twitter • Facebook

Steve

  • Global Moderator
  • Storm
  • *****
  • Posts: 1344
    • DW8454
    • KOHAVON11
    • Avon Weather
  • Station Details: Davis Wireless VP2 Plus w/24 hr FARS, Leaf & Soil Station, 2017 27" iMac 3.8 GHz i5, 40GB RAM, Mac OS 10.14, WeatherCat 3
Re: Meteotemplate
« Reply #275 on: November 03, 2016, 07:17:04 PM »
OK, I've got 2011 through 2015 .csv files all fixed and ready to import as separate year files. I'll do 2016 to the current date once they are all in.

Except it didn't work as expected...

The file for 2011 begins September 7, 2011 through the end of December 31, 2011.

But it only loaded through November 12, 2011 at 00:20:00 and then stopped. Nothing is significantly different in the next line other than a slightly higher pressure. No limits are exceeded, no extra comma or character.

Here was the error:
Quote
The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at webmaster@avon-weather.morrisgarage.com to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.

Thoughts? Suggestions? There is nothing in the error log near the time of this import, just from my previous errors.

[EDIT] I tried importing again and it added five more lines. Another try didn't change anything.
Steve - Avon, Ohio, USA


CWOP: DW8454 • WU: KOHAVON11 • AWEKAS • CoCoRaHS • Skywarn •  UK Met Office
MWWN •PWSweather • WeatherBug • WeatherCloud • Twitter • Facebook

Steve

  • Global Moderator
  • Storm
  • *****
  • Posts: 1344
    • DW8454
    • KOHAVON11
    • Avon Weather
  • Station Details: Davis Wireless VP2 Plus w/24 hr FARS, Leaf & Soil Station, 2017 27" iMac 3.8 GHz i5, 40GB RAM, Mac OS 10.14, WeatherCat 3
Re: Meteotemplate
« Reply #276 on: November 03, 2016, 07:33:46 PM »
Seems like I'm writing a book here. :)

It seems file size related. I deleted all the lines that were imported and it then saved the rest of 2011. I imported the 2012 file and it made it to March 5 at 18:07. So about ten weeks worth of data at five minute intervals. I'll continue importing a chunk at a time.
Steve - Avon, Ohio, USA


CWOP: DW8454 • WU: KOHAVON11 • AWEKAS • CoCoRaHS • Skywarn •  UK Met Office
MWWN •PWSweather • WeatherBug • WeatherCloud • Twitter • Facebook

wurzelmac

  • Storm
  • *****
  • Posts: 1246
    • ITIROLPR2
    • Wetterstation Prägraten am Großvenediger
  • Station Details: Davis Vantage PRO2 Plus (24h fan aspirated, wireless) with UV/Solar | Weather Envoy Data Logger | Mac mini Mid 2011 2,7 GHz Intel Core i7 with 12 GB of RAM | 1x Canon EOS 1300D | macOS 10.14.6 Mojave
Re: Meteotemplate
« Reply #277 on: November 03, 2016, 07:45:36 PM »
Steve,

my Wunderground import ends up somewhere in  January 2013 - I got really crazy why I wasn't able to add data before this date. Then I realized that I changed my station at this time and have had much more sensors. So if you have added sensors or have changed/upgraded your station in the meantime take that in care.

BTW, I began to import data from recent to ancient - first 2016, then 2015, ... , till I ended up in 2008.

Good luck - if you are running the website with all data in there it is simply crazy what you can do with it. Fingers crossed!
Reinhard


Steve

  • Global Moderator
  • Storm
  • *****
  • Posts: 1344
    • DW8454
    • KOHAVON11
    • Avon Weather
  • Station Details: Davis Wireless VP2 Plus w/24 hr FARS, Leaf & Soil Station, 2017 27" iMac 3.8 GHz i5, 40GB RAM, Mac OS 10.14, WeatherCat 3
Re: Meteotemplate
« Reply #278 on: November 03, 2016, 09:20:41 PM »
Whew!!! All data has been uploaded to the database from September 7, 2011 until this minute in five minute increments. Each year had to be uploaded in six chunks of about 19,000 lines each. Plus the CRON job is uploading as it is supposed to so the weather station shows as live!

That was a big chunk of time to figure out the problems and edit/upload the files. I'm not sure how that compares to doing it via Weather Underground.

 [bounce]

Now, on to the next item...

Thank you Jaclyn and Reinhard!!!!!
Steve - Avon, Ohio, USA


CWOP: DW8454 • WU: KOHAVON11 • AWEKAS • CoCoRaHS • Skywarn •  UK Met Office
MWWN •PWSweather • WeatherBug • WeatherCloud • Twitter • Facebook

Blicj11

  • Storm
  • *****
  • Posts: 3275
    • EW3808
    • KUTHEBER6
    • Timber Lakes Utah
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP™ Data Logger | iMac (late 2013), 3.5 GHz Intel Core i7, 24 GB RAM, macOS Mojave | Sharx SCNC2900 Webcam | Supportive Wife
Re: Meteotemplate
« Reply #279 on: November 03, 2016, 09:43:00 PM »
So Steve, what do you do in your spare time?  ;)
Blick


jachym

  • Strong Breeze
  • ***
  • Posts: 210
    • Meteotemplate
  • Station Details: WH1080
Re: Meteotemplate
« Reply #280 on: November 03, 2016, 10:40:02 PM »
LOL, yes this is down to the maximum execution time for PHP set on the server and the speed at which it is able to process the data. So you must do it in chunks.

mcrossley

  • Strong Breeze
  • ***
  • Posts: 242
  • Sorry, I'm not a WeatherCat user!
    • ICHESHIR25
    • Wilmslow Astro Weather
  • Station Details: WH1090
Re: Meteotemplate
« Reply #281 on: November 04, 2016, 10:08:51 AM »
You could try adding a function to "set_time_limit(seconds);" at the top of the script?
Mark

jachym

  • Strong Breeze
  • ***
  • Posts: 210
    • Meteotemplate
  • Station Details: WH1080
Re: Meteotemplate
« Reply #282 on: November 04, 2016, 10:20:54 AM »
In 99% of cases, the maximum_execution time for PHP cannot be changed unless you are running your own server. This is because 99% of web hosting providers (including mine) block changing certain parameters from the php.ini. These usually are those parameters that directly affect speed etc. In my case, I for example cannot change execution time, maximum file upload size, maximum number of PHP scripts executed simultaneously etc.
In other words - parameters that affect server load. And the higher hosting plan, which costs twice as much, has these set to much higher values. Same goes for the memory limit, time limit etc. So even if you run the

set_ini....

command its not going to do anything, it will be overridden by that global server setup. This is only possible if you are using your own server and for some parameters that do not direclty affect the server load.

mcrossley

  • Strong Breeze
  • ***
  • Posts: 242
  • Sorry, I'm not a WeatherCat user!
    • ICHESHIR25
    • Wilmslow Astro Weather
  • Station Details: WH1090
Re: Meteotemplate
« Reply #283 on: November 04, 2016, 11:38:55 AM »
That's why I added a question mark ;) Is it worth adding a timeout anyway, if it doesn't work no harm done, if it does then...

Safe mode has to be off for this function to have effect as well. Not sure what a good value would be as I have no idea how long these scripts run.
Mark

jachym

  • Strong Breeze
  • ***
  • Posts: 210
    • Meteotemplate
  • Station Details: WH1080
Re: Meteotemplate
« Reply #284 on: November 04, 2016, 11:51:04 AM »
The problem is that once the script times out, you can no longer do anything.

So what I did for the WU import for example is that after each day is uploaded, it overwrites the cache file and in that file it just saves "I successfully saved day XY". Then the last day it is able to execute, but not finish, will not even be in this cache file, so I included instructions you must then go to this cache text file where you will see a date and that date will be the last fully uploaded day and you have to execute the import again starting from this day. And depending on server speed you have to repeat this procedure untill you import all the data.

With WU it is particularly problematic because they only give you data on a daily basis, so you have to load each day on a separate page, so importing a yr would mean separately loading 365 pages and WU servers are also not the fastest, in my experience, you can import approximately 2 weeks to 3 months in one go, depending on your server speed.