Author Topic: Data Import - Syntax .cat?  (Read 5305 times)

jonn68

  • Calm
  • *
  • Posts: 8
    • EW9588
    • IBADENWR148
    • Wetter Ebersbach-Weiler
  • Station Details: Davis Vantage Pro
Data Import - Syntax .cat?
« on: July 29, 2016, 04:03:41 PM »
Since for a few days i am using weathercat, it's a nice software. My weather station is a Davis Vantage Pro with a Raspberry(Meteohub) for logging the data. All two days i made a collection from meteohub, this works perfectly.

Now i had a lot of data from 2007(csv Format), i am reading here i can put the historic data in the cat file. My Problem is the syntax for preparing the csv file.

Which syntax i need for the cat file?

Thank you

Jonn

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Need to make your own importer (Re: Data Import - Syntax .cat?)
« Reply #1 on: July 29, 2016, 11:28:20 PM »
Dear John and WeatherCat faithful,

Since for a few days i am using weathercat, it's a nice software. My weather station is a Davis Vantage Pro with a Raspberry(Meteohub) for logging the data. All two days i made a collection from meteohub, this works perfectly.

I'm glad you found WeatherCat and like the software!  :)  Most of us predate the Meteohub technology, but there is at least one other WeatherCat user who has this sort of setup.

Now i had a lot of data from 2007(csv Format), i am reading here i can put the historic data in the cat file. My Problem is the syntax for preparing the csv file.

Which syntax i need for the cat file?

Unfortunately, WeatherCat doesn't have any import tools.  However, if you are handy with a bit of scripting you can probably make an importer of your own.  The data format for the WeatherCat .cat can be found at the top of the files themselves.   I suppose you already know that you'll find the WeatherCat data files in this directory:

/Users/elagache/Library/WeatherCatData/Location1/2016

Make a copy of the file 7_WeatherCatData.cat and open it with any text editor.  TextEdit will due although you would be better off with a dedicated text editor like TextWrangler.  At the top of the file you find a key with all the values and just looking that content will allow you quickly figure out how the data is organized.  If you have data before 2016, you'll need to put the data in a separate folder with the number of that year just like you see the 2016 folder.

You don't have to create the 7_WeatherCatData.cat.hrs files.  WeatherCat will automatically index any new files it finds when it first launches.

Feel free to ask us any additional questions you might have and let us know if you decide to "roll your own" importer.  There might be others who would be interested should you succeed!

Cheers, Edouard  [cheers1]

Blicj11

  • Storm
  • *****
  • Posts: 3941
    • EW3808
    • KUTHEBER6
    • Timber Lakes Weather
  • Station Details: Davis Vantage Pro2 Plus | WeatherLinkIP Data Logger | iMac (2019), 3.6 GHz Intel Core i9, 40 GB RAM, macOS Ventura 13.6 | Sharx SCNC2900 Webcam | WeatherCat 3.3 | Supportive Wife
Re: Data Import - Syntax .cat?
« Reply #2 on: July 29, 2016, 11:43:03 PM »
Jonn:

I don't have anything to add to Edouard's response but want to welcome you to WeatherCat and the Forum!
Blick


jonn68

  • Calm
  • *
  • Posts: 8
    • EW9588
    • IBADENWR148
    • Wetter Ebersbach-Weiler
  • Station Details: Davis Vantage Pro
Re: Data Import - Syntax .cat?
« Reply #3 on: July 31, 2016, 07:07:23 AM »
Thank you, now i am testing for the best solution :-).


jonn68

  • Calm
  • *
  • Posts: 8
    • EW9588
    • IBADENWR148
    • Wetter Ebersbach-Weiler
  • Station Details: Davis Vantage Pro
Re: Data Import - Syntax .cat?
« Reply #4 on: August 01, 2016, 07:39:39 PM »
The data from 2016 are imported, i do it with Libre Office and Tedit, duration for every month, 10 minutes. Now i am writing a bash programm for the another years.

I am writing a manual for WsWin and Meteohub if i finished :-)

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Would you be willing to share? (Re: Data Import - Syntax .cat?)
« Reply #5 on: August 01, 2016, 10:40:02 PM »
Dear John and WeatherCat scripters,

The data from 2016 are imported, i do it with Libre Office and Tedit, duration for every month, 10 minutes. Now i am writing a bash programm for the another years.

Best of luck with your endeavor!  While I was working at San Diego Supercomputer Center, I ended up writing quite a few Perl scripts to convert data from one format to another.  It is a bit tedious, but UNIX does provide quite a few tools for this sort of work.  If you succeed in creating your bash script, would you be willing to share it with others?  The problem of converting data does come up periodically.

Thanks for sharing your experiences with us!

Cheers, Edouard  [cheers1]

jonn68

  • Calm
  • *
  • Posts: 8
    • EW9588
    • IBADENWR148
    • Wetter Ebersbach-Weiler
  • Station Details: Davis Vantage Pro
Re: Data Import - Syntax .cat?
« Reply #6 on: August 02, 2016, 05:49:52 PM »
Now i had a problem:

Rain/July:
Weathercat:29.0mm
Meterhub: 51.0mm
7_WeatherCatDat.cat analyzed in LibreOffice: 51.0mm
The Davis console has the same value like meteohub.

for another month the same problem...

Why the difference in the amount of rain?


elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Re: Data Import - Syntax .cat?
« Reply #7 on: August 02, 2016, 10:23:39 PM »
Dear jonn68 and WeatherCat troubleshooters,

Now i had a problem:

Rain/July:
Weathercat:29.0mm
Meterhub: 51.0mm
7_WeatherCatDat.cat analyzed in LibreOffice: 51.0mm
The Davis console has the same value like meteohub.

for another month the same problem...

Why the difference in the amount of rain?

That is an unfortunate situation that does happen from time to time.  Were you uploading data to any service like Weather Underground?  If you could get another source for the rainfall perhaps that would represent a "tie-breaker."

Cheers, Edouard

jonn68

  • Calm
  • *
  • Posts: 8
    • EW9588
    • IBADENWR148
    • Wetter Ebersbach-Weiler
  • Station Details: Davis Vantage Pro
Re: Data Import - Syntax .cat?
« Reply #8 on: August 06, 2016, 08:42:51 PM »
Dear jonn68 and WeatherCat troubleshooters,

Now i had a problem:

Rain/July:
Weathercat:29.0mm
Meterhub: 51.0mm
7_WeatherCatDat.cat analyzed in LibreOffice: 51.0mm
The Davis console has the same value like meteohub.

for another month the same problem...

Why the difference in the amount of rain?

That is an unfortunate situation that does happen from time to time.  Were you uploading data to any service like Weather Underground?  If you could get another source for the rainfall perhaps that would represent a "tie-breaker."

Cheers, Edouard

I am uploading to Wunderground, how i can use the rainfall data from Wunderground?

Jonn

elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Which number does Wunderground have? (Re: Data Import - Syntax .cat?)
« Reply #9 on: August 06, 2016, 11:08:21 PM »
Dear Jonn and WeatherCat data accountants,

Now i had a problem:

. . . . .

Why the difference in the amount of rain?
I am uploading to Wunderground, how i can use the rainfall data from Wunderground?

I was hoping that you would see that Weather Underground would display one of the two rainfall numbers you had seen for the months in conflict.  If you see the same number in two places, I would conclude that's the correct number.

Let us know what you uncover, . . . . .

Cheers, Edouard

jonn68

  • Calm
  • *
  • Posts: 8
    • EW9588
    • IBADENWR148
    • Wetter Ebersbach-Weiler
  • Station Details: Davis Vantage Pro
Re: Data Import - Syntax .cat?
« Reply #10 on: August 07, 2016, 06:47:26 AM »
Ok, i misunderstood you. The same amount for rainfall in:

Wunderground, Meteohub, weathercat file analyzed in Libre Office and finally i imported the file in my office computer with WsWin and Windows 7, the same result.



elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
What does your console report? (Re: Data Import - Syntax .cat?)
« Reply #11 on: August 07, 2016, 11:08:55 PM »
Dear Jonn and WeatherCat troubleshooters,

Wunderground, Meteohub, weathercat file analyzed in Libre Office and finally i imported the file in my office computer with WsWin and Windows 7, the same result.

What is the value for rainfall reported by your Davis console?  WeatherCat simply reads the value from your console for these rainfall totals.  I've accidentally "zapped" my console and that caused it to lose some of the rainfall earlier that month.  You might have a similar problem.  It is possible to correct the values in the console so that everything matches up and WeatherCat will report the correct values once more.

I hope that helps.  Let us know what you discover.

Cheers, Edouard

WCDev

  • WeatherCat Developer
  • Administrator
  • Storm
  • *****
  • Posts: 2911
    • CW9739
    • ISCOTLAN25
    • Trixology
  • Station Details: Main Station: Vantage Pro-2, 24hr fars, solar, soil/leaf station, extra temp stations, no U.V. WeatherLink IP.
Re: Data Import - Syntax .cat?
« Reply #12 on: August 08, 2016, 03:59:37 PM »
If there's an error in the rainfall for the month in WeatherCat, then it'll be an error in the data file.

Problems I've seen in the past when people as importing data include:
Wrong line endings in the file
Wrong units being used (rainfall is in mm's)
Not enough data - you need to aim for at least half hour between samples, preferably more :)

jonn68

  • Calm
  • *
  • Posts: 8
    • EW9588
    • IBADENWR148
    • Wetter Ebersbach-Weiler
  • Station Details: Davis Vantage Pro
Re: Data Import - Syntax .cat?
« Reply #13 on: August 09, 2016, 03:57:54 PM »
Today:

NOAA Report Rainfall: 1.02mm.

Weathercat Data Editor rainfall: 0.52mm

Weathercat Controll unit: 0.52mm


Cheers

WCDev

  • WeatherCat Developer
  • Administrator
  • Storm
  • *****
  • Posts: 2911
    • CW9739
    • ISCOTLAN25
    • Trixology
  • Station Details: Main Station: Vantage Pro-2, 24hr fars, solar, soil/leaf station, extra temp stations, no U.V. WeatherLink IP.
Re: Data Import - Syntax .cat?
« Reply #14 on: August 09, 2016, 08:46:01 PM »
The value displayed in the controller window is the daily rain value from your console, the value in the NOAA report is the calculated value for the day.

Where there is a difference, it's normally an indication that sometime during the day your daily rain fell to 0 then went back to its previous value.

WeatherCat calculates rain over a period by adding up the differences between samples. Your daily rain should be slowly increasing throughout the day as it rains. If it drops, then comes back up, this will effectively double your rain - for example, if the rain is:
0mm, 2.0mm, 2.0mm, 2.0mm, 0mm, 2.0mm
across that period WeatherCat will see 4mm's.

To correct this, open the data editor and scroll down the raw rain column looking for a drop in the value to 0 followed by it going back up in subsequent samples. Edit the 0 to the previous sample value then reboot WeatherCat - they will then agree.

The last one of these I saw, the problem was occurring because the computer was going to sleep, which was causing a valid record to come in but with a rainfall of 0, which then picked back up on subsequent samples.