Author Topic: Data conversion - Problems accurately converting rain to millimeters  (Read 4210 times)

monymony

  • Strong Breeze
  • ***
  • Posts: 173
  • CoCoRaHS - MD-CC-16
    • EW3643
    • KMDELKTO10
    • Cecil County Maryland Weather
  • Station Details: Davis Vantage Pro 2
Hello Everyone,

I am running out of ideas and figured I would post to see anyone has any suggestions.  My Weatherlink data stored rain in inches.  WC of course stores rain as millmeters.  During my conversion I used the standard 1 inch = 25.4 mm for my conversion.  I have noticed that my converted totals in WC do not match my original tables.  For instance, my Oct 2007 rain total was 6.00in, but WC shows 5.87in as the total.  I did an export from WC of my Oct 2007 data and my total from the exported WC file is 5.97in even though from inside the UI the total is showing 5.87in.  Now I can account for three separate days where some rounding issue account for 0.01in for each occurrence (need to deal with that separately), but I still cannot explain the why this is off so much.  I have other months that vary too so this is just an example. 

I guess my first question is has anyone converted to WC that also had to convert from inches and if so did you have any issues?  I am trying to avoid making manual updates to my rain total, and really want to get my data imported properly to eliminate this issue.

Thanks!



elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Round off error in converter? (Re: Data conversion - Problems)
« Reply #1 on: August 24, 2013, 07:57:22 PM »
Dear MonyMony and WeatherCat fans,

I am running out of ideas and figured I would post to see anyone has any suggestions.  My Weatherlink data stored rain in inches.  WC of course stores rain as millmeters.  During my conversion I used the standard 1 inch = 25.4 mm for my conversion.  I have noticed that my converted totals in WC do not match my original tables.  For instance, my Oct 2007 rain total was 6.00in, but WC shows 5.87in as the total.

I've never had this problem precisely, but it seems to me that you might be having a problem with your converter software rounding off values incorrectly.  Not all computations are done using binary coded decimal and they will introduce errors even if you are using correct conversion formulas.

Sorry, not much help, but perhaps something you can explore.

Cheers, Edouard  [cheers1]

monymony

  • Strong Breeze
  • ***
  • Posts: 173
  • CoCoRaHS - MD-CC-16
    • EW3643
    • KMDELKTO10
    • Cecil County Maryland Weather
  • Station Details: Davis Vantage Pro 2
Re: Data conversion - Problems accurately converting rain to millimeters
« Reply #2 on: August 26, 2013, 01:23:42 PM »
Hi Edouard,

I am sure this is somehow related to a rounding issue.  I just cannot seem to trap where things are going astray.  Maybe I will get lucky and someone reading this will provide me with some inspiration.  I guess I can dream... right!

Jay



elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
What converter tool are you using? (Re: Data conversion)
« Reply #3 on: August 26, 2013, 09:37:58 PM »
Hi Jay and WeatherCat fans,

Sorry as grimly repeated around here, I've been busy with so I really race through reading postings here.

I am sure this is somehow related to a rounding issue.  I just cannot seem to trap where things are going astray.  Maybe I will get lucky and someone reading this will provide me with some inspiration.  I guess I can dream... right!

What sort of software tool are using to modify the data?  I'm sure it has been mentioned somewhere, but if you let me know how you are trying to do this, perhaps I can offer you some ideas.

Cheers, Edouard  [cheers1]

monymony

  • Strong Breeze
  • ***
  • Posts: 173
  • CoCoRaHS - MD-CC-16
    • EW3643
    • KMDELKTO10
    • Cecil County Maryland Weather
  • Station Details: Davis Vantage Pro 2
Re: Data conversion - Problems accurately converting rain to millimeters
« Reply #4 on: August 27, 2013, 12:54:00 PM »
Hi Edouard,

I have been using Microsoft Access and wrote my own conversion routine from Weatherlink to WC.  I will post details as soon as time permits but it will probably be a day or two. 

Jay



monymony

  • Strong Breeze
  • ***
  • Posts: 173
  • CoCoRaHS - MD-CC-16
    • EW3643
    • KMDELKTO10
    • Cecil County Maryland Weather
  • Station Details: Davis Vantage Pro 2
Re: Data conversion - Problems accurately converting rain to millimeters
« Reply #5 on: August 30, 2013, 04:18:24 AM »
Hi Edouard,

Here is a example of the problem I am facing.  For November 12, 2007 I recorded a total of 0.17in of rain with my Davis Vantage Pro 2 and Weatherlink.  I have pulled the data I had from my Weatherlink table and saved this as WeatherLink Data.CSV.  This file shows my time entries where rain was recorded, the value in inches, the converted value in millimeters and the running total for the day in millimeters. 

The file WeatherCat Data Export 2253 Sun Nov 11 2007 to 2251 Mon Nov 12 2007.csv is a one day export I did from WeatherCat (Not sure why the export started on 11/11/07 so I submitted a separate posting for this under bugs).  Anyway if you look at this file you will see the imported Weatherlink data in WC shows the total daily precipitation is correctly exported and shows a total of 0.17" for this day, but if I look at the stats for the day from the WC UI I see only 0.15"(see attached screenshot).

I am also attaching the data from my 11_WeatherCatData file to show the time covered that had precipitation as stored by WC. 

I hope this helps explain why I am going crazy trying to figure out why my rain totals are off.  Let me know if you need anything further.

Jay



elagache

  • Global Moderator
  • Storm
  • *****
  • Posts: 6494
    • DW3835
    • KCAORIND10
    • Canebas Weather
  • Station Details: Davis Vantage Pro-2, Mac mini (2018), macOS 10.14.3, WeatherCat 3
Sounds like you need Stu's help. (Re: Data conversion)
« Reply #6 on: August 30, 2013, 11:09:47 PM »
Hi Jay and WeatherCat fans,

Here is a example of the problem I am facing. 

. . . . .

Anyway if you look at this file you will see the imported Weatherlink data in WC shows the total daily precipitation is correctly exported and shows a total of 0.17" for this day, but if I look at the stats for the day from the WC UI I see only 0.15"(see attached screenshot).

Okay this sounds very different than I thought you were experiencing.  If the data is getting into WeatherCat and then exporting back out correctly, then your converter isn't the problem.  Sounds like either you have overlooked something about the WeatherCat import format, or there is a bug somewhere.  I assume that Stu is still plenty busy trying to catch up after his vacation, but once things settle down for him, you might need his help in cracking this one.

Sorry, that's all I can offer.

Cheers, Edouard

monymony

  • Strong Breeze
  • ***
  • Posts: 173
  • CoCoRaHS - MD-CC-16
    • EW3643
    • KMDELKTO10
    • Cecil County Maryland Weather
  • Station Details: Davis Vantage Pro 2
Re: Data conversion - Problems accurately converting rain to millimeters
« Reply #7 on: August 31, 2013, 02:52:45 AM »
Sounds like either you have overlooked something about the WeatherCat import format, or there is a bug somewhere.

I really hope it is something simple that i overlooked.

I will continue to play with this and see if I can learn more about where things are going wrong.

Jay



WCDev

  • WeatherCat Developer
  • Administrator
  • Storm
  • *****
  • Posts: 2911
    • CW9739
    • ISCOTLAN25
    • Trixology
  • Station Details: Main Station: Vantage Pro-2, 24hr fars, solar, soil/leaf station, extra temp stations, no U.V. WeatherLink IP.
Re: Data conversion - Problems accurately converting rain to millimeters
« Reply #8 on: August 31, 2013, 08:36:39 AM »
Hi Jay,
Make sure your rain starts at 0 (the P: field) for each day and be sure to remove the relevant .hrs file if the raw data file is modified  - this'll cause it to rebuild the hourly data from the modified raw data when it launches.

Cheers,
Stu.

monymony

  • Strong Breeze
  • ***
  • Posts: 173
  • CoCoRaHS - MD-CC-16
    • EW3643
    • KMDELKTO10
    • Cecil County Maryland Weather
  • Station Details: Davis Vantage Pro 2
Re: Data conversion - Problems accurately converting rain to millimeters
« Reply #9 on: September 02, 2013, 05:14:44 PM »
Hi Stu,

Thanks for getting back to me on this issue.  I wish the fix was that simple but I have confirmed that my P: field data is starting at 0 at the beginning of each day.  I also have become very familiar with deleting the relevant .hrs file.  I am guessing that the problem is occurring with how the .hrs file is generating but again that is just a guess.  I also think that this issue is related to a bug report (Recent Statistics for Precipitation don't make sense http://athena.trixology.com/index.php?topic=899.0) where during a power outage some of the raw data I recorded directly with WC did not properly record, but again not knowing all the magic that happens to generate the .hrs file and how exactly WC is arriving at the daily precip totals.

I will be happy to provide you more details or work with you directly to help pinpoint this issue.

Regards,

Jay



WCDev

  • WeatherCat Developer
  • Administrator
  • Storm
  • *****
  • Posts: 2911
    • CW9739
    • ISCOTLAN25
    • Trixology
  • Station Details: Main Station: Vantage Pro-2, 24hr fars, solar, soil/leaf station, extra temp stations, no U.V. WeatherLink IP.
Re: Data conversion - Problems accurately converting rain to millimeters
« Reply #10 on: September 07, 2013, 01:41:19 PM »
Hi Jay, in the file you posted the rain doesn't start at 0:

34790 t:120000 T:6.83 Ti:22.11 D:4.83 Pr:1026.8167 W:0 Wd:0 Wc:6.83 Wg:0 Ph:3.3 P:0.25 H:87 Hi:35 Pm:8.64 Py:283.21 V:4

it's 0.25.

Cheers,
Stu.

Edit: Just to clarify, the data should be:
34790 t:120000 T:6.83 Ti:22.11 D:4.83 Pr:1026.8167 W:0 Wd:0 Wc:6.83 Wg:0 Ph:3.3 P:0.00 H:87 Hi:35 Pm:8.64 Py:283.21 V:4
34791 t:120005 T:6.78 Ti:22.11 D:4.78 Pr:1026.9522 W:0 Wd:0 Wc:6.78 Wg:0 Ph:3.56 P:0.51 H:87 Hi:35 Pm:8.89 Py:283.46 V:4

monymony

  • Strong Breeze
  • ***
  • Posts: 173
  • CoCoRaHS - MD-CC-16
    • EW3643
    • KMDELKTO10
    • Cecil County Maryland Weather
  • Station Details: Davis Vantage Pro 2
Re: Data conversion - Problems accurately converting rain to millimeters
« Reply #11 on: September 17, 2013, 12:09:22 AM »
Hi Stu,

I have checked my data at at this time there was an active precipitation event going on.

34784 t:112330 T:7.22 Ti:22.17 D:4.17 Pr:1026.4442 W:0 Wd:0 Wc:7.22 Wg:0 Ph:1.52 P:0.76 H:81 Hi:35 Pm:7.62 Py:282.19 V:4
34785 t:112335 T:7.17 Ti:22.11 D:4.44 Pr:1026.4781 W:0 Wd:0 Wc:7.17 Wg:0 Ph:1.02 P:0.76 H:83 Hi:35 Pm:7.62 Py:282.19 V:4
34786 t:112340 T:7.06 Ti:22.11 D:4.56 Pr:1026.5458 W:0 Wd:0 Wc:7.06 Wg:0 Ph:1.02 P:1.02 H:84 Hi:35 Pm:7.87 Py:282.45 V:4
34787 t:112345 T:6.94 Ti:22.11 D:4.78 Pr:1026.4781 W:0 Wd:0 Wc:6.94 Wg:0 Ph:1.02 P:1.02 H:86 Hi:35 Pm:7.87 Py:282.45 V:4
34788 t:112350 T:6.94 Ti:22.11 D:4.78 Pr:1026.6474 W:0 Wd:0 Wc:6.94 Wg:0 Ph:1.27 P:1.27 H:86 Hi:35 Pm:8.13 Py:282.7 V:4
34789 t:112355 T:6.83 Ti:22.11 D:4.67 Pr:1026.8844 W:0 Wd:0 Wc:6.83 Wg:0 Ph:3.3 P:1.52 H:86 Hi:35 Pm:8.38 Py:282.96 V:4
34790 t:120000 T:6.83 Ti:22.11 D:4.83 Pr:1026.8167 W:0 Wd:0 Wc:6.83 Wg:0 Ph:3.3 P:0.25 H:87 Hi:35 Pm:8.64 Py:283.21 V:4
34791 t:120005 T:6.78 Ti:22.11 D:4.78 Pr:1026.9522 W:0 Wd:0 Wc:6.78 Wg:0 Ph:3.56 P:0.51 H:87 Hi:35 Pm:8.89 Py:283.46 V:4
34792 t:120010 T:6.78 Ti:22.11 D:4.94 Pr:1026.7151 W:0 Wd:0 Wc:6.78 Wg:0 Ph:5.08 P:0.76 H:88 Hi:35 Pm:9.14 Py:283.72 V:4

Does that mean that there should always be a P:0 event at the start of each day even when there is an active rain event happening?  My Davis Weatherlink data does have a rain amount recorded with my midnight reading which corresponds with the p:0.25 entry.  Hope this makes sense.

Thanks!

Jay