Author Topic: Database Size - can it get too big?  (Read 2733 times)

EphraimGadsby

  • Gentle Breeze
  • **
  • Posts: 16
  • Station Details: Vantage Vue, Vantage ISS, WeatherLink Datalogger
Database Size - can it get too big?
« on: September 17, 2017, 02:47:20 PM »
Afternoon everyone,

Hopefully a quick one, as I have no problem just a simple query.

I have had WeatherCat up and running for 3 months now, and have been watching the database size growing with interest.
I have plenty of free space available on a dedicated WeatherCat mac (recycled Mini Late 2009) so I'm not worried about running out, but...

... Can the database ever get too big? I don't know the ins and outs of such things, as I'm not really an IT person. Just wondering whether these things are designed to grow and grow infinitely, or whether I'll start running into problems in 5-10 years time. I'd like to think I'm in this for the long haul, after all.

Regards,

Ephraim.

Weatheraardvark

  • Gale
  • ****
  • Posts: 397
  • I hear tornado sirens, lets go climb on the roof
    • C0005
    • KIADESMI1
    • Des Moines Weather and Climate
  • Station Details: Davis VP2 Plus, Fars, Extra sensors
Re: Database Size - can it get too big?
« Reply #1 on: September 17, 2017, 03:56:02 PM »
With large databases, I suggest backing them up in case the computer dies.   I had that happen when  my pC  boot drive  passed away. Fortunately I had been backing up my data to an external hard drive daily and that helped a lot when I went with Parallels with Win 10, I was able to bring over my data.

Right now, I have WC uploading to a 4TB external drive. I figure the backup data won't come close to that, but you do need a plan to store that data off site.  I periodically trust that Icloud will not fail, and I periodically upload to that.  I also have the 2 step verification that they suggested a while back.  I do a lot to protect the data.

Can the data base get too big, I am assuming it has some limits as all data bases have.   
Davis VP2 Plus; 24h  FARS; Extra Temp Humid sensor (2); Extra Temp Station (2);;Weatherlink IP;USB; MAC Ventura 13.3

; https://www.weatherlink.com/embeddablePage/show/8a7585dd06404bde81d5229b09f84ebb/summary

wurzelmac

  • Storm
  • *****
  • Posts: 1361
    • ITIROLPR2
    • Wetterstation Pr?graten am Gro?venediger
  • Station Details: Davis Vantage PRO2 Plus (24h fan aspirated, wireless) with UV/Solar | Weather Envoy Data Logger | Mac mini 2023 M2 8/256 | 1x Canon EOS 1300D | macOS Sonoma 14.2
Re: Database Size - can it get too big?
« Reply #2 on: September 17, 2017, 04:41:01 PM »
Hi,

I am collecting data since 2008 and so far (WeatherCatData is 3.08 GB at the very moment) there was not one single problem with the database of WeatherCat.  ThU5:-)
Reinhard


Weatheraardvark

  • Gale
  • ****
  • Posts: 397
  • I hear tornado sirens, lets go climb on the roof
    • C0005
    • KIADESMI1
    • Des Moines Weather and Climate
  • Station Details: Davis VP2 Plus, Fars, Extra sensors
Re: Database Size - can it get too big?
« Reply #3 on: September 17, 2017, 06:42:56 PM »
Still back it up to an external source.    Backing it up to the same hard drive is foolish.   IF the mac drive goes down, you lose everything.   I do use Time machine, and using CrashPlan to backup my computer elsewhere.   it makes sense.

Davis VP2 Plus; 24h  FARS; Extra Temp Humid sensor (2); Extra Temp Station (2);;Weatherlink IP;USB; MAC Ventura 13.3

; https://www.weatherlink.com/embeddablePage/show/8a7585dd06404bde81d5229b09f84ebb/summary

wurzelmac

  • Storm
  • *****
  • Posts: 1361
    • ITIROLPR2
    • Wetterstation Pr?graten am Gro?venediger
  • Station Details: Davis Vantage PRO2 Plus (24h fan aspirated, wireless) with UV/Solar | Weather Envoy Data Logger | Mac mini 2023 M2 8/256 | 1x Canon EOS 1300D | macOS Sonoma 14.2
Re: Database Size - can it get too big?
« Reply #4 on: September 17, 2017, 06:53:41 PM »
Still back it up to an external source.    Backing it up to the same hard drive is foolish.
To calm you down:
Backup to external Hard Drive and to TimeMachine.  [bounce]
Reinhard


EphraimGadsby

  • Gentle Breeze
  • **
  • Posts: 16
  • Station Details: Vantage Vue, Vantage ISS, WeatherLink Datalogger
Re: Database Size - can it get too big?
« Reply #5 on: September 17, 2017, 07:01:28 PM »
Don't worry, I'm backing by data up using WeatherCat's backup feature, and these are then copied into the cloud for safekeeping, plus Time Machine is in operation too.

One of the main reasons for asking, it that my upload speed out here is so chronically slow, that at some point uploading daily to the cloud won't be an option! Currently WeatherCatData is only 19.5Mb though, so simple enough.

Who knows, by the time it gets into hundreds of Mb, super fast broadband might even have reached rural England...

xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: Database Size - can it get too big?
« Reply #6 on: September 17, 2017, 08:35:05 PM »
Quote
recycled Mini Late 2009
Am I safe in assuming that the internal drive has been replaced? Drives will fail, the only question is when. I consider 5 years about the limit and most drive makers only offer 3 year guarantees, now.

Quote
upload speed out here is so chronically slow, that at some point uploading daily to the cloud won't be an option!
You might want to consider a plan that makes 'incremental' backups; it doesn't re-upload everything, just the parts that have been changed/added since the last BU (similar to Time Machine). That certainly can cut down on the upload file sizes and times.

There's a discussion comparing Dropbox and Google Cloud at Digital Trends that might be interesting. I don't use any 'cloud' backup service even though we have very high download speeds here. Upload speeds are usually much, much slower that down load speeds, anyway. I have two external drives connected to my mini, one for Time Machine and one for Carbon Copy Cloner. There are actually four identical drives which I try to rotate every week or so. Unfortunately, the un-mounted drives are still in the same room as the connected ones. I've not yet adopted the off-site storage philosophy! [blush] On the other hand, if the room/house 'disappears', I'll have more pressing problems than reporting my backyard Wx... [rolleyes2] :)
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: Database Size - can it get too big?
« Reply #7 on: September 17, 2017, 09:11:14 PM »
The Dev can certainly answer your question more directly, but from my limited collection of data, it would appear that your drive will die well before you run out of room for the files.

I only have one whole year of data (I started only in 2015). If we're talking about the WeatherCatData.cat and WeatherCatData.cat.hrs files, they are divided into "Months" directories. There are twelve of those for a full year. WOW! My 24 files for 2016 (my only full year so far) is only 94.1 MB. [The .dat file measures 7,933,952 physical bytes while the .cat.hrs needs only 663,552 physical bytes. ~8.1MB total/month] Even if you had 'only' a 500GB drive (assuming you can even find one today). That should contain at least 400+ years worth of data, allowing for file system overhead and a few hundred bad sectors. I'm not sure what device your children's descendant's, descendant's, descendant's, descendant's... might be using when that "drive" needs replacing. [lol]

The data appears to be simple text files. One 'record' per capture (I have WC set up to collect data every two minutes). Each line does have a 'record number', they start at zero and the one I looked at stops at 43199. Fortunately, the 32 bit barrier is already bypassed. ThU32:-)
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system


EphraimGadsby

  • Gentle Breeze
  • **
  • Posts: 16
  • Station Details: Vantage Vue, Vantage ISS, WeatherLink Datalogger
Re: Database Size - can it get too big?
« Reply #8 on: September 17, 2017, 10:25:44 PM »
All,

Thanks for the input so far. Just to be clear, I'm certainly not worried about running out of space to store the WeatherCatData folder. My query (though I realise I wasn't precise) was around performance and whether a database can just continue to grow without being unwieldy or slowing a system down - in the same way large spreadsheets with millions of rows can take forever to open.

With regard to my ageing hard drive and it's imminent doom: let it fail is my attitude. It's only there to run WeatherCat. So long as my data is backed up daily (which it is through writing the 20:00 backup file into the the cloud each day, and copying the 2 occurrences of it on the system drive via Time Machine) then it doesn't matter if my mac drive fails, does it?

Let me explain my thinking:

To replace my hard drive now means I'll have to power down my system, faff with installing a new one, then spend an afternoon installing the OS, plus WeatherCat, drivers etc. All this time I'm relying on the Davis's data logger to store the data which will eventually be incorporated into the database, once retrieved and the logger downloaded.

That is a significant disruption.

So, instead I'll leave the aged mac to soldier on and keep backing up as usual. When I come home from work one day and find it's died, I'll have to carry out the process above anyway... only I won't need to manually power down the system at least!

In reality, I have two macs. The elder of the two runs WeatherCat. When it dies, I'll simply repurpose my current "daily use" mac to run WeatherCat and buy a new one to replace my daily mac. At nearly 10 years old, this one owes me nothing.

When all's said and done, the Davis logs for 2 weeks without any input from me or a Mac. That's plenty of time to replace either a hard drive, or a whole computer.

Just so long as my backup is safe!

There you go... not quite the orthodox "belt and braces" approach most folks take as contingency, but this mac only runs WeatherCat, there is no other data on this system I care to keep, no photos, emails, music etc. All that is done by my newer mac. This one simply sits in the corner of a quiet room and plods on stoically with it's weather data farming. 

Hope you don't all think I'm too insane.

xairbusdriver

  • Storm
  • *****
  • Posts: 3126
    • EW7115 (E7115)
    • KTNGERMA20
    • Mid-South Weather
  • Station Details: Davis VP2 wireless + remote Anemometer/2014 Mac min - 10.15.7/WC 3.0.5
Re: Database Size - can it get too big?
« Reply #9 on: September 17, 2017, 10:38:51 PM »
"Insane"?! Sounds like a plan, even if different than some others. Maybe a plan developed by A. E. Newman? [lol]

I'm hoping your "stoic" mini lasts for at least another 5 months and 'Santa' brings you a new Mac! ThU32:-) That would enable you to have a 'retirement' ceremony for the mini instead of a 999 call! cmu:-)
THERE ARE TWO TYPES OF COUNTRIES
Those that use metric = #1 Measurement system
And the United States = The Banana system