New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
NoSQL Solution Suggestions!
Hi Gents and Gals,
I am currently looking for suggestions on a NoSQL solution for a very large, high number of rows database. My estimates are around 5 billion rows over the next up coming year. I have looked at couchdb, RethinkDB, and Mongodb, and I like all of them, but they each have there own small problems. Is there some others I should look into, or if you have a better implementation of one of the above I would gladly like to know.
Rough Table Format:
| id(system Specific) | IP address | Value | unix timestamp | unix timestamp 2 | application |
Thanks,
Mun
Comments
Redis (http://redis.io/) has been working well for me . Give a try
http://www.intersystems.com/cache/
This is used in healthcare and considered as the fastest in the world. Only drawback is its command based
It looks like you want to do time series data. There are many databases specifically architected around that.
The two names that come up a lot are http://graphite.wikidot.com/ and http://opentsdb.net/
I would also look into Cassandra and Couchbase.
P.S. NoSQL databases don't really have rows
.
If your using NoSQL don't really think relational. With that aside, take a look at Cassandra.
Correct, they don't have rows... sorry been looking at mysql too much as of late. In any case it will probably have 5 billion documents.
It really depends how you want to query the data.
Are you doing some log analysis? if so, try logstash, elasticsearch for adhoc query.
I think a traditional DB with sharding and archive should suffice your need btw.
Take a look at TokuMX. It's MongoDB fixed up.
IMHO, NoSQL stuffs will be a good choice if there is high variety in db structure and you expect it to be changed fast in the future. If you have a clear structure for your database, question yourself again if relational database is better fit. If it is only about volume, a server with a lot of RAM may help.
As funny as it may sound, my hat's in for MySQL (actually either Percona or MariaDB variants) with HandlerSocket for system of record. I like to actually backup my data.
But, depending on how you want to query it though, one of the other NoSQL will inevitably act as an intermediary. eg: Hadoop->HBase, Logstash->Elasticsearch, graph analysis->Neo4j
Actually I am currently using mysql (thus the rows incident). I will stick with it until it gets unbearable, but right now it is a champ with close to 10 million rows.
Mun
Instead of changing out the DB, why not just look into sharding/partitioning? That's how the big dogs do it.
What video card do you use for acceleration and encryption?
http://www.tokutek.com/products/tokudb-for-mysql/ the mysql replacement
Why not MySQL? I've been using it for several years and no regrets whatsoever
I am using MYSQL, just there is lots of horror stories of MYSQL and a few billion rows of data.
Mun
Twitter and Facebook (and many other big places) use MySQL for billions of rows. It's sharded, of course, but it works well.
They also use their own custom forked one tuned for their needs.
RethinkDB as the name suggests
MySQL when tuned a little works great
If you have the RAM for it, this sounds like a use case where Redis would do well
Redis' site has it listed as a data-structure server which IMO is a pretty good description of what it does; it's effectively a large one-dimensional array of a group fixed-length fields similar to what you might make in a C struct with a bunch of functions for operating on the data like a SQL DB would have.
MySQL with partitioning
rethinkDBs only down side that I have found is zero security. It just assumes everything is good and kosher and instantly begins sharding to the host
, otherwise I would use it. I really really liked it otherwise.
Mun