My name is Mukesh, I worked with fairly large (or medium large) scale websites as my previous assignments – and now in LeaseWeb’s cloud team, as an innovation engineer. When I say large scale I’m talking about a website serving 300 million webpages per day (all rendered within a second), a website storing about half a billion photos & videos, a website with an active user base of ~10 million, a web application with 3000 servers …and so on!
We all know it takes a lot to keep sites like these running especially if the company has decided to run it on commodity hardware. Coming from this background, I’d like to dedicate my first blog post to the subject of scalable databases.
A friend of mine, marketing manager by profession, inspired by technology, asked me why are we using MySQL in knowing that it does not scale (or there is some special harry potter# magic?). He wanted to ask, from what reasons we have chosen MySQL? And are there any plans to move to another database?
Well the answer for later one is easy “No, we’re not planning to move to another database”. The former question however, can’t be answered in a single line.
#Talking of Harry Potter, what do you think about ‘The Deathly Hallows part -II’?
Think about Facebook – a well recognised social networking website. Facebook handles more than 25 billion page views per day; even they use MySQL.
The bottleneck is not MySQL (or any common database). Generally speaking, every database product in the market has the following characteristics to some extent:
- PERSISTENCE: Storage and (random) retrieval of data<
- CONCURRENCY: The ability to support multiple users simultaneously (lock granularity is often an issue here)
- DISTRIBUTION: Maintenance of relationships across multiple databases (support of locality of reference, data replication)
- INTEGRITY: Methods to ensure data is not lost or corrupted (features including automatic two-phase commit, use of dual log files, roll-forward recovery)
- SCALABILITY: Predictable performance as the number of users or the size of the database increase
This post deals about scalability, which we hear quite often when we talk about large systems/big data.
Data volume can be managed if you shard it. If you break the data on different servers at the application level, the scalability of MySQL is not such a big problem. Of course, you cannot make a JOIN with the data from different servers, but choosing a non-relational database doesn’t help either. There is no evidence that even Facebook uses (back in early 2008 its very own) Cassandra as primary storage, and it seems that the only things that’s needed there is a search for incoming messages.
I believe it’s a bad idea to risk your main base on new technology. It would be a disaster to lose or damage the database, and you may not be able to restore everything. Besides, if you’re not a developer of one of these newfangled databases and one of those few who actually use them in combat mode, you can only pray that the developer will fix bugs and issues with scalability as they become available.
In fact, you can go very far on a single MySQL without even caring about a partitioning data at the application level. While it’s easy to scale a server up on a bunch of kernels and tons of RAM, do not forget about replication. In addition, if the server is in front of the memcached layer (which simply scales), the only thing that your database cares is writes. For storing large objects, you can use S3 or any other distributed hash table. Until you are sure that you need to scale the base as it grows, do not shoulder the burden of making the database an order of magnitude more scalable than you need it.
- Use MySQL or other classic databases for important, persistent data.
- Use caching delivery mechanisms – or maybe even NoSQL – for fast delivery
- Wait until the dust settles, and the next generation, free-semantics relational database rises up.