Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

From your experience, what's the upper limit of rows in a MyISAM table can MySQL handle efficiently on a server with Q9650 CPU (4-core, 3.0G) and 8G RAM.

I currently have a table with 15 million rows. It's pretty fast. If the scale increases to 1 billion rows, do I need to partition it into 10 tables with 100 million rows each?

share|improve this question

migrated from stackoverflow.com Nov 1 '09 at 17:46

This question came from our site for professional and enthusiast programmers.

1  
Question: why MyISAM all of a sudden? Are you planning on using fulltext indexing? –  shylent Oct 31 '09 at 10:21
1  
ISAM is great for hugely massive non-relational data. And with a billion rows every millisecond counts. –  Mark Henderson Nov 1 '09 at 19:56

4 Answers 4

up vote 15 down vote accepted

I would not worry about application performance with 1 billion rows on a machine that can keep the indexes in memory. If you are serious about reaching 1 billion rows, you first have to do some math:

  • What is your record size, and multiply it by 1 billion?
  • Next, you need to compute the size of the indexes (more than one index, I'm guessing), and add that on.
  • Do you have transactional requirements that you want to have row-level locking for?
  • Is this an append-heavy table, or a read heavy table?

Next, move into your application uptime requirements.

  • How are you going to back up 1B rows?
  • How are you going to deal with a corrupt 1B row table?
  • How often will you need to run an OPTIMIZE TABLE?
  • How are you going to deal with doing a schema change against a 1B row table? (Adding an index on a 35 million row table on a 2gh dual core box with 2gb ram took me 45 minutes recently.)

I would worry more about the data lifecycle and data management of a multi-gigabyte table file of that magnitude before worrying about performance. With replication, you can make up a lot of the performance. Keeping the data sane and restoring from even small disasters (like corruption induced from bad ram) is more likely going to trouble you first.

I would also encourage you to take the table you have -- and add 1B rows of test data to it. This is extremely insightful to watch what happens to your system. Run some EXPLAINs on your queries against this new huge dataset. Time how long it takes to backup, and restore. You might need to adjust some requirements.

This is an interesting article about 1 billion rows in mysql.

share|improve this answer

Depends on the queries you're running. If you're doing SELECT * FROM table it's generally going to run a lot faster than a query with ten JOINs.

share|improve this answer
    
not necessarily! If that table was very wide.... –  Mitch Wheat Oct 31 '09 at 9:34
    
Which reinforces my point - it depends a lot on how that data is used and structured. :-) –  ceejayoz Oct 31 '09 at 9:46
    
It's also (I thought) fairly commonly known that SELECT * is quite a bit slower than SELECT col1,col2,col3,col4...coln –  Mark Henderson Nov 1 '09 at 19:57

depends on your hardware, data, what query you run and what you consider fast. for simple ("select * from table where foo='bla'") queries, the calculation is easy: if your query uses a index and that index fits into the filesystem buffer of your OS, it'll be fast. if it doesn't fit, the query runs slower (how much slower depends on the amount of data mysql has to read and the speed of your disks)

however, i would use a ACID compliant db like postgres with such tables, your dont want to repair a table with a billion rows

share|improve this answer

Just to add to some of the comments above, I've billion row table before on quad-xeons, although with 32Gb RAM, not just 8.

To make sure our performance is good the tables are simplified and normalised as much as possible to keep them thin, and then have just a couple of indexes on them. The main point of those tables, the really large ones, for me, was just to write down time-series data. Lots of writes, all in order, and very few reads. The reads that were necessary were always searching for specific times against another column or 2, and so the index could take care of that.

The tables held on the SAN were backed up automatically by SRDF and on the occasion things did go wrong (disk full, etc) it took about 4 hours to repair.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.