up vote 0 down vote favorite
1
share [g+] share [fb]

The problem:

Our new project aims to use a very...very large database. Basicly ~one million of new records have to be stored per month. This requirement gives to the enterprise application a very big pressure under the thousands or millions of users.

I'm aware of some techniques like horizontal/vertical partitioning and caching. But still the question remains: How to partition the data and where to implement the logic which handles the queries.

Also would be interesting if you guys can share your experiences about which database framework would be the best choice for that task.

So I'm looking for the best or common practices/architectures to handle millions of records of databases.

Thanks your time.

Update:

Hey guys. Sorry for the late response. Actually we are in a really early state of the planning phase. So basicly we want a good start point about databasehandling. Which framework or practice would be a best choise for us. Anyway this system will be a datawarehouse system which holds relations and data between doctors and patients. This means clinical informations or forms between patient and doctor or other large amount of data.

Update 2: Also would be interested in which database framework would be a best choice for partitioning: PostgreSQL, DB2 or Oracle's one? Give me your experince please.

link|improve this question
1  
Where is the typical size (bytes) of one of these rows so you can get to a Projected rows * row size (bytes) = Total bytes. From there you can covert that total and get an estimate how mnay MB or GB of storage. – jl. Jul 14 '11 at 13:03
1  
Is this an Online Transaction Processing or a Data Warehouse scenario? – jl. Jul 14 '11 at 13:04
2  
Accounting/financial applications might partion by month or some other time frame. More information on the kind of app supported would be useful. – jl. Jul 14 '11 at 13:05
I'm with @jl, if you looking at ~1,000,000 rows a month you're going to want to partition the data based on month and/or week of the year. This will make it easier to store the data but it's going take additional work for the devs to write the logic to add and query the data. – Scott Warren Jul 14 '11 at 13:43
3  
You know, ~1 million records a month isn't that big. And, without more details about your application and requirements, all anyone can do is blather on about something that might not benefit you at all. Tell us about your application - what inserts records, and who queries them? In very large databases, delayed convergence is an acceptable side-effect of partitioning. – mfinni Jul 14 '11 at 13:53
show 2 more comments
feedback

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
or
required, but never shown
discard

Browse other questions tagged or ask your own question.