I have 4TB database in company. As many different analytics require that full data the current database is not able to handle concurrent session from 100s of users.

Is it possible to have databse replication done on s3 so that issue is solved.

I mean

To start with

  1. First do the full export of 4TB on s3
  2. Now i want to do incremental updates like we do with databse replication on slaves. But here i want to have it on S3 instead of host. Even if i have to do it once or twice in day that is fine.

but i don't want to export full TB everyday

Is there any way

  • 3
    Sounds like what you need are multiple read replicas, not a cobbled-together replication solution using an object store. – EEAA Jun 1 '16 at 1:00
  • Have you looked into XFS? – earthmeLon Jun 1 '16 at 1:02
  • @EEAA thanks for that . I think is not possible to have structure like master slave . Is there any way i can do that other than slaves things . The compnay has decided to do like that so i have to find some way. – Karl Jun 1 '16 at 5:35
  • 1
    It is absolutely possible. Why would it not be? wiki.postgresql.org/wiki/… – EEAA Jun 1 '16 at 11:46
  • 1
    You might also benefit from a caching layer or more RAM. – ceejayoz Jun 1 '16 at 15:13

You cannot do this.

No RDBMS allow you to keep data in S3. S3 is just an internet object store, each access need you to download the WHOLE object using particular API. It is only practical if the object is small enough to load one time and fit into the memory, e.g. something as small as sqlite that allow you to load in-memory and little to no change to the object.

"Replication RDB Data changes to S3"? Even some RDBMS "invent" such crazy features to use S3 multipart upload, your company are going to get a HUGE S3 and AWS download bills on the massive number of PUT/LIST/GET request.

Prepare to build a Data-mart/data warehouse , build aggregation logic (trigger, proc to prepare some aggregation) for your operation RBDMS, because there is no silver bullets for massive analytical data request.

  • Thanks for your answer. Its not like i want to use s3 as main data store but as databse dumps . example for intial dump i will have 100 csv files for 100 tables in db . Then next step is to capture deltas may be daility to start and then apply those ndelats and then reupload to s3 . The company don't worry about costs , so thats not problem . is that possible – Karl Jun 1 '16 at 22:55
  • @Karl : then you should look for "transaction log backup", that you can copy to S3. Please rephrase and update your question including the title. Perhaps clarify what is your objective instead of jumping straight to some "solution". I still don't see how moving transaction update log to s3 will solved the "heavy load from 100 concurrent user " problem. – mootmoot Jun 2 '16 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.