Tag Info

Hot answers tagged

163

Ship hard drives across the ocean instead. At 11 Mbps with full utilization, you're looking at just shy of 90 days to transfer 10 TB. 11 Mbps = 1.375 MBps = 116.015 GB/day. 10240 GB / 116.015 GB/day = ~88.3 days.


24

I'd say rsync, at 11 MB/s you will look at 10-14 days and even if you get interrupted, rsync will easily start where it stopped last time. At 11 Mbps I'd ship the hard disks like suggested above :)


19

Yes, globbing expansion is alphabetical. From the Bash man page: Pathname Expansion After word splitting, unless the -f option has been set, bash scans each word for the characters *, ?, and [. If one of these characters appears, then the word is regarded as a pattern, and replaced with an alphabetically sorted list of file names ...


15

I currently support 220 servers up to 96 TB (totalling 2 PB or so), some in clusters of up to 240 TB, that my team built. Here are my advices : use a good, reliable hardware RAID controller : possible choices are 3Ware 96xx or 97xx, LSI 92xx, Areca 16xx, Adaptec 5xx5... Of course, with a Battery Backup Unit because power failures occur sometimes. use only ...


13

Rsync of course. At least you can continue at any time after break , and its without any pain.


10

Never underestimate the bandwidth of a station wagon full of tapes -- Trad. In your case, disks or tapes sent by courier, but the principle still applies. If you're not concerned about latency, this will be vastly cheaper than the network bandwidth to transfer 10TB of data in any reasonable length of time.


9

You should use rsync. It will compress the data and de-duplicate it before sending. It can also resume partial transfers, which is very important for any large transfers. Its likely it doesn't transfer 10TB; if its logs and text and such it could well be under 1TB; perhaps way below 1TB. There are tools that do a better job of compression than rsync and ...


7

It depends on what your purpose is. If you're looking for backups for disaster recovery (server exploded, datacentre burnt down, etc) then the short answer is they may not do backups at all. We have a client who deals in sensitive government data, and part of their mandate is that we are not permitted to do backups or backups onto removable media. We are ...


5

Honestly, I think $5k for the drives is a bit steep... but that's a whole other subject. The setup sounds sound-enough, but in the event of a drive-failure... having a single-volume that is 24tb will take FOREVER to rebuild. (ever tried to read 3tb of data split across 9 other disks?) It would be better to have smaller raid-sets and join them together to ...


5

A few thoughts here: 1) Don't do that. Seriously. Millions of tables would be a nightmare, and would likely cause far more problems than it solves. 2) If you really want to break the table up into multiple tables, you don't need to use that many. Depending on your hardware, I'd expect 50 million rows to be no problem, so you could divide up your data into ...


5

11Mbps? This is quite a limitation you have here. In your situation I would simply: Clone the data Compress it Rent servers on both ends with at least 10 times more bandwidth (in the same data centers or on your end in a data center near you). Transfer the files Apply the data to the new server. If you really have no solution to increase bandwidth... ...


4

I know this is already accepted but have you considered taking your disks to a data center/provider/host where you can get more bandwidth? It'll probably cost you some money but copying 10240Gb to backup disks and sending of will also cost both time and money (2 x money). Also you'll be sure your disks don't break in transport.


4

If a drive is formatted with a MBR then the maximum recognised size is 2.2TiB. Now this works slightly counter intuative when you use drives larger than 2.2TiB. A 3TB drive on a system supporting up to 2.2TiB will show up as 3TB-2.2.TiB, which is about 750GB. A 4TB (marketing TB, probably slightly less) minus 2.2TiB is about 1.6TiB. You might be able ...


4

Index maintenance should be done based on existing fragmentation, not blindly. With a clustered IDENTITY column, you should not have much to worry about. SQL Fool's defrag script will help. 200 million rows is not that much and not yet worth partitioning IMHO because of the query overhead, many table names requiring dynamic SQL etc. Unless you have a tiny ...


4

Use rsync.


3

MySQL can handle that size without serious problems (we store well over 100 million). What is the growth rate of the data? 1000 inserts a minute/hour/day? Will the growth be constant or will records be added in bulk (i.e. 100000 records in a batch)? Your usage is the key. If you are doing arbitrary queries with aggregate functions (sum, etc.) performance ...


3

Is rsync smart/efficient at detecting which files to copy/delete? Rsync is extremely efficient at detecting and updating files. Depending on how your files change, you might find a smaller number of large files are far easier to sync then lots of small files. Depending on what options you choose, on each run it is going to stat() every file on both ...


3

Should I split my large files? rsync is smart, but very large files can be dramatically less efficient to synchronize. Here's why: If only a part of a file changes, then rsync is smart enough to only send that part. But to figure out which part to send, it has to divide the file into logical chunks of X bytes, build checksums for each chunk (on both sides), ...


3

We generally use RAID5 or 6 for backup disks as it gives the best bang-for-buck once you ignore RAID 0 :-) so I'd go for that rather than JBODs One thing you might consider is buying your disks in separate batches rather than all 20 at once as if there is a manufacturing defect in a batch, they may fail at similar times. You also may wish to consider using ...


3

Like Chopper said, we don't do product and service recommendations - so everything I'm going to tell you is pretty generic. You're going to have to go to your thinking closet (possibly your crying corner after seeing some of the price tags) and figure out how to make this happen in your environment. If you're asking "How do I back up this server?", which ...


3

While I have to agree on the "ship it using harddrives" answer in this case, here a copy solution I use when I have to copy large amounts of files for the first time: While rsync is good to keep two data storages in sync, it introduces quite a bit of unnecessary overhead for the initial transfer. I figured that the fastest way is to tar which gets piped ...


2

I don't know if there are limits for particular DB engines but I've met with some Department of Defense admins who manage a MySQL system which imports 10TB every day and they are not experiencing this kind of problem. Rather than use macros I recommend writing a script to parse the input file and create insert statements from it, which can be fed straight ...


2

Have you considered IPoAC? A single pigeon may be able to carry tens of gigabytes of data in around an hour, which on an average bandwidth basis compares very favorably to current ADSL standards, even when accounting for lost drives.


2

Again, first suggestion is to ship the drives. Second suggestion is to use rsync to rsyncd, not over SSH. I've tried many things and it is usually the fastest. Remember to turn on compression. Also, look at increasing or decreasing the rsync buffer size to get the optimal transfer rate. It may also help to increase your MTU size. This only helps if routers ...


2

I'm pretty sure that /tmp and tmpfs are RAM-based, so decompressing them and storing them in RAM may be the reason that things grind to a halt. The OS starts swapping like crazy so you begin to really notice the performance hit. Must this operation occur in /tmp? Adding an actual answer: http://wiki.centos.org/TipsAndTricks/ExpandLV


2

Update: Since you've now confirmed that /tmp is not a symbolic link and that there are no mounts at /tmp. It's basically saying that / is filling up. How do you fix that? 2 options. I'd insert another hard disk and mount it at /tmp by inserting an appropriate entry in /etc/fstab. Or as a temporary measure move /tmp to another drive like /dev/sdb1 ...


2

Most of their data is stored on their own GFS filesystem, and GFS requires that there are at least three copies of every 64 MB block that makes a file (GFS uses 64 MB blocks). Having that said, I don't think they bother with backups, as they have at least three copies of every file, and blocks on failing node can be quickly replaced by simply replicating ...


2

One option that would fit well with your use-case, especially if your requirements keep growing, is an HSM (Hierarchical Storage Manager). I've installed several HSMs ranging up to 150TB of disk and 4PB of tape. The idea is that an HSM manages the lifecycle of data to reduce the overall cost of storage. Data is initially stored on disk but almost ...


2

First, 4×10⁷ columns with three integers in each is not toooo big. I suppose even Sqlite or less robust databases can handle this number of records. Now, if you ask about efficiency, everything depends on the usage of the database. All you said is how is stored the data, but more important is how is it used. Consider the following situations: The numbers ...


2

Splitting into this many tables is a nightmare, and not at all recommended. Among other complications, think of the complexity required to add a new user--do you have to dynamically create a new table? The answer is simply better indexing, specifically designed around the queries you are using. Since you haven't detailed those queries, I can't give you ...



Only top voted, non community-wiki answers of a minimum length are eligible