Questions tagged [large-data]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
-1 votes
2 answers
25 views

MySQL replcating huge changing data

Hi I have a master MySQL server with 950 gb of data and slave has stopped for some reason and there is a difference of 80gb of data. so I started to setup new slave . Since the data is changing and ...
user avatar
8 votes
2 answers
1k views

How to overwrite a very large hard drive (18TB) with random data using shell commands in Linux

I would like to overwrite a very large hard drive (18TB) with random bytes, to then check smart data for reallocated sectors or other errors. Since badblocks has some limitations on number of blocks ...
user avatar
0 votes
1 answer
179 views

GCSFuse v0.33.2 - Large objects (11GB) upload to GCSBucket Fails

Receiving below errors when we try to upload 11 GB file to bucket; also, filesystem is going into "????????" as we can't list the objects available in the bucket. This requires umount and ...
user avatar
1 vote
1 answer
827 views

Varnish crashes on bigger files

Overall, I serve very small files. Think about images and small videos. Caching these with Varnish is like a breeze and doesn't give me any issues. The problem I am having, is when I am downloading a ...
user avatar
  • 113
1 vote
1 answer
140 views

Multiple disks acting as one filesystem, while still allowing each disk to be accessed alone

I have a fairly large dataset (~160TB) that need to be delivered to a client every so often. This dataset consists of fairly large files, usually between 2Gb and 20Gb each. They exist on a BeeGFS ...
user avatar
  • 485
5 votes
3 answers
443 views

Can I validate a large file download piecemeal over http

I'm downloading a large file over http via wget, 1.2TB. The download takes about a week and has contained corruptions twice now (failed md5 check, which takes days to run by itself). Is there a good ...
user avatar
1 vote
2 answers
1k views

Moving very large (~100 Gb) from one server to the other

We're moving servers, and I need to transfer all the data from Server A to Server B. I have a tar.gz of about 100Gb that contains all the Server A files. I'd really like to avoid downloading the ...
user avatar
2 votes
1 answer
2k views

Reading 65k rows hangs PHP/MySQL

I am developing a PHP application that processes IP adresses. I am working with mysql tables containing up to 4 billion rows. I have a script that currently needs to fetch 65536 adresses from this ...
user avatar
1 vote
2 answers
4k views

Is it better to use rsync over SSHFS or over CIFS as remote repository, having no option for rsyncd?

I have a NAS that is only capable of CIFS, AFS, SSH (no rsyncd capabilities, no NFS). I have to backup very large files (vm images) and I usually set a rsync server on the backup device then I do a ...
user avatar
1 vote
1 answer
375 views

mysqld : multiple tmpdir & balancing

Our 1Tb tempdir can sometimes be completely used by mysqld, resulting in disk-full and query errors. This can be due to lots of mid-size queries, or couple of very big queries. We have a 5Tb raid ...
user avatar
0 votes
0 answers
120 views

Tips on Copying 3 Database files nearing 250 GB?

We are upgrading from SQL Workgroup 2005 to SQL Standard 2012. We have plenty of databases of smaller sizes that copied over with straight windows file copy in a reasonable time frame. We have one ...
user avatar
  • 11
1 vote
1 answer
572 views

Offsite backup solutions for 40TB of data/35 Million Files on single drive [closed]

I've got a server with 40TB of data and 35 million files. While the server itself has raid and all that jazz, I'm concerned about what would happen should something physically destroy the server (fire,...
user avatar
  • 21
4 votes
2 answers
5k views

4TB HGST SATA drive only shows 1.62 TB in Windows Server 2012

I'm using a Supermicro X9SRE-3F motherboard with the latest BIOS and 2x 4TB drives connected to the on-board SATA controller. If I set the BIOS to RAID and create a RAID 1 array, the array shows up ...
user avatar
4 votes
2 answers
502 views

Tomcat Denial of Service due to large packets

I had asked this question on ITSecurity, but I felt this question is better placed here. On a recent assesment, I found that sending large (>5 MB) requests to a tomcat server causes 100% CPU usage on ...
user avatar
  • 143
2 votes
2 answers
30k views

Increasing the size available to, or changing location of, /tmp

I am currently in charge of a server running Red Hat and a bioinformatics webapp that deals with huge files, some of which are over 100GB when uncompressed. The act of decompressing these files is ...
user avatar
  • 333
96 votes
11 answers
14k views

Transfer 10 TB of files from USA to UK datacenter

I am migrating my server from the USA to the UK from one data center to another. My host said I should be able to achieve 11 megabytes per second. The operating system is Windows Server 2008 at both ...
user avatar
  • 1,215
8 votes
8 answers
4k views

24TB RAID 6 configuration

I am in charge of a new website in a niche industry that stores lots of data (10+ TB per client, growing to 2 or 3 clients soon). We are considering ordering about $5000 worth of 3TB drives (10 in a ...
user avatar
  • 1,003
5 votes
2 answers
2k views

Is rsync a good candidate for failover implementation (very large dataset)?

I have a large a set of data (+100 GB) which can be stored into files. Most of the files would be in the 5k-50k range (80%), then 50k - 500k (15%) and >500k (5%). The maximum expected size of a file ...
user avatar
7 votes
2 answers
5k views

Import a 260GB csv file into MySQL

I have a really big csv ~260GB and I want to import it into MySQL. I use the following mysql script in MacOS DROP TABLE IF EXISTS tmp_catpath_5; create table tmp_catpath_5( a1 BIGINT(20),...
user avatar
  • 173
9 votes
3 answers
21k views

How do large companies backup their data?

How do companies who handle large amounts of data, for example Google or Facebook, backup everything? According to this Google platform article on Wikipedia, Google has an estimated 450,000+ servers ...
user avatar
1 vote
3 answers
2k views

Sync Large File Shares

What is the best way to sync large sets of files between to geographically separate locations so that both locations have the same file? We have a constantly changing 500gig file share in Location1 ...
user avatar
1 vote
6 answers
98 views

SQL Server 2000 tables

We currently have an SQL Server 2000 database with one table containing data for multiple users. The data is keyed by memberid which is an integer field. The table has a clustered index on memberid. ...
user avatar
  • 161
72 votes
3 answers
21k views

In Bash, are wildcard expansions guaranteed to be in order?

Is the expansion of a wildcard in Bash guaranteed to be in alphabetical order? I am forced to split a large file into 10 Mb pieces so that they can be be accepted by my Mercurial repository. So I was ...
user avatar
  • 867