Tagged Questions

The tag has no wiki summary.

learn more… | top users | synonyms

2
votes
2answers
66 views

Fastest way to move a large file from an ESXi host to another

I have two VMWare ESXI 5 hosts and I want to move a 2 TB .vmdk file from one to another. I tried downloading the file via vSphere Client to upload it to the other host via vSphere Client, but it seems ...
2
votes
2answers
264 views

Best parameters for ext4 filesystem for storing virtualbox disk images

I have partition, which I'm going to use for keeping VirtualBox disk images (VDI, or may be VMDK). They are fixed-size (not dynamically-allocated) images. There are no other files/directories on ...
3
votes
1answer
29 views

How to show the hexvalues of a line of a large file in Unix?

I am importing data from a large text file into a database and am getting an error on line X of the file. If I look at the line with the less viewer I do not see anything strange, because, most ...
0
votes
1answer
985 views

Apache places uploaded files in /tmp. Can I change that to another directory?

Is there a configuration that changes the directory where apache web server temporarily places uploaded files? I have access to httpd/conf.d I'm on a machine where /tmp is very size constrained and ...
0
votes
0answers
68 views

Tips on Copying 3 Database files nearing 250 GB?

We are upgrading from SQL Workgroup 2005 to SQL Standard 2012. We have plenty of databases of smaller sizes that copied over with straight windows file copy in a reasonable time frame. We have one ...
7
votes
4answers
3k views

Rsync show progress for individual file

I'm using rsync to upload/download a pretty large file (20Gb). I'm aware of --progress option, but what it does in my case is just showing me the result at the end. My problem is that during some ...
1
vote
1answer
4k views

IIS7 - how to configure server to serve large (GBs) files?

I wanted to download some backups (about 4..5Gb in size) via HTTP and was surprised that IIS7 didn't allowed me to do so. The attempt was seemingly blocked - got a 404 page. Where should I go in the ...
21
votes
4answers
1k views

Pushing Large Files to 500+ Computers

I work with a team to manage 500-600 rented Windows 7 computers for an annual conference. We have a large amount of data that needs to be synced to these computers, up to 1 TiB. The computers are ...
17
votes
6answers
3k views

Which filesystem for large LVM of disks (8 TB)?

I have a Linux server with many 2 TB disks, all currently in a LVM resulting in about 10 TB of space. I use all this space on an ext4 partition, and currently have about 8,8 TB of data. Problem is, I ...
0
votes
1answer
315 views

resumable tar over ssh

I'd like to use tar cf - | ssh user@hostname tar xf - to copy a very large set of files. The only problem is that it will be over a very flakey VPN link. Is it possible to somehow set up a resumable ...
7
votes
2answers
571 views

Sorting large binary files

Is there Unix utility for sorting large files containing fixed length binary records? In other words, I'm looking for something like sort(1) but for binary files with fixed length records. I could ...
5
votes
1answer
7k views

Optimizing rsync for transferring huge files quickly

This is what I need: Rsync daemon running on source server. Destination server does a rsync pull as below: rsync -ptv rsync://source_ip:document/source_path/*.abc destination_path/ Huge binary files ...
13
votes
3answers
2k views

Building a 1,000M row MySQL table

This question is reposted from Stack Overflow based on a suggestion in the comments, apologies for the duplication. Questions Question 1: as the size of the database table gets larger how can I tune ...
0
votes
2answers
109 views

Storage setup for large files

I need to store over 200TB of data (all types, biggest being video files) and be able to access it over a local network. The files will be accessed for editing or searches. I don't need versioning, ...
0
votes
3answers
483 views

How to upload a large file in pieces to my server?

How can upload a large SQL dump file it's 1GB, is there any software that uploads the file and resumes it later or just cut it into pieces and then join them back on the server ? because ftp upload is ...
7
votes
2answers
485 views

Unix sort for partially ordered data sets

So I have an enormously large file (around 10GB) and need to sort it, just like in using 'sort' utility, but kindof more effectively. Problem is, that I don't have memory, CPU power, time, nor free ...
1
vote
3answers
1k views

500 Internal Server Error when uploading large files to MediaWiki

When I upload a large (>245 Mb) file to my MediaWiki wiki (using Upload file, same as for images), I see: Internal Server Error The server encountered an internal error or misconfiguration ...
4
votes
2answers
1k views

Import a 260GB csv file into MySQL

I have a really big csv ~260GB and I want to import it into MySQL. I use the following mysql script in MacOS DROP TABLE IF EXISTS tmp_catpath_5; create table tmp_catpath_5( a1 ...
1
vote
1answer
559 views

/ partition as xfs in Centos 5 ? , any advantage for apps using large files?

I have been investigating a bit previously to install Centos 5 in a box that it will use applications managing large files (i think more than 300 MB minimal). I was thinking in , as / has a lot of ...
0
votes
1answer
157 views

Sharing large files throughout an organization

We have non-technical staff members who want to collaborate on large files (~500MB). [I believe the files are SMART Notebook files with embedded videos and flash components]. I'm not quite sure what ...
2
votes
3answers
3k views

Free space on ZFS file system unexpectedly missing

Going to completely rephrase this question since it's still an outstanding production issue several months later. I have a FreeNAS 0.7.2 box, based on FreeBSD 7.3-RELEASE-p1, running ZFS with 4x1TB ...
0
votes
2answers
659 views

How to delete last n lines of a file without copying?

Size of log file: 10 GB Free space in partition: 6 GB Need to split log file into smaller pieces, and gzip those. But there's not enough room to run something like split(1), which leaves the ...
4
votes
1answer
723 views

MySQL and large pages/hugetlb performance gain in numbers?

MySQL has large page/hugetlb support for InnoDB. And there's plenty of posts (example) on the topic. But does anyone have examples of what performance changes they saw ? I have seen other database ...
1
vote
3answers
1k views

Sync Large File Shares

What is the best way to sync large sets of files between to geographically separate locations so that both locations have the same file? We have a constantly changing 500gig file share in Location1 ...
1
vote
6answers
265 views

Distribute terrabyte files to the public from web server

We need to set up a website which makes two or three large files publicly available - the files will be 1 or 2 terrabytes each. Although they will be public, in practise I expect only a relatively ...
1
vote
5answers
933 views

How to backup database dumps which only minor changes between the backups?

I have a posgresql dumpfile every night and i like to keep every one of them. Each is 14 GB in size, so my backupdrive will be full soon. The difference from one day to another are only some 100MB. ...
2
votes
8answers
3k views

Is there a Linux equivalent of Yousendit? [closed]

We want to get functionality similar to Yousendit or one of the other services that allows end users to upload large files. But we have a few Linux servers and want to use these instead of a third ...
14
votes
3answers
2k views

In Bash, are wildcard expansions guaranteed to be in order?

Is the expansion of a wildcard in Bash guaranteed to be in alphabetical order? I am forced to split a large file into 10 Mb pieces so that they can be be accepted by my Mercurial repository. So I was ...
2
votes
1answer
94 views

Perforce Restore From Multiple Checkpoint Files?

I am working with a very large (~11GB) checkpoint file and trying to do a -jr (journal restore) operation. About half way through the file, I'm hitting an entry which causes an error to occur. I'm ...
2
votes
5answers
1k views

Storing large amounts of data

Say I'd like to store 40 terabyte of data for frequent access, mainly media files, through a web interface. Should I build 4 computers with 10 terabyte each? Any limitations I should know about? What ...
3
votes
6answers
3k views

Any Recommendations for a Web Based Large File Transfer System? [closed]

I'm looking for a server software product that: Allows my users to share large files with: The general public securely to 1 or more people (notification via email, optionally with a token that ...
0
votes
2answers
2k views

Large database in SQL Server Express 2008

I've got a large backup that I need to partially install into our instance of SQL Server express 2008. The backup file is over the 4096Mb limit that restricts SQL Server Express. Is there any way to ...
1
vote
1answer
85 views

how to make a very large radmind image, faster

Making the new Snow Leopard radmind image* for my lab involves manipulating over 50GB of applications, including passing them over the network. Each try takes four hours or more, and if it fails ...
1
vote
1answer
437 views

Recommendations for a secure (and simple) dropbox system

I've got an Ubuntu 9.10 Server that I'd like to leverage for large file delivery. Ideally the system would be: Easy for end users download and upload to. Secure web-based Free (I know I know. ...
1
vote
1answer
270 views

Flash Media Server and large files

The company I work for uses Flash Media Server 3.5 to stream video content. The video files themselves are big, ranging from 500mb to 1gig. We've been running into an issue of when we are streaming ...
10
votes
8answers
10k views

Copy large file from one Linux server to another

I'm attempting to copy a 75 gigabyte tgz (mysql lvm snapshot) from a Linux server in our LA data center to another Linux server in our NY data center over a 10MB link. I am getting about 20-30Kb/s ...
10
votes
4answers
4k views

Linux: How to break a large file into smaller files?

I have a giant file (>20 gigs) sitting on my source machine and I need to transfer it to my target machine. For the purposes of this question, let's assume that I do not have network connectivity ...
78
votes
15answers
81k views

Copying a large directory tree locally? cp or rsync?

I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp. I'm worried about permissions ...
2
votes
5answers
1k views

rsync of >4GB files

silly one, do you have any problems with rsync'ing large [ >4GB ] files under modern linux? [ 32bit, 64bit, large file support turned on ]? i've done some tests on my own between 2 64bit boxes and ...
3
votes
3answers
891 views

Network throttling for large file upload process

My application need to have a file upload component which will be uploading very large(>1gb) files. I'm yet to decide on the protocol (HTTP or FTP) to go for (Any help in this regard will be highly ...
13
votes
8answers
1k views

RAID configuration for large NAS

I'm thinking of building a 24 1TB disk NAS box, but I'm not sure what the best drive configuration is. I'm looking at using the areca ARC-1280ML-2G controller, and hanging all 24 drives off of it. ...
7
votes
2answers
1k views

Why can't I create files larger than 2 GB on Linux?

I wanted to share some knowledge I picked up when I ran into trouble using libpcap and snort to sniff a high-capacity (1 GB full duplex; 2 GB max aggregate) network link. The applications would sniff ...
12
votes
23answers
16k views

What is the simplest and fastest way to transfer large file through a Windows network?

I have a Window Server 2000 machine running MS SQL Server that stores over 20GB of data. The database is backed-up every day to the second harddrive. I want to transfer those backup files to another ...
5
votes
7answers
967 views

What is the best way to keep data synchronized between two locations?

For example, I would like to keep a lot of music files synchronized on two different hard drives. What is the most efficient way to do this? The more automated the method, the better!