The largefiles tag has no wiki summary.
0
votes
0answers
79 views
How to solve 2038 error swfupload?
I use swfupload on my debian server.
I have already incrased limits in my php options to:
But it writes me 2038 error after 500-600MB uploading. I can upload files with 200mb, or 300MB, but can't ...
0
votes
0answers
29 views
Can slocate include file sizes?
I want to get a list of the large files on my system but I don't want to have to run the find command to produce this year.
Can the slocate command be used to include file size information? I don't ...
5
votes
2answers
206 views
Rsync show progress for individual file
I'm using rsync to upload/download a pretty large file (20Gb). I'm aware of --progress option, but what it does in my case is just showing me the result at the end.
My problem is that during some ...
1
vote
1answer
258 views
IIS7 - how to configure server to serve large (GBs) files?
I wanted to download some backups (about 4..5Gb in size) via HTTP and was surprised that IIS7 didn't allowed me to do so. The attempt was seemingly blocked - got a 404 page.
Where should I go in the ...
20
votes
4answers
520 views
Pushing Large Files to 500+ Computers
I work with a team to manage 500-600 rented Windows 7 computers for an annual conference. We have a large amount of data that needs to be synced to these computers, up to 1 TiB. The computers are ...
0
votes
2answers
262 views
De-duplicating backup tool on a block basis? [closed]
I am looking for an (ideally free as in speech or beer) backup tool for Unix-like OS which can store deduplicated backups, i.e. only nonredundant content takes up additional space.
I already looked ...
16
votes
6answers
1k views
Which filesystem for large LVM of disks (8 TB)?
I have a Linux server with many 2 TB disks, all currently in a LVM resulting in about 10 TB of space. I use all this space on an ext4 partition, and currently have about 8,8 TB of data.
Problem is, I ...
0
votes
1answer
158 views
resumable tar over ssh
I'd like to use tar cf - | ssh user@hostname tar xf - to copy a very large set of files. The only problem is that it will be over a very flakey VPN link. Is it possible to somehow set up a resumable ...
2
votes
0answers
81 views
Sorting large binary files
Is there Unix utility for sorting large files containing fixed length binary records?
In other words, I'm looking for something like sort(1) but for binary files with fixed length records.
I could ...
3
votes
1answer
2k views
Optimizing rsync for transferring huge files quickly
This is what I need:
Rsync daemon running on source server. Destination server does a rsync pull as below:
rsync -ptv rsync://source_ip:document/source_path/*.abc destination_path/
Huge binary files ...
12
votes
3answers
1k views
Building a 1,000M row MySQL table
This question is reposted from Stack Overflow based on a suggestion in the comments, apologies for the duplication.
Questions
Question 1: as the size of the database table gets larger how can I tune ...
0
votes
2answers
97 views
Storage setup for large files
I need to store over 200TB of data (all types, biggest being video files) and be able to access it over a local network. The files will be accessed for editing or searches. I don't need versioning, ...
0
votes
3answers
373 views
How to upload a large file in pieces to my server?
How can upload a large SQL dump file it's 1GB, is there any software that uploads the file and resumes it later or just cut it into pieces and then join them back on the server ?
because ftp upload is ...
7
votes
2answers
312 views
Unix sort for partially ordered data sets
So I have an enormously large file (around 10GB) and need to sort it, just like in using 'sort' utility, but kindof more effectively.
Problem is, that I don't have memory, CPU power, time, nor free ...
1
vote
3answers
770 views
500 Internal Server Error when uploading large files to MediaWiki
When I upload a large (>245 Mb) file to my MediaWiki wiki (using Upload file, same as for images), I see:
Internal Server Error
The server encountered an internal
error or misconfiguration ...
4
votes
2answers
513 views
Import a 260GB csv file into MySQL
I have a really big csv ~260GB and I want to import it into MySQL. I use the following mysql script in MacOS
DROP TABLE IF EXISTS tmp_catpath_5;
create table tmp_catpath_5(
a1 ...
1
vote
1answer
515 views
/ partition as xfs in Centos 5 ? , any advantage for apps using large files?
I have been investigating a bit previously to install Centos 5 in a box that it will use applications managing large files (i think more than 300 MB minimal).
I was thinking in , as / has a lot of ...
0
votes
1answer
144 views
Sharing large files throughout an organization
We have non-technical staff members who want to collaborate on large files (~500MB). [I believe the files are SMART Notebook files with embedded videos and flash components].
I'm not quite sure what ...
2
votes
3answers
1k views
Free space on ZFS file system unexpectedly missing
Going to completely rephrase this question since it's still an outstanding production issue several months later.
I have a FreeNAS 0.7.2 box, based on FreeBSD 7.3-RELEASE-p1, running ZFS with 4x1TB ...
0
votes
2answers
412 views
How to delete last n lines of a file without copying?
Size of log file: 10 GB
Free space in partition: 6 GB
Need to split log file into smaller pieces, and gzip those. But there's not enough room to run something like split(1), which leaves the ...
2
votes
0answers
421 views
MySQL and large pages/hugetlb performance gain in numbers?
MySQL has large page/hugetlb support for InnoDB. And there's plenty of posts (example) on the topic.
But does anyone have examples of what performance changes they saw ?
I have seen other database ...
1
vote
3answers
1k views
Sync Large File Shares
What is the best way to sync large sets of files between to geographically separate locations so that both locations have the same file?
We have a constantly changing 500gig file share in Location1 ...
1
vote
6answers
245 views
Distribute terrabyte files to the public from web server
We need to set up a website which makes two or three large files publicly available - the files will be 1 or 2 terrabytes each. Although they will be public, in practise I expect only a relatively ...
1
vote
5answers
626 views
How to backup database dumps which only minor changes between the backups?
I have a posgresql dumpfile every night and i like to keep every one of them. Each is 14 GB in size, so my backupdrive will be full soon.
The difference from one day to another are only some 100MB. ...
2
votes
8answers
2k views
Is there a Linux equivalent of Yousendit? [closed]
We want to get functionality similar to Yousendit or one of the other services that allows end users to upload large files. But we have a few Linux servers and want to use these instead of a third ...
9
votes
3answers
1k views
In Bash, are wildcard expansions guaranteed to be in order?
Is the expansion of a wildcard in Bash guaranteed to be in alphabetical order? I am forced to split a large file into 10 Mb pieces so that they can be be accepted by my Mercurial repository.
So I was ...
2
votes
1answer
75 views
Perforce Restore From Multiple Checkpoint Files?
I am working with a very large (~11GB) checkpoint file and trying to do a -jr (journal restore) operation. About half way through the file, I'm hitting an entry which causes an error to occur. I'm ...
2
votes
5answers
1k views
Storing large amounts of data
Say I'd like to store 40 terabyte of data for frequent access, mainly media files, through a web interface. Should I build 4 computers with 10 terabyte each? Any limitations I should know about? What ...
3
votes
6answers
2k views
Any Recommendations for a Web Based Large File Transfer System? [closed]
I'm looking for a server software product that:
Allows my users to share large files with:
The general public
securely to 1 or more people (notification via email, optionally with a token that ...
0
votes
2answers
1k views
Large database in SQL Server Express 2008
I've got a large backup that I need to partially install into our instance of SQL Server express 2008. The backup file is over the 4096Mb limit that restricts SQL Server Express. Is there any way to ...
1
vote
1answer
81 views
how to make a very large radmind image, faster
Making the new Snow Leopard radmind image* for my lab involves manipulating over 50GB of applications, including passing them over the network. Each try takes four hours or more, and if it fails ...
1
vote
1answer
371 views
Recommendations for a secure (and simple) dropbox system
I've got an Ubuntu 9.10 Server that I'd like to leverage for large file delivery. Ideally the system would be:
Easy for end users download and upload to.
Secure
web-based
Free (I know I know. ...
1
vote
1answer
249 views
Flash Media Server and large files
The company I work for uses Flash Media Server 3.5 to stream video content. The video files themselves are big, ranging from 500mb to 1gig. We've been running into an issue of when we are streaming ...
7
votes
8answers
8k views
Copy large file from one Linux server to another
I'm attempting to copy a 75 gigabyte tgz (mysql lvm snapshot) from a Linux server in our LA data center to another Linux server in our NY data center over a 10MB link.
I am getting about 20-30Kb/s ...
9
votes
4answers
2k views
Linux: How to break a large file into smaller files?
I have a giant file (>20 gigs) sitting on my source machine and I need to transfer it to my target machine. For the purposes of this question, let's assume that I do not have network connectivity ...
51
votes
15answers
41k views
Copying a large directory tree locally? cp or rsync?
I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp.
I'm worried about permissions ...
2
votes
5answers
1k views
rsync of >4GB files
silly one,
do you have any problems with rsync'ing large [ >4GB ] files under modern linux? [ 32bit, 64bit, large file support turned on ]? i've done some tests on my own between 2 64bit boxes and ...
3
votes
3answers
829 views
Network throttling for large file upload process
My application need to have a file upload component which will be uploading very large(>1gb) files. I'm yet to decide on the protocol (HTTP or FTP) to go for (Any help in this regard will be highly ...
13
votes
8answers
1k views
RAID configuration for large NAS
I'm thinking of building a 24 1TB disk NAS box, but I'm not sure what the best drive configuration is. I'm looking at using the areca ARC-1280ML-2G controller, and hanging all 24 drives off of it.
...
7
votes
2answers
875 views
Why can't I create files larger than 2 GB on Linux?
I wanted to share some knowledge I picked up when I ran into trouble using libpcap and snort to sniff a high-capacity (1 GB full duplex; 2 GB max aggregate) network link. The applications would sniff ...
11
votes
23answers
10k views
What is the simplest and fastest way to transfer large file through a Windows network?
I have a Window Server 2000 machine running MS SQL Server that stores over 20GB of data. The database is backed-up every day to the second harddrive. I want to transfer those backup files to another ...
5
votes
7answers
883 views
What is the best way to keep data synchronized between two locations?
For example, I would like to keep a lot of music files synchronized on two different hard drives. What is the most efficient way to do this? The more automated the method, the better!
