Tag Info

Hot answers tagged

7

ClusterSSH, Capistrano, pconsole, and many other tools exist to execute commands across many Linux servers. In addition, you might look into configuration management tools like Puppet, Chef, SaltStack, or Ansible in terms of orchestrating your entire environment. Generally, the AWS command line interface is for interfacing with the product and services ...


6

I'm not a distributed file system ninja, but after consolidating as many drives I can into as few machines as I can, I would try using iSCSI to connect the bulk of the machines to one main machine. There I could consolidate things into hopefully a fault tolerant storage. Preferably, fault tolerant within a machine (if a drive goes out) and among machines ...


6

Actually, I don't think there are that many realistic options. In order of preference my picks would be: Amazon S3. Meets all your requirements, and your optional qualities too. Has a very good track record of uptime and support. It is not in-house; but is that really not a requirement you could work around, f.x. using VPN access or just good old HTTPS... ...


5

The High Scalability blog has a informative entry on YouTube. I don't think there is anything like the scientific papers on GFS for YouTube -- mostly because Youtube probably didn't use quite so radical technology, at least not until it was integrated with Google. (This is just a guesstimate, I have no inside information to go by.)


5

Having dealt with similar issues in Clustering scenarios, I'm familiar with the situation you describe. Such systems frequently have the concept of a quorum, which is why such systems require an odd number of member nodes. The quorum is used to determine the majority and minority partitions. The quorum is the number, greater than half, that defines what is ...


4

For SMTP part you can use lookup tables in Postfix. Depending on your choice and number of user accounts you can use local files, MySQL, or OpenLDAP for storing the lookup info. I have done this with OpenLDAP and it works great, even though the initial setup can be a bit painful. For the POP/IMAP part Perdition is a nice choice. It can also retrieve the ...


4

If it were me, I would be using GlusterFS. The current release is pretty solid and I know people at some very large installations in both the HPC and Internet space that are relying on it in their production systems. You can basically tailor it to your needs by laying out the components as you need them. Unlike Lustre, there are no dedicated metadata servers ...


4

PHP store it's session in plain files. Have you tried storing them on a common storage? Similiar question: Share PHP sessions in cloud file system Here is an article about this specific issue and different approaches: http://kevin.vanzonneveld.net/techblog/article/enhance_php_session_management/


4

This can be accomplished with DFS-R. It's straightforward. No, DFS uses an algorithm to determine which copy "wins" when there is a conflict. It's not really meant for simultaneous access at different sites. It will replicate changes, but it will not lock the file at other sites. You might want to look at a different solution for this. You can use a VPN ...


4

Based on the above requirements, Ceph may be what you're after. http://ceph.newdream.net/ Ceph provides a distributed, POSIX-compliant file system, that you can mount as a block device using the Rados block device. This is implemented directly in modern Linux kernels (2.6.37+). There's even a Qemu/KVM storage driver which means you can mount Ceph ...


3

First, log files can be compressed at really high ratios. I find my log files compress at a 10:1 ratio. If they compress to even a 5:1 ratio, that's only 5GB, or 20% of your storage capacity. Given that you have more than enough storage, the specific compression algorithm isn't too important. You could... Use zip files if Windows users will be accessing ...


3

I've worked with two tools in the past. Zabbix is a free and open source software. It is claimed on their website that it has been tested with 10.000 nodes. NetIQ Security Manager (or NetIQ Application Manager) is a closed and expensive software. It is very easy to scale up but you will need several servers to do so (database and collectors mainly).


3

Assuming that this isn't made up http://users.nagios.org/directory/Yahoo,-Inc/details says that Yahoo uses it for 100,000 machines but has 2000 instances deployed. And I assume that DNX would suit for "management" of the instances. Also just found Merlin which seems to be able to monitor/check 153000 hosts in ~6s rather than 1hr


3

you cand find different ssh clients on the net which connect to multiple clients at the same time, so you can execute commands at the same time. one of these clients is cluster ssh. you could give it a try or search for an other client on the net.


3

For simple command runs, such as greping for a specific string on the server's logs, use a parallel SSH client like pssh or dsh. For more complicated tasks you might want to take a look at MCollective.


3

You might have a look at Splunk for a software-based solution. Q1 Labs also makes hardware-based solutions. Although I have to admit I'm a bit surprised that the way you're currently doing things isn't able to keep up. It might be helpful to be more specific about your needs (i.e.: # of potential transactions, additional log file types, etc.)


3

I think you want Flume. It seems to hit most of the points you are looking for - multiple sources, reliability (E2E guarantee), the ability to write to HDFS (distributed, fault-tolerant storage, integrates into Hadoop for map/reduce. Edit: I'd also like to mention Scribe as another possibility. It's C++ based, written by Facebook, but it seems to have ...


3

There is no cheap answer to this, and nothing appropriate for a 5-PC office. If he thinks that doing this means he only has to buy 1 license for the software, he's incorrect. Any method of doing this is going to expensive, period. Citrix XenApp (formerly presentation server) was made for this, but you have to have good server-class hardware to run it on, ...


3

Given that use case, I'd almost think you were better off with one big SharePoint instance - that would provide the locking mechanism you need, and make the question of DFS and VPN connections somewhat moot.


3

M/Monit is inexpensive and the source code is available. It's only €229 for unlimited hosts. If you have consistent access to the servers (via VPN or otherwise), it makes sense because it does exactly what you're asking for. Try the evaluation and see how it works for you.


3

It's hard to "do better" than stock iptables on Linux servers, IMHO. The tool is straight forward, effective, and well-known by many Linux sysadmins. It sounds to me the problem you're really trying to tackle has less to do with the firewall itself and more to do with managing a security policy across multiple hosts. That speaks configuration management to ...


3

In short, AWS + Cloudformation + Ansible (or some other CM software). With cloudformation, you can define your "nuts and bolts" infrastructure - networks, servers, RDS instances, etc., and deploy this infrastructure in a reliable, repeatable manner. This same cloudformation manifest would be deployed once for each customer. Once the infrastructure is ...


2

Generally switching to a distributed build environment means changing the build chain, as you don't want to use different compilers in development and production. That you mention games makes me assume that you are on VS and use windows, but I will offer some general recommendations Distmake http://distmake.sourceforge.net distcc ...


2

For things that are not unique (that is, replicated services, performing exactly the same duties) I usually share one common server principal. This works well if external entities see the same domain name for instance, as it maintains that illusion. It also means that if a user switches from instance-1 to instance-49 they won't have to perform another ...


2

Might be fun to try this with gnu parallel, maybe something like this: Put the servers in a file servers.txt. Then: parallel --sshlogins servers.txt "grep foo logfile" I haven't tried myself.


2

export these folders via NFS mount them on a single machine with apache running (under document root) as a tree use zip to compress them- good compress ratio, zip can be opened from all OSes list files in Apache -so you are giving users readonly access (log files are not suppose to be edit, right)


2

Buildbot is a system to automate the compile/test cycle. Buildbot supports clusters of machines.


2

One program can only run on one computer. Some tools can use several threads, but that means threads on the same physical computer. HPC mostly means you have specially written software that starts several parallel applications on different computers that are programmed to share the results. Your web browser is not one of those programs. HPC usually is ...


2

It sounds like you are a prime candidate for a cloud-based computing solution. I would investigate using something like Amazon's EC2, which automates all of the steps you listed down to the click of a button. And don't restrict yourself to just EC2, there are plenty of competitors out there. But yes, in short, that is a fairly common practice.


2

You can use S3 for serving files directly without cloudfront to save on cost. Some Scripiting required possibly. You will have ample help on this with jfgi. If you sign up now as a new user you get ample resources for the free tier. I think its much easier than using shared hosting once you get the hang of it. It like driving on the other side of the wheel. ...



Only top voted, non community-wiki answers of a minimum length are eligible