Tag Info

Hot answers tagged

7

ClusterSSH, Capistrano, pconsole, and many other tools exist to execute commands across many Linux servers. In addition, you might look into configuration management tools like Puppet, Chef, SaltStack, or Ansible in terms of orchestrating your entire environment. Generally, the AWS command line interface is for interfacing with the product and services ...


6

Actually, I don't think there are that many realistic options. In order of preference my picks would be: Amazon S3. Meets all your requirements, and your optional qualities too. Has a very good track record of uptime and support. It is not in-house; but is that really not a requirement you could work around, f.x. using VPN access or just good old HTTPS... ...


6

I'm not a distributed file system ninja, but after consolidating as many drives I can into as few machines as I can, I would try using iSCSI to connect the bulk of the machines to one main machine. There I could consolidate things into hopefully a fault tolerant storage. Preferably, fault tolerant within a machine (if a drive goes out) and among machines ...


5

The High Scalability blog has a informative entry on YouTube. I don't think there is anything like the scientific papers on GFS for YouTube -- mostly because Youtube probably didn't use quite so radical technology, at least not until it was integrated with Google. (This is just a guesstimate, I have no inside information to go by.)


5

Having dealt with similar issues in Clustering scenarios, I'm familiar with the situation you describe. Such systems frequently have the concept of a quorum, which is why such systems require an odd number of member nodes. The quorum is used to determine the majority and minority partitions. The quorum is the number, greater than half, that defines what is ...


4

Based on the above requirements, Ceph may be what you're after. http://ceph.newdream.net/ Ceph provides a distributed, POSIX-compliant file system, that you can mount as a block device using the Rados block device. This is implemented directly in modern Linux kernels (2.6.37+). There's even a Qemu/KVM storage driver which means you can mount Ceph ...


4

This can be accomplished with DFS-R. It's straightforward. No, DFS uses an algorithm to determine which copy "wins" when there is a conflict. It's not really meant for simultaneous access at different sites. It will replicate changes, but it will not lock the file at other sites. You might want to look at a different solution for this. You can use a VPN ...


4

If it were me, I would be using GlusterFS. The current release is pretty solid and I know people at some very large installations in both the HPC and Internet space that are relying on it in their production systems. You can basically tailor it to your needs by laying out the components as you need them. Unlike Lustre, there are no dedicated metadata servers ...


4

PHP store it's session in plain files. Have you tried storing them on a common storage? Similiar question: Share PHP sessions in cloud file system Here is an article about this specific issue and different approaches: http://kevin.vanzonneveld.net/techblog/article/enhance_php_session_management/


3

I think you want Flume. It seems to hit most of the points you are looking for - multiple sources, reliability (E2E guarantee), the ability to write to HDFS (distributed, fault-tolerant storage, integrates into Hadoop for map/reduce. Edit: I'd also like to mention Scribe as another possibility. It's C++ based, written by Facebook, but it seems to have ...


3

First, log files can be compressed at really high ratios. I find my log files compress at a 10:1 ratio. If they compress to even a 5:1 ratio, that's only 5GB, or 20% of your storage capacity. Given that you have more than enough storage, the specific compression algorithm isn't too important. You could... Use zip files if Windows users will be accessing ...


3

For SMTP part you can use lookup tables in Postfix. Depending on your choice and number of user accounts you can use local files, MySQL, or OpenLDAP for storing the lookup info. I have done this with OpenLDAP and it works great, even though the initial setup can be a bit painful. For the POP/IMAP part Perdition is a nice choice. It can also retrieve the ...


3

Assuming that this isn't made up http://users.nagios.org/directory/Yahoo,-Inc/details says that Yahoo uses it for 100,000 machines but has 2000 instances deployed. And I assume that DNX would suit for "management" of the instances. Also just found Merlin which seems to be able to monitor/check 153000 hosts in ~6s rather than 1hr


3

I've worked with two tools in the past. Zabbix is a free and open source software. It is claimed on their website that it has been tested with 10.000 nodes. NetIQ Security Manager (or NetIQ Application Manager) is a closed and expensive software. It is very easy to scale up but you will need several servers to do so (database and collectors mainly).


3

you cand find different ssh clients on the net which connect to multiple clients at the same time, so you can execute commands at the same time. one of these clients is cluster ssh. you could give it a try or search for an other client on the net.


3

For simple command runs, such as greping for a specific string on the server's logs, use a parallel SSH client like pssh or dsh. For more complicated tasks you might want to take a look at MCollective.


3

You might have a look at Splunk for a software-based solution. Q1 Labs also makes hardware-based solutions. Although I have to admit I'm a bit surprised that the way you're currently doing things isn't able to keep up. It might be helpful to be more specific about your needs (i.e.: # of potential transactions, additional log file types, etc.)


3

Given that use case, I'd almost think you were better off with one big SharePoint instance - that would provide the locking mechanism you need, and make the question of DFS and VPN connections somewhat moot.


3

There is no cheap answer to this, and nothing appropriate for a 5-PC office. If he thinks that doing this means he only has to buy 1 license for the software, he's incorrect. Any method of doing this is going to expensive, period. Citrix XenApp (formerly presentation server) was made for this, but you have to have good server-class hardware to run it on, ...


3

It's hard to "do better" than stock iptables on Linux servers, IMHO. The tool is straight forward, effective, and well-known by many Linux sysadmins. It sounds to me the problem you're really trying to tackle has less to do with the firewall itself and more to do with managing a security policy across multiple hosts. That speaks configuration management to ...


3

M/Monit is inexpensive and the source code is available. It's only €229 for unlimited hosts. If you have consistent access to the servers (via VPN or otherwise), it makes sense because it does exactly what you're asking for. Try the evaluation and see how it works for you.


3

In short, AWS + Cloudformation + Ansible (or some other CM software). With cloudformation, you can define your "nuts and bolts" infrastructure - networks, servers, RDS instances, etc., and deploy this infrastructure in a reliable, repeatable manner. This same cloudformation manifest would be deployed once for each customer. Once the infrastructure is ...


2

For distributed process control I can recommend MCollective, which is a great open-source tool for running commands on any number of systems based on various real-time metadata. The Marionette Collective AKA MCollective is a framework to build server orchestration or parallel job execution systems. Primarily we’ll use it as a means of programmatic ...


2

First thing I would say, is you may need to reset your expectations of complexity. The subject for your question alone includes: distributed fault-tolerant network block device Each of these alone is generally a topic of at least moderate complexity. Combining all three of them, and you're not going to accomplish it without a little bit of work. I ...


2

As for Facebook, Google, etc: The database servers are not all in one location and certainly not all in total sync all the time. They all employ a distributed system over several clusters of servers for different geographical areas. Clusters are distributed in many countries. Frequency of updates between clusters depends on the need of the system to ...


2

Not sure how constructive this is, however google claims to have near real time syncronicity. They even have their own atomic clocks at their data centers for proper synchronization. Wired has an article on it: http://www.wired.com/wiredenterprise/2012/11/google-spanner-time/


2

Any database would do this. LDAP is one such. any generic RDBMS is another. DNS and TXT records would do this. Secure - from what? Snooping and impersonation? Use IPSec. Distributed - here's where you need to define your terms. You can "distribute" the storage layer for a lot of these options above. Do you need instant or eventual consistency? Do certain ...


2

You can use S3 for serving files directly without cloudfront to save on cost. Some Scripiting required possibly. You will have ample help on this with jfgi. If you sign up now as a new user you get ample resources for the free tier. I think its much easier than using shared hosting once you get the hang of it. It like driving on the other side of the wheel. ...


2

It sounds like you are a prime candidate for a cloud-based computing solution. I would investigate using something like Amazon's EC2, which automates all of the steps you listed down to the click of a button. And don't restrict yourself to just EC2, there are plenty of competitors out there. But yes, in short, that is a fairly common practice.


2

Might be fun to try this with gnu parallel, maybe something like this: Put the servers in a file servers.txt. Then: parallel --sshlogins servers.txt "grep foo logfile" I haven't tried myself.



Only top voted, non community-wiki answers of a minimum length are eligible