Tag Info

Hot answers tagged

19

My WordPress Performance and Caching Stack This is a great WordPress performance stack for a low to mid range single server or VPS. I am classifying mid range as single core with around 1G of memory and fairly fast drives. On your box this would be capable of serving over 10K page views per hour Server Stack Linux - Either Debian Lenny or Ubuntu Nginx ...


19

Potential Issues I have a couple points of issue with using SSDs for production databases at the present time The majority of database transactions on a the majority of websites are reads not writes. As Dave Markle said, you maximize this performance with RAM first. SSDs are new to the mainstream and enterprise markets and no admin worth his salt is going ...


15

I primarily work with an application that has zero horizontal scaling potential. Even though it runs on Linux, the application, data structures and I/O requirements force me to "scale up" onto progressively larger systems in order to accommodate increased user workloads. Many legacy line-of-business and transactional applications have these types of ...


14

That limit is per-directory, not for the whole filesystem, so you could work around it by further sub-dividing things. For instance instead of having all the user subdirectories in the same directory split them per the first two characters of the name so you have something like: top_level_dir |---aa | |---aardvark1 | |---aardvark2 |---da | |---dan | ...


14

Arpit, if you imagine that the absolutely smallest likely web response, even if it's a static text file is one Ethernet packet (~1,500 bytes) then 500,000 of them works out at around 750,000,000 bytes, or roughly 7.5 gigabit. So unless your server has very easily offloaded 10Gb NICs (and it doesnt't, the one you've got is one hundred times slower) and have ...


11

It's called horizontal scalability (or, scale out). You don't add more resources to your system (vertical scalability), you add more machines to your system, and load balance it. The advantage is being 'elastic', that is, when you get hit by high traffic you automatically deploy more machines to handle the load, and in times where the traffic is small you ...


10

Yes you can. Even for some Windows 2003 workloads as little as 384MiB suffices, so 512MiB is a pretty good estimation, be it a little high. RAM should not be a problem, neither should CPU. A 100 VMs is a bit steep, but it is doable, especially if the VMs are not going to be very busy. We easily run 60 servers (Windows 2003 and RHEL) on a single ESX server. ...


10

That's a very, very involved question that lots of smart people spend lots of time thinking about. That said, there are a few tried and true mechanisms you can introduce that will help with scalability. Ultimately, though, it boils down to your application and how it should specifically be scaled. For example, you would scale Oracle differently than you ...


10

Postgres can scale up to as many processors as you want to install, and your OS can handle/manage effectively. You can install Postgres on a 128 core machine (or even a machine with 128 physical processors) and it will work fine. It may even work better than on a 64 core machine if the OS scheduler can handle that many cores. Postgres has been shown to ...


9

First question: Why are you on a relational database to begin with, if you don't need ACID properties? It sounds like you're doing some kind of non-transactional work, so getting a RDMBS with transactions is probably too heavy for your environment. Second question: What kind of data are you storing? You sound like you need a column-store database, and ...


9

First off, I would be more wary of what your internet provider will think. Its pretty much a universal rule that hosting web sites off of personal connections--regardless of traffic--will violate your ToS and end up with you receiving a warning, or getting your service canceled. As for your answer, you're not going to have any issues hardware wise. You ...


9

OK, there is already an accepted answer, but there is something to add.. The most common 'classical' ways of scaling the load balancer tier are (in no particular order): DNS Round Robin to publicize multiple IP addresses for the domain. For each IP address, implement a highly available server pair (2 servers cooperating on keeping one IP address working ...


8

Stackable means that there is a method to connect more disks to the same NAS controller, usually via SAS or something like that, so you can increase the capacity of the same device. While the Netgear will be available with a 12x2TB (24GB) configuration shortly, according to their website, I don't think that this system will fit your needs, as you will be ...


8

It isn't necessarily a piece of hardware doing this but a complete system that has been designed to scale. This not only encompasses the hardware but more importantly the application design, database design (relational or otherwise), networking, storage and how they all fit together. A good starting point for your curiosity on finding out how some of the ...


8

What I've seen with medium to large companies is redundant storage devices such as NetApp or EMC. In fact I was talking to an EMC rep about email storage a little while ago, and he said huge email servers are a very common sale for them. Basically they take all of the storage issues away from the application. Performance for a lot of short random reads ...


8

Load balancers can't easily be scaled by other load balancers since there will inherently be a single load balancer on the chain somewhere maintaining the connections. That said, balancers such as LVS or HAProxy have absurd capacity in the Gbps range. Once you get beyond the capabilities of a single load balancer (software, hardware, whatever), then you'll ...


7

If you are bound to ext2/ext3 the only possibility I see is to partition your data. Find a criterion that splits your data into manageable chunks of similar size. If it's only about the profile images I'd do: Use a hash (e.g SHA1) of the image Use the SHA1 as file and directory name For example the SQUID cache does it this way: f/4b/353ac7303854033 ...


7

I build IPTV systems and you're not going to like it but you've got a very tough task ahead of you, especially considering your budget. Let's go through it step by step (I'm aiming this answer to a broader audience than just you if you don't mind); The first thing anyone building such a system has to do is define its customers in terms of locations, OSs, ...


7

So, most of what you've got there is pretty straightforward to scale. PgSQL onto it's own machine to relieve a pile of CPU/disk IO/memory consumption as a first step. Sphinx into it's own little world too. Definitely switch to resque to allow easy horizontal scaling of your workers. But the files... yes, the files are difficult. They're always ...


7

In old Compaq vernacular the number of CPU sockets was traditionally referred-to as "1-way" for single socket, "2-way" for dual socket, "4-way", etc. I suspect that you're looking at a machine with a single CPU socket.


7

As sysadmin for an environment with dozens of DNS servers and thousands of domains, I feel (well, felt) your pain. We solved it with puppet and templates. All our domains and servers also have entries in our infrastructure database (even the zones get generated from there, but that's irrelevant for now). So we do roughly the following: Master nameservers: ...


7

From a developer perspective I can say that nearly every traditional mainstream database engine out there can only scale up and scaling out is very much an after thought. In recent years with the need for greater scalability and highly available systems there have been efforts to make existing databases scale out. But because the designs are hindered by ...


6

One major problem with a large environment like that would be disaster prevention and data protection. If the server dies, then 100 VMs die with it. You need to plan for some sort of failover of the VMs, and to plan for some sort of "extra-VM" management that will protect your VMs in case of failure. Of course, this sort of redundancy means increased cost ...


6

Google doesn't just store a lot of data on a lot of servers, they have written software that distributes the load and storage automatically, while recovering from hardware faults transparently (like BigTable and Map/Reduce). The alternative is: a. You use someone else's infrastructure (Amazon S3, Akami) b. You use expensive storage vendor's products ...


6

There are no limitations built into Subversion. However, there are practical limits if the repositories are used a lot. You might have a misconception that leads you to believe you do something special at install time to choose limits. You don't. When you use the 'svnadmin create /path/here' command, that is when you choose things like which back-end to ...


6

To accurately plan capacity for this, you need to investigate the characteristics of the application and the load it'll place on the terminal. First off, if this is a critical line-of-business app that will lose the business money if it's not available for a few hours, you want to be looking at 2+ terminal servers running in parallell. Basic load-balancing ...


6

The best anyone here is going to be able to offer you is a wild-ass guess. This isn't something we can estimate for you; we don't have access to your code, your sql statements, or your MySQL and Apache configurations to determine their tuning. You need to grab some HTTP benchmarking software and measure. That is the only way to get a meaningful answer to ...


6

Dedicated, Dedicated, Dedicated, Dedicated </steveballmer> Ok, that's got that out of my system. If I were seriously considering a site with traffic as high as this, and really, 500khits/hour is a LOT. I'd really really consider building my own network and cluster to host it. I'd star off probably with a 4 node system. 2 frontends running ...


6

This is a fairly broad design issue, and is pretty complex. Each of these components involves its own significant design considerations. The level you've presented it is at a pretty abstract level, so the best we can provide are pretty abstract answers. One server per service will provide better service than co-hosting services to some extent. ...



Only top voted, non community-wiki answers of a minimum length are eligible