Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
60

I can't speak for Windows instances, but I will presume that their base characteristics are fairly similar to Linux instances. Your estimate for bandwidth usage is 100 simultaneous video downloads (I am not sure if you mean downloading the file or streaming the video - I will assume the latter). If we take a stream rate of 512kbps, you need about 51Mbit/s ...


54

iftop is cool and lightweight. ntop is even cooler but web-based and uses a daemon.


51

Scaling on the backend In a very simple setup, one DNS entry goes to one IP which belongs to one server. Everybody the world over goes to that single machine. With enough traffic, that's just too much to handle long before you get to be YouTube's size. In a simple scenario, we add a load balancer. The job of the load balancer is to redirect traffic to ...


48

For all intents and purposes, no, this is not possible. Even if it were, the technical and contractual logistics required would cripple your business. Think through this a bit more: Joe user at University signs up for your service. You then approach one of the University's many providers (which one? How might you know what provider Joe user's traffic ...


33

An unmanaged switch won't have the feature you're looking for and connecting two ports between both switches will create a switch loop, which will effectively render the switches and the network unusable. A managed switch should have the feature that you're looking for, which is called Link Aggregation (LAG). Before purchasing a managed switch make sure to ...


29

Depending on the frequency at which you exchange data with the customer, it could be cheaper, faster, and more efficient to mail them a storage drive, then pay to have it overnighted back to you when populated with data. This would cost a tiny fraction of what your agreement plan would cost long-term. Turnaround time may be nearly the same if lots of data is ...


28

If you want to be sure, you can use traceroute 1.2.3.4. This will list all the routers between the host running the command and the device with 1.2.3.4 IP address.


27

Remembered that I had bookmarked a similar post a while back, and Cyberx86 posted an excellent answer with benchmark tests :) Serverfault answer Edit From what I've been able to find on the AWS forums - It doesn't seem like the support people from Amazon want to answer that question. Their advice is to test it with an external source: AWS forum post from ...


23

Several techniques are used for large sites. www.youtube.com -> any number of IP addresses Let's look in DNS: www.youtube.com is an alias for youtube-ui.l.google.com. youtube-ui.l.google.com has address 74.125.226.14 youtube-ui.l.google.com has address 74.125.226.0 youtube-ui.l.google.com has address 74.125.226.1 youtube-ui.l.google.com has address 74.125....


21

IPTraf is another common real-time bandwidth monitor on Linux IPTraf-ng is a updated fork of IPTraf with ipv6 support


17

So, the problem here turned out to be a hardware issue. Things were working fine a month ago, and one does not expect failed hardware to still "work" in a degraded mode, but that's what was happening. The troubleshooting step that revealed the issue was to actually look at the ethernet port lights on the back of the cable modem. Instead of the green "...


17

It sounds like in effect you want to pay a users ISP to zero-rate traffic to your site, similar how to some cell carriers allow you to stream video from certain websites without impacting your allocation. If you are a major company like Google or Netflix then this has a ghost of a chance of being feasible, otherwise most companies will not talk to you -- ...


16

Yes, you can do this, if both switches support link aggregation (which goes by a variety of names such as bonding, NIC teaming, port trunking, etc.). You're unlikely to find this feature in consumer grade hardware or unmanaged switch. The answer to the question you referred to even specifically mentioned LACP, one common protocol for link aggregation.


15

If you use "localhost" as hostname for connecting to your MySQL server, MySQL won't use TCP, but sockets. This should be the fastest way to connect to a locally running MySQL server. If your server has the external IP set up "properly", that is, it is not behind a firewall or proxy in an internal network, the traffic won't leave your server as it knows that ...


14

To do this you can make usage of tc alone with u32 filters or combined with iptables marking (maybe more straightforward if you don't want to learn the complex filters syntax). I'll in the following post detail the former solution. Simulating your setup As an example, let's consider A, B, C and D running 10 Mbit/s virtual interfaces. You basically want : ...


14

MB/s states how many Megabytes per second the drive can handle as throughput. IOPS states how many single operations per seconds can be handled. Sequential access means that for example one big file is read, random access means you're reading single parts of different files. If you look for a drive for database usage, you should look for: An enterprise, ...


13

Use LACP for NFS. Use MPIO for iSCSI. If your hypervisor hosts don't have storage interface redundancy, that's where you should focus your attention; no hacks, no bullshit. Add an additional NIC to your hosts and configure MPIO.


13

How do you measure speed of each download ? If you just look at the number the download client displays, then it's probably an average speed calculated over long period of time, i.e. if you've been downloading something for 10 minutes at 100 KB/s and then the speed jumps to 200 KB/s and continues at that rate for another minute, the average over the whole ...


12

The download managers most likely work as explained in the the trickle paper. A process utilizing BSD sockets may perform its own rate limiting. For upstream limiting, the application can do this by simply limiting the rate of data that is written to a socket. Similarly, for downstream limiting, an application may limit the rate of data it ...


12

You are wrong to imagine that YouTube (aka Google) has only one server; this inforgraphic might help illustrate the scale of the system that backs that service. Even if you only have one point of presence you can absolutely have more than one server behind a single name, and even IP, using tools like load balancers and all. Google, though, have an awful ...


12

What I learned so far (correct me if any of it is wrong): Consumer/small office connections are typically asymmetrical, offering much less upload bandwidth, which is critical for a server, than download bandwidth. Consumer and small business broadband providers usually only list maximal bandwidth with no guarantees. By contrast, the "pro-grade" ...


12

If each host only has one interface for iSCSI, then you won't be able to use MPIO with the setup you've described here. However, you should be able to configure the FreeNAS system to use Link Aggregation (LACP), so that you can service two hosts simultaneously each at 1Gb (for a total of 2Gb from the FreeNAS). Instead of MPIO, look into LACP (or, get a ...


11

Network bandwidth is typically expressed in quantity of bits per unit of time -- e.g. 45Mb/sec (small b), or 45Mbit/sec. This expresses a rate of transfer. The amount of data transferred is typically quoted in an absolute quantity of bytes moved - e.g. a 50MB (large B) file, or 50MBytes of data. Most colocation providers sell bandwidth by transfer rate - ...


11

I finally settled for just setting the outgoing bandwidth/latency on the server, and then doing the same on the client, effectively reaching the same result. These are the commands I ran on the server and client respectively to reach my goals: Server: 4 Mbit 50 ms tc qdisc add dev eth0 handle 1: root htb default 11 tc class add dev eth0 parent 1: classid ...


10

nethogs, in case you need traffic analys per application - darkstat also works (web frontend)


9

Citing Wikipedia: Capacity Per lane (each direction): v1.x: 250 MB/s (2.5 GT/s) v2.x: 500 MB/s (5 GT/s) v3.0: 1 GB/s (8 GT/s) v4.0: 2 GB/s (16 GT/s) 16 lane slot (each direction): v1.x: 4 GB/s (40 GT/s) v2.x: 8 GB/s (80 GT/s) v3.0: 16 GB/s (128 GT/s)


9

The open source router firmware DD-WRT appears to support link aggregation; if you're lucky enough to have bought hardware compatible with it then you could re flash the OEM firmware to DD-WRT to gain the feature. Be warned that doing so is warranty voiding; and if something goes wrong you can brick your hardware so caveat emptor. Doing this will also ...


9

Each port has a maximum speed of e.g. 1 GB/sb in one direction, so sending to two other machines will limit the overall send throughput to this value. It could receive the same amount from other machines because of the full-duplex transfer. Switches have a limit of how much traffic the backplane can handle. It is usually lower than the theoretical combined ...


9

Since it seems to be a solution, I am adding it as answer: When creating a VM, the default network adapter is an emulated Intel E1000E. This adapter works in most operating systems without additional drivers, but is unstable and can only use 1Gbit. To use full 10 Gbit traffic between vms on the same host (or through 10Gbit connections to your network) you ...


Only top voted, non community-wiki answers of a minimum length are eligible