3

For example, Google has over 500 IP address, yet only around 20 data-centers. How can this be.

From what I have read, a typical (huge) data-center will have 1 global uplink, a router, and a main switch. Each cluster is hooked up to the main switch. In the cluster is another switch (its uplink leads the the main switch) which each node is connected to. Is this correct?

If a website is running off a huge cluster and the upload speed is the bottleneck, can you have 2 or more uplinks running to one cluster?

Also, IF you have multiple data-centers for one site, can you configure FTP and other similar applications to upload to each center?

Improve this question
5
  • How did you known that google has over 500 Ip address?
    – Ricardo Polo Jaramillo
    Oct 26, 2011 at 16:18
  • 1
    ITHM "over 9000!"
    – adaptr
    Oct 26, 2011 at 16:22
  • "Google has over 500 IP address, yet only around 20 data-centers. How can this be." - If you think that one dacenter = 1 IP address, you really need to study networking some more.
    – Massimo
    Oct 26, 2011 at 16:29
  • 2
    Whoa whoa whoa ... let's no bash alec, he's asking an honest question here. Go do a google image search for "datacenter topology" and look how many bad examples there are. This isn't an easy subject to learn through intuition unless you've seen a large DC in action.
    – Joseph Kern
    Oct 26, 2011 at 16:40
  • Agreed, but thinking that 1 DC = 1 IP address... shrug
    – Massimo
    Oct 26, 2011 at 20:22

3 Answers 3

Reset to default
12

For example, Google has over 500 IP address, yet only around 20 data-centers. How can this be.

You seem to have a Fundamental Misunderstanding of How The Internet Works. One datacenter houses many servers, and typically has many IP addresses.
IP addresses are not like postal addresses - they can move around (through the magic of BGP)

From what I have read, a typical (huge) data-center will have 1 global uplink, a router, and a main switch. Each cluster is hooked up to the main switch. In the cluster is another switch (its uplink leads the the main switch) which each node is connected to. Is this correct?

What you have read is grossly incorrect. Any datacenter worth hosting in will have multiple uplinks (multiple physical connections to multiple providers - either networks they're buying transit from or peers).

Core switches and routers are typically configured with at least failover-pair redundancy.
Access switches (what the servers plug in to) may be redundant in other ways (the two common ones are each server connects to two separate access switches with separate paths back to the core, or you have two servers each connected to a separate access switch with separate paths back to the core)

If a website is running off a huge cluster and the upload speed is the bottleneck, can you have 2 or more uplinks running to one cluster?

This question makes no sense in the context of how internet traffic flows. You need a better understanding of routing - short version "Traffic follows the shortest path from A to B. If that path is congested the traffic will be slow." - There are more magic things that can be done to reroute traffic around congestion.

Regarding the endpoint server, there are ways to trunk or bond ethernet connections to give individual servers more bandwidth within the datacenter. This is often not useful for increasing speed because the bottleneck is usually out on the internet somewhere...

You are also abusing the word cluster (what KIND of cluster matters).

Also, IF you have multiple data-centers for one site, can you configure FTP and other similar applications to upload to each center?

Short answer: No.
Long answer: Look into distributed filesystems, remote-site mirroring (replication), and geographical redundancy in general.

Improve this answer
3
  • 10
    Most people do have a fundamental misunderstanding about the Internet.
    – Joseph Kern
    Oct 26, 2011 at 16:26
  • 1
    I would say that my own understanding of how the internet operates has some fundamental gaps -- For example I hand-wave a lot of the BGP stuff as "magic" -- I have a vague concept of what's going on, and that's enough to know I don't want to know any more (let someone else deal with that level of crazy!)
    – voretaq7
    Oct 26, 2011 at 19:52
  • 2
    If this was an easy subject, we wouldn't need serverfault. :-)
    – Joseph Kern
    Oct 26, 2011 at 21:44
11

I wonder how you come to these conclusions about how datacenters operate.

Unless it is a very small datacenter, they will have multiple uplinks to two or more NOCs for their AS number(s), and may or may not run their own BGP services.

Larger datacenters will typically have multiple redundant links (that is, 4 or more physical connections) to separate backbone NOCs or peer exchanges.

When you're talking about HUGE datacenters (as google's certainly are), they typically ARE NOCs, or are co-located inside a peer exchange so as to have direct (shortest-path) uplinks to the backbone.

None of this is in any way related to how individual servers or services are run, or presented to the internet.

I also seem to remember that Google employs large quantities of dark fiber (direct, non-internet connections) between their own datacenters. This provides additional redundancy as well as increased inter-site throughput.

Improve this answer
5
  • 7
    He also seems to think that a service provider would only have one public IP per datacenter. NAT has really cramped the brains of a lot of people who have no experience outside of it.
    – mfinni
    Oct 26, 2011 at 16:09
  • 1
    As you may have noticed, I didn't even go there.
    – adaptr
    Oct 26, 2011 at 16:12
  • 1
    Minor correction - AS numbers are purely a BGP thing, and if they're peering via their AS number, BGP is not optional.
    – Shane Madden
    Oct 26, 2011 at 16:15
  • @adaptr - but why not? :-)
    – voretaq7
    Oct 26, 2011 at 16:15
  • @ShaneMadden: not entirely. They may own an AS, yet elect to have their uplinks deal with BGP for that AS. In that case, their uplinks won't be equal-links, but more sort of failover. I have seen this done, although I will grant it is far from common. Technically speaking, owning an AS does not mean "are running BGP"; it means "has authority over these IP spaces". The IP spaces can still be managed by someone else :)
    – adaptr
    Oct 26, 2011 at 16:19
5

Looking through google images for datacenter topologies it's easy to see how you might have come to this conclusion. Most of the examples are for small data centers. They have very little in common with topologies in the scale of Google or Microsoft or Yahoo.

Here's a good example of an internal view of a datacenter (this is not even close to the scale of a single Google, Microsoft, or Yahoo DC):

enter image description here

And here's an example of what might be lurking at the border of the "Campus Core" in the previous diagram (ignore the labels, this is just to illustrate):

enter image description here

Network topology can be far more complex than router-switch-server. It just depends on what you're trying to do.

Improve this answer
6
  • +1 for a great example of decent network architecture.
    – voretaq7
    Oct 26, 2011 at 16:26
  • It took me a while to find ...
    – Joseph Kern
    Oct 26, 2011 at 16:27
  • 1
    Unfortunately, those are just LAN diagrams, which doesn't really answer any of the OP's confusion about public IPs or internet connections.
    – mfinni
    Oct 26, 2011 at 16:53
  • Right, but he was always tangentially asking about data centers in general. IMO his confusion doesn't begin at the AS level. It seems like a fundamental misunderstanding of networking.
    – Joseph Kern
    Oct 26, 2011 at 17:30
  • @mfinni Those LAN diagrams are good examples that generalize out to give a reasonable idea of the architecture of a typical datacenter (as opposed to most of the "One PC --- One switch --- One Router --- Big Internet Cloud" diagrams). It's an inductive leap, but one I'd hope a sysadmin (or sysadmin hopeful) can make...
    – voretaq7
    Oct 26, 2011 at 19:50