0

I do not know much about networking at all, but I have given the following my best shot.

So I have a HP c7000 enclosure with BL460 g6 blades that have the latest (minimal, so no ifconfig command) version of CentOS installed on them. At the back of the enclosure, there are 4x Cisco Catalyst Blade 3020 switches installed in interconnect bays 1, 2, 5, and 6. During installation of of CentOS, I plugged an ethernet cable from the back of the switch into my local network (which has a DHCP server), and CentOS detected that an ethernet connection exists. After installation however, I had no internet connection at all (ping 8.8.8.8, 1.1.1.1, google.com resulted in 'cannot resolve hostname'). Also, all of the gear is 2nd hand, so I have not setup/configured the Cisco switches at all. I was able to connect to the console page of the switch but since I did not know the username/password, I was unable to configure it.

Also, I tried to ssh into the blades from a separate device on the same network, but it could not connect. Furthermore, running nmcli shows 3 network connections, lo, enp0s0 and enp0s1, and both the enp0s0 and enp0s1 were 'unmanaged'. I have tried changing the config file for them to be managed, resetting the network manager as well as the blade itself but it did not work. After this, I tried connecting a HP 4GB Fibre Channel Pass-thru Module into interconnect bay 3, and iLO showed that the blades correctly map to the right ports into the pass through module. I then connected that port to a separate switch and connected that to the local network, however that did not work.

The goal I am trying to achieve is to have the blades run as a computing cluster, using OpenMPI for communication between blades and not connected to the internet in any way (only connected to the local network to control using ssh). How would I go about achieving this? As far as I know, having the 2 switches in the first 2 interconnect bays allows for the blades to communicate to each other, however even nmcli shows that their is not ip address for the blades, only the hardware address for the ports. Any help will be much appreciated!

  • We need more information. First of all, we need to know what is the current situation with the network. There are two options here: use the ip a command to show you the current status. Also, using the nmcli connnection show <con-name> for each connection that exists would be very helpful. Since you had IPs during the setup, I doubt that there is a problem with the switches, but I do suspect a "wrong" network setup in the boxes. – Peter May 6 '18 at 8:55
  • @Peter Unfortunately I will only be able to access the server in a few days, but I will update you on what I get. Also, the IPs I got were from the HP iLO web interface, and not the blade/centos itself. I also tried to access the iLO remote management for each blade (from the iLO web interface), but it only opened up a web page trying to connect to the IP for that blade, which did not work (connection could not be established, and I tried on multiple blades). – Matthew May 6 '18 at 13:16
1

Those Catalyst Blade 3020 switches are very versatile devices, and you really want to have them in a known configuration. So, one of your first steps should be performing a password reset procedure on them. Granted, based on your experimentation, they may be in a fairly simple configuration... or there may be remnants of their previous configuration that will cause frustrating surprises to you.

Getting the second-hand switches to a known configuration is a must. Trying to configure the network interfaces of the blades won't be very useful if you don't even know for sure whether the switches are going to pass the traffic through in a form that is compatible with the rest of the network, or not.

The switches can be configured to provide specific blades access to specific uplink ports only (and/or VLANs and whatnot...), so your installation-time testing proves that access is configured from that specific port to the specific blade(s) you tested only.

And if the network the switch modules are now connected to does not have a DHCP server, the blades won't get an IP address out of thin air. Especially if the minimal installation does not include avahi-daemon or has it disabled.

(avahi-daemon would assign the interfaces a secondary 169.254.*.* IP address that may vary from one reboot to the next, selected in such a way that it won't conflict with any other host using the same scheme.)

Since you said you don't know much about networking, can you get a network administrator that is familiar with Cisco gear to assist you with the switches? That would probably be very helpful.

Also, learn to take full advantage of the management features of the chassis and the iLO connections.

The first two I/O module slots are hardwired to the integrated NICs on the blades: 2 NICs per half-height blade (like BL460 G6), one for each I/O module slot. The remaining slots can be varied, but you cannot just plug in an I/O module to a slot and have it work: for the I/O module slots 3..8, the blades must also have a corresponding mezzanine card (a palm-sized flat circuit board) installed to them.

You should think I/O modules and mezzanine cards as regular NICs and HBAs split down the middle: the PCIe bus part is in the mezzanine card in each blade, and the Ethernet/FC/InfiniBand-specific hardware is in the I/O modules, bundled up as groups of 16.

If your blade has a FibreChannel HBA mezzanine card in the Mezz1 slot, and you put an Ethernet switch into I/O module slot 3 or 4, it won't work: you're effectively trying to connect a front-end of a FibreChannel SAN card to the back-end of an Ethernet NIC. The chassis management knows that this is not a workable combination, and refuses to power up the blade.

The mapping between the I/O module slots and the blade mezzanine slots is hardwired and not configurable. Here you can find the mapping diagrams: the first one is applicable to your situation.

  • Thanks for your reply. I had previously attempted to reset one of the switches and then connect to the console (only through ethernet, I do not have a serial port/connection) and was met with a username/password form. I dont know if I did the reset properly (I will make sure next time), but given that it is all 2nd hand stuff, would the username and password still stay after the reset? Also, would the switches only need 1 ethernet connection to the local network? Also, I am not sure how the port mapping works as there are 16 slots in the server chassis but only 8 ethernet ports on the switches – Matthew May 6 '18 at 15:22
  • You have quite a learning curve ahead of you, then. Each of those switch modules has 16 "internal" ports (i.e. already hardwired to the blade slots) and 8 external ports. 4 of those external ports are switchable between plain old copper ports and SFP module slots for fiber connections. 2 more can be switched into an internal crossover between switch modules in adjacent slots. Yes, in theory you could cover the entire chassis with just one network cable... if you don't need much bandwidth per blade. Fault tolerance would also be good thing to have. – telcoM May 6 '18 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.