Home > Hardware, Home Lab, IBM, RackSwitch > IBM RackSwitch–40GbE comes to the lab!

IBM RackSwitch–40GbE comes to the lab!

Last year, I had a post about 10GbE coming to my home lab (https://vnetwise.wordpress.com/2014/09/20/ibm-rackswitch10gbe-comes-to-the-lab/).  This year, 40GbE comes! 

This definitely falls into the traditional “too good to pass up” category.  A company I’m doing work for picked up a couple of these, and there was enough of a supply that I was able to get my hands on a pair for a reasonable price.  Reasonable at least after liquidating the G8124’s from last year.  (Drop me a line, they’re available for sale! Smile)

Some quick high level on these switches, summarized from the IBM/Lenovo RedBooks (http://www.redbooks.ibm.com/abstracts/tips1272.html?open):

  • 1U Fully Layer 2 and Layer 3 capable
  • 4x 40Gbe QSFP+ and 48x 10GbE SFP+
  • 2x power supply, fully redundant
  • 4x fan modules, also hot swappable.
  • Mini-USB to serial console cable (dear god, how much I hate this non-standard part)
  • Supports 1GbE Copper Transceiver – no issues with Cisco GLC-T= units so far
  • Supports Cisco Copper TwinAx DAC cabling at 10GbE
  • Supports 40GbE QSFP+ cables from 10GTek
  • Supports virtual stacking, allowing for a single management unit

Front panel of the RackSwitch G8264

Everything else generally falls into line with the G8124.  Where those are listed as “Access” switches, these are listed as “Aggregation” switches.  Truly, I’ll probably NEVER have any need for this many 10GbE ports in my home lab, but I’ll also never run out.  Equally, I now have switches that match production in one of my largest environments, so I can get good and familiar with them.

I’m still on the fence about the value of the stacking.  While these are largely going to be used for ISCSI or NFS based storage, stacking may not even be required.  In fact there’s an argument to be made about having them be completely segregated other than port-channels between them, so as to ensure that a bad stack command doesn’t take out both.  Also the Implementing IBM System Networking 10Gb Ethernet Switches guide, it shows the following limitations:

When in stacking mode, the following stand-alone features are not supported:
Active Multi-Path Protocol (AMP)
BCM rate control
Border Gateway Protocol (BGP)
Converge Enhanced Ethernet (CEE)
Fibre Channel over Ethernet (FCoE)
IGMP Relay and IGMPv3
Link Layer Detection Protocol (LLDP)
Loopback Interfaces
MAC address notification
Port flood blocking
Protocol-based VLANs
Router IDs
Route maps
sFlow port monitoring
Static MAC address addition
Static multicast
Uni-Directional Link Detection (UDLD)
Virtual NICs
Virtual Router Redundancy Protocol (VRRP)

That sure seems like a lot of limitations.  At a glance, I’m not sure anything there is end of the world, but it sure is a lot to give up. 

At this point, I’m actually considering filling a number of ports with GLC-T’s and using that for 1GbE.  A ‘waste’, perhaps, but if it means I can recycle my 1GbE switches, that’s an additional savings.  If anyone has a box of them they’ve been meaning to get rid of, I’d be happy to work something out. 

Some questions that will likely get asked, that I’ll tackle in advance:

  • Come on, seriously – they’re data center 10/40GbE switches.  YES, they’re loud.  They’re not, however, unliveable.  They do quite down a bit after warm up, where they run everything at 100% cycle to POST.  But make no mistake, you’re not going to put one of these under the OfficeJet in your office and hook up your NAS to it, and not shoot yourself. 
  • Power is actually not that bad.  These are pretty green, and drop power to unlit ports.  I haven’t hooked up a Kill-a-Watt to them, but will tomorrow.  They’re on par with the G8124’s based on the amp display on the PDU’s I have them on right now. 
  • Yes, there are a couple more Winking smile  To give you a ballpark, if you check eBay for a Dell PowerConnect 8024F and think that’s doable – then you’re probably going to be interested.  You’d lose the 4x10GBaseT combo ports, but you’d gain 24x10GbE and 4x 40GbE.
  • I’m not sure yet if there are any 40GbE compatible HBA – just haven’t looked into it.  I’m guessing Mellanox ConnectX-3 might do it.  Really though, even at 10GbE, you’re not saturating that without a ton of disk IO. 

More to come as I build out various configurations for these and come up with what seems to be the best option for a couple of C6100 hosts. 

Wish me luck!

Categories: Hardware, Home Lab, IBM, RackSwitch
  1. Ken DeMaria
    January 30, 2016 at 11:57 AM

    Hi there, this is a great and very informative article. I came across a G8052F recently and am trying to get it configured in my home lab as well. It seems, however that although finding firmware is easy, downloading it is not, as IBM is requiring a valid log in/serial number/support contract. Do you have any idea how to go about getting firmware support for these?

    • January 30, 2016 at 12:53 PM

      So as long as you have an IBM login – and I don’t recall how I got mine, but I don’t have anything under support and it’s not an account from one of my previous employers – you can get to the downloads section, as you found.

      I don’t know if you NEED a valid serial number – but you can search eBay for a unit and look for serial number sticker photos and try those 🙂 Then the download link I got had one for customers and one for “if you’re an authorized 3rd party for the customer and agree…” which has whatever legal ramifications if you click “continue”, you get the files.

      Hopefully that helps!

      • Ken DeMaria
        January 30, 2016 at 2:57 PM

        You sir, are full of excellent information. Thank you!

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: