C6100 IPMI Issues with vSphere 6

So I’m not 100% certain if the issues I’m having on my C6100 server are vSphere 6 related or not.  But I have seen similar issues before in my lab, so it may be one of a few things.

After a recent upgrade, I noted that some of my VM’s seemed “slow” – which is hard to quantify.  Then this morning I wake up to having internet but no DNS, so I know my DC is down.  Hosts are up though.  So I give them a hard boot, connect to the IPMI KVM, and watch the startup.  To see “loading IPMI_SI_SRV…” and it just sitting there.

In the past, this seemed to be related to a failing SATA disk, and the solution was to pop it up – which helped temporarily until I replaced the disk outright.  But these are new drives.  Trying the same here did not work, though I only tried the spinning disks and not the SSD’s.  Rather than mess around, I thought I’d find a way to see if I could disable IPMI at least to troubleshoot.

Turns out, I wasn’t alone – though just not specific to vSphere 6:

https://communities.vmware.com/message/2333989

http://www.itblah.com/installing-or-upgrading-to-vmware-vsphere-hypervisor-5-esxi-5-using-the-interactive-method/

https://xuri.me/2014/12/06/avoid-vmware-esxi-loading-module-ipmi_si_drv.html

That last one is the option I took:

• Press SHIFT+O during the Hypervisor startup
• Append “noipmiEnabled” to the boot args

Which got my hosts up and running. 

I haven’t done any deeper troubleshooting, nor have I permanently disabled the IPMI with the options of:

Manually turn off or remove the module by turning the option “VMkernel.Boot.ipmiEnabled” off in vSphere or using the commands below:

# Do a dry run first:
esxcli software vib remove –dry-run —vibname ipmi–ipmi–si–drv
# Remove the module:
esxcli software vib remove —vibname ipmi–ipmi–si–drv

We’ll see what comes when I get more time…

Modifying the Dell C6100 for 10GbE Mezz Cards

In a previous post, Got 10GbE working in the lab – first good results, I talked about getting 10GbE working with my Dell C6100 series.  Recently, a commenter asked me if I had any pictures of the modifications I had to make to the rear panel to make these 10GBE cards work.  As I have another C6100 I recently acquired (yes, I have a problem…), that needs the mods, it seems only prudent to share the steps I took in case it helps someone else.

First a little discussion about what you need:

• Dell C6100 without the rear panel plate to be removed
• Dell X53DF/TCK99 2 Port 10GbE Intel 82599 SFP+ Adapter
• Dell HH4P1 PCI-E Bridge Card

You may find the Mezz card under either part number – it seems that the X53DF replaced the TCK99.  Perhaps one is the P/N and one is the FRU or some such.  But you NEED that little PCI-E bridge card.  It is usually included, but pay special attention to the listing to ensure it does.  What you DON’T really need, is the mesh back plate on the card – you can get it bare. 

Shown above are the 2pt 10GbE SFP+ card in question, and also the 2pt 40GbE Infiniband card.  Above them both is the small PCI-E bridge card.

You want to remove the two screws to remove the backing plate on the card.  You won’t be needing it, and you can set it aside.  The screws attach through the card and into the bracket, so once removed, reinsert the screws to the bracket to keep from losing them.

Here we can see the back panel of the C6100 sled.  Ready to go for cutting.

You can place the factory rear plate over the back plate.  Here you can see where you need to line it up and mark the cuts you’ll be doing.  Note that of course the bracket will sit higher up on the unit, so you’ll have to adjust for your horizontal lines. 

If we look to the left, we can see the source of the problem that causes us to have to do this work.  The back panel here is not removable, and wraps around the left corner of the unit.  In systems with the removable plate, this simply unscrews and panel attached to the card slots in.  In the right hand side you can see the two screws that would attach the panel and card in that case.

Here’s largely what we get once we complete the cuts.  Perhaps you’re better with a Dremel than I am. Note that the vertical cuts can be tough depending on the size of the cutting disk you have, as they may have interference from the bar to remove the sled. 

You can now attach the PCI-E bridge card to the Mezz card, and slot it in.  I found it easiest to come at about 20 degree angle and slot in the 2 ports into the cut outs, then drop the PCI-E bridge into the slot.  When it’s all said and done, you’ll find it pretty secure and good to go.

That’s really about it.  Not a whole lot to it, and if you have it all in hand, you’d figure it out pretty quick.  This is largely to help show where my cut lines ended up compared tot he actual cuts and where adjustments could be made to make the cuts tighter if you wanted.  Also, if you’re planning to order, but are not sure if it works or is possible, then this is going to help out quite a bit.

Some potential vendors I’ve had luck with:

http://www.ebay.com/itm/DELL-X53DF-10GbE-DUAL-PORT-MEZZANINE-CARD-TCK99-POWEREDGE-C6100-C6105-C6220-/181751541002? – accepted $60 USD offer.

http://www.ebay.com/itm/DELL-X53DF-DUAL-PORT-10GE-MEZZANINE-TCK99-C6105-C6220-/181751288032?pt=LH_DefaultDomain_0&hash=item2a513890e0 – currently lists for $54 USD, I’m sure you could get them for $50 without too much negotiating.

EVALExperience now includes vSphere 6!

I know I’ve had both a lot of local VMUG members as well as forum members where I frequent, asking about when vSphere 6, vCenter 6, and ESXi 6 would be available as part of EVALExperience – as understandably, people are anxious to get their learning, labbing, and testing on. 

I’m happy to announce that it looks like it’s up.  If you head over to the VMUG page found at http://www.vmug.com/p/cm/ld/fid=8792 you’ll note that they show:

NEW! vSphere 6 and Virtual SAN 6 Now Available!

Of course, if you’ve signed up with VMUG, you should be getting the e-mail I just received as well.  I’m not certain if it would go to all VMUG Members, only those that are already EVALExperience subscribers, or what. 

What is now included, as per the e-mail blast is:

NOW AVAILABLE! VMware recently announced the general availability of VMware vSphere 6, VMware Integrated OpenStack and VMware Virtual SAN 6 – the industry’s first unified platform for the hybrid cloud! EVALExperience will be releasing the new products and VMUG Advantage subscribers will be able to download the latest versions of:

• vCenter Server Standard for vSphere 6
• vSphere with Operations Management Enterprise Plus
• vCloud Suite Standard
• Virtual SAN 6
• *New* Virtual SAN 6 All Flash Add-On

It is worth noting that the product download has been updated and upgraded.  They do call out that the old product and keys will no longer be available.  I can understand why, as part of this will be to help members stay current.  But it would be nice if you could use the N-1 version for a year of transition, etc.  Not everyone can cut over immediately and some people use their home labs to mirror the production environment at work so they can come home and try something they couldn’t at the office.

Some questions I’ve had asked, and the answers I’m aware of:

1. How many sockets are included?  The package includes 6 sockets for 3 hosts.
2. Are the keys 365 days or dated expiry?  I understand they’re dated expiry, so if you install a new lab 2 weeks before the end of your subscription, you’ll see 14 days left, not 364.
3. What about VSAN?  There had previously been a glitch which gave only one host worth of licences – which clearly does not work.  This has been corrected. 

Just a friendly reminder, as a VMUG leader to look into the VMUG Advantage membership.  As always, VMUG membership itself is free, come on down and attend a local meeting (the next Edmonton VMUG is June 16 and you can sign up here – http://www.vmug.com/p/cm/ld/fid=10777). 

In addition, your VMUG Advantage subscriber benefits include:  

• FREE Access to VMworld 2015 Content
• 20% discount on VMware Certification Exams & Training Courses (If you have a $3500 course you need/want, plus a $225 exam, for $3725 total, spending $200 or so on a VMUG Advantage to make your costs $2800+$180=$2980 is a great way to get $745 off.  This is the sell you should be giving your employer )
• $100 discount on VMworld 2015 registration (This is the only “stackable” discount for VMworld.  Pre-registration/early-bird ends on June 8th I believe)
• 35% discount on VMware’s Lab Connect
• 50% discount on Fusion 7 Professional
• 50% discount on VMware Player 7 Pro
• 50% discount on VMware Workstation 11

Happy labbing, and if you’re local, hope to see you on June 16!

IBM RackSwitch–40GbE comes to the lab!

Last year, I had a post about 10GbE coming to my home lab (https://vnetwise.wordpress.com/2014/09/20/ibm-rackswitch10gbe-comes-to-the-lab/).  This year, 40GbE comes! 

This definitely falls into the traditional “too good to pass up” category.  A company I’m doing work for picked up a couple of these, and there was enough of a supply that I was able to get my hands on a pair for a reasonable price.  Reasonable at least after liquidating the G8124’s from last year.  (Drop me a line, they’re available for sale! )

Some quick high level on these switches, summarized from the IBM/Lenovo RedBooks (http://www.redbooks.ibm.com/abstracts/tips1272.html?open):

• 1U Fully Layer 2 and Layer 3 capable
• 4x 40Gbe QSFP+ and 48x 10GbE SFP+
• 2x power supply, fully redundant
• 4x fan modules, also hot swappable.
• Mini-USB to serial console cable (dear god, how much I hate this non-standard part)
• Supports 1GbE Copper Transceiver – no issues with Cisco GLC-T= units so far
• Supports Cisco Copper TwinAx DAC cabling at 10GbE
• Supports 40GbE QSFP+ cables from 10GTek
• Supports virtual stacking, allowing for a single management unit

Everything else generally falls into line with the G8124.  Where those are listed as “Access” switches, these are listed as “Aggregation” switches.  Truly, I’ll probably NEVER have any need for this many 10GbE ports in my home lab, but I’ll also never run out.  Equally, I now have switches that match production in one of my largest environments, so I can get good and familiar with them.

I’m still on the fence about the value of the stacking.  While these are largely going to be used for ISCSI or NFS based storage, stacking may not even be required.  In fact there’s an argument to be made about having them be completely segregated other than port-channels between them, so as to ensure that a bad stack command doesn’t take out both.  Also the Implementing IBM System Networking 10Gb Ethernet Switches guide, it shows the following limitations:

When in stacking mode, the following stand-alone features are not supported:
Active Multi-Path Protocol (AMP)
BCM rate control
Border Gateway Protocol (BGP)
Converge Enhanced Ethernet (CEE)
Fibre Channel over Ethernet (FCoE)
IGMP Relay and IGMPv3
IPv6
Link Layer Detection Protocol (LLDP)
Loopback Interfaces
MAC address notification
MSTP
OSPF and OSPFv3
Port flood blocking
Protocol-based VLANs
RIP
Router IDs
Route maps
sFlow port monitoring
Static MAC address addition
Static multicast
Uni-Directional Link Detection (UDLD)
Virtual NICs
Virtual Router Redundancy Protocol (VRRP)

That sure seems like a lot of limitations.  At a glance, I’m not sure anything there is end of the world, but it sure is a lot to give up. 

At this point, I’m actually considering filling a number of ports with GLC-T’s and using that for 1GbE.  A ‘waste’, perhaps, but if it means I can recycle my 1GbE switches, that’s an additional savings.  If anyone has a box of them they’ve been meaning to get rid of, I’d be happy to work something out. 

Some questions that will likely get asked, that I’ll tackle in advance:

• Come on, seriously – they’re data center 10/40GbE switches.  YES, they’re loud.  They’re not, however, unliveable.  They do quite down a bit after warm up, where they run everything at 100% cycle to POST.  But make no mistake, you’re not going to put one of these under the OfficeJet in your office and hook up your NAS to it, and not shoot yourself. 
• Power is actually not that bad.  These are pretty green, and drop power to unlit ports.  I haven’t hooked up a Kill-a-Watt to them, but will tomorrow.  They’re on par with the G8124’s based on the amp display on the PDU’s I have them on right now. 
• Yes, there are a couple more   To give you a ballpark, if you check eBay for a Dell PowerConnect 8024F and think that’s doable – then you’re probably going to be interested.  You’d lose the 4x10GBaseT combo ports, but you’d gain 24x10GbE and 4x 40GbE.
• I’m not sure yet if there are any 40GbE compatible HBA – just haven’t looked into it.  I’m guessing Mellanox ConnectX-3 might do it.  Really though, even at 10GbE, you’re not saturating that without a ton of disk IO. 

More to come as I build out various configurations for these and come up with what seems to be the best option for a couple of C6100 hosts. 

Wish me luck!

Got 10GbE working in the lab–first good results

I’ve done a couple of posts recently on some IBM RackSwitch G8124 10GbE switches I’ve picked up.  While I have a few more to come with the settings I finally got working and how I figured them out, I have had some requests from a few people as to how well it’s all working.   So a very quick summary of where I’m at and some results…

What is configured:

• 4x ESXi hosts running ESXi v5.5 U2 on a Dell C6100 4 node
• Each node uses the Dell X53DF dual 10GbE Mezzanine cards (with mounting dremeled in, thanks to a DCS case)
• 2x IBM RackSwitch G8124 10GbE switches
• 1x Dell R510 Running Windows 2012 R2 and StarWind SAN v8.  With both an SSD+HDD VOL, as well as a 20GB RAMDisk based VOL.  Using a BCM57810 2pt 10GbE NIC

Results:

IOMeter against the RAMDisk VOL, configured with 4 workers, 64 threads each, 4K 50% Read/50% Write, 100% Random:

StarWind side:

Shows about 32,000 IOPS

And an Atto Bench32 run:

Those numbers seem a little high.

I’ll post more details once I’ve had some sleep, I had to get something out, I was excited

Soon to come are some details on the switches, for ISCSI configuration without any LACP other than for inter-switch traffic using the ISL/VLAG ports, as well as a “First time, Quick and Dirty Setup for StarWind v8”, as I needed something in the lab that could actually DO 10GbE, and  had to use SSD and/or RAM to get it to have enough ‘go’ to actually see if the 10GbE was working at all.

I wonder what these will look like with some PernixData FVP as well…

UPDATED – 6/10/2015 – I’ve been asked for photos of the work needed to Dremel in the 10GbE Mezz cards on the C6100 server – and have done so!  https://vnetwise.wordpress.com/2015/06/11/modifying-the-dell-c6100-for-10gbe-mezz-cards/

HOWTO: IBM RackSwitch G8124 – Stacking and Port Channels

Welcome to a work in progress J I fully suspect I’ll end up having to circle around and update some of this as I actually get more opportunity to test. I’m still working on some infrastructure in the lab to let me test these switches to their fullest, but in the meantime I’m looking to try to figure out how to get them setup the way I would if I had them at a client site. In general, this means supporting stacking or vPC LACP Port Channels, and connectivity to Cisco Nexus 5548’s.

I managed to find a PDF that shows just such a configuration: http://www.fox-online.cz/ibm/systemxtraining/soubory/czech-2013-bp-final_slawomir-slowinski.pdf

The first figure covers a scenario with teamed NIC’s, with either a Windows host or vSphere ESXi with vDS and LACP:

The second option shows how one might do it with individual non-teamed NIC’s:

The importance of these slides is that the confirm:

• Cisco Nexus vPC connectivity if certainly a valid use case.
• The IBM/BNT/Blade terminology for vPC is vLAG – I can live with that

What isn’t shown on THESE slides is some model information:

• IBM G8000 48x 1GbE switches DO support stacking
• IBM G8052 52x 1GbE switches do NOT support stacking, but support vLAG
• IBM G8124 24x 10GbE switches do NOT support stacking, but support vLAG
• IBM Virtual Fabric 10GbE BladeChassis switches DO support stacking

So there goes my hope for stacking. Not really the end of the world, if it supports vPC(vLAG). So with that in mind, we’ll move on.

I did manage to find a fellow who’s documented the VLAG and VRRP configuration on similar switches: http://pureflexbr.blogspot.ca/2013/10/switch-en4093-vlag-and-vrrp-config.html

So with some piecing together, I get, for Switch 2 (Switch 1 was already configured):

# Configure the LACP Trunk/Port-Channel to be used for the ISL, using ports 23 and 24

interface port 23-24

tagging

lacp mode active

# Set the LACP key to 200

lacp key 200

pvid 4094

exit

!

# Configure VLAN 4094 for the ISL VLAN and move the ports into it.

vlan 4094

enable

name "VLAN 4094"

member 23-24

!

# Set a new STPG of 20 with STP disabled

no spanning-tree stp 20 enable

# Add ports 23 and 24 to said STPG

interface port 23-24

no spanning-tree stp 20 enable

exit

# Create the VLAN and IP Interface

interface ip 100

# Remember that this is on Switch2, so it is using IP2

# Change this when configuring Switch1

ip address 10.0.100.252 255.255.255.0

# configure this subnet configuraiton for VLAN4094

vlan 4094

enable

exit

!

# Configure the vLAG

vlag tier-id 10

# Indicate that the ISL VLAN is 4094

vlag isl vlan 4094

# As we’re on Switch2, this IP will be for Switch1 as the Peer

vlag hlthchk peer-ip 10.0.100.251

# Specify that same LACP ISL key of 200

vlag isl adminkey 200

# Enable the VLAG

vlag enable

!

If all goes well, you’ll see:

Sep 25 22:58:02 NW-IBMG8124B ALERT vlag: vLAG Health check is Up

Sep 25 22:58:11 NW-IBMG8124B ALERT vlag: vLAG ISL is up

Now, the questions I have for this:

· How do I create an actual vLAG – say using Ports 20 on both switches?

· What traffic is passing on this vLAG ISL? Is this just a peer-configuration check, or is it actually passing data? I’m going to assume it’s functioning as a TRUNK ALL port, but I should probably sift through the docs

· When will I have something configured that can use this J

Expect me to figure out how to configure the first in the next few days. It can’t be that much harder. In the meantime, I’m also building up a HDD+SSD StarWind SAN in a host with 2x 10GbE SFP+ that should let me configure port channels all day long. For now, I don’t really need them, so it might be a bit before I come back to this. Realistically, for now, I just need ISCSI, which doesn’t really want any LACP, just each switch/path to be in its own subnet/VLAN/fabric, with individual target/initiator NIC’s, unteamed. So as soon as I get a device up that can handle 10GbE traffic, I’ll be testing that!

HOWTO: IBM RackSwitch G8124 – Initial Configuration

With the acquiring of my new G8124F 10GbE switches (https://vnetwise.wordpress.com/2014/09/20/ibm-rackswitch10gbe-comes-to-the-lab/) , we need to look at the basic configuration. This is going to include general switch management that will be generic to any switches, such as:

• Setting hostname and management IP on the OoB interface
• DNS, SysLog, NTP
• Management users
• Confirming we can back up the config files to a TFTP server
• RADIUS – I expect to need a HOWTO of its own, largely because I’m going to have to figure out what the RADIUS Server side requires

Information we’ll need:

Top Switch:

• Hostname: NW-IBMG8124A
• IP: 10.0.0.94
• MGMT_A: NW-PC6248_1/g39 – VLAN 1 – Access
• p24 -> NW-IBMG8124B/p24
• p23 -> NW-IBMG8124B/p23
• p01 -> NW-ESXI04 vmnic5

Bottom Switch:

• Hostname: NW-IBMG8124B
• IP: 10.0.0.95
• MGMT_A: NW-PC6248_1/g39 – VLAN 1 – Access
• p24 -> NW-IBMG8124A/p24
• p23 -> NW-IBMG8124A/p23

Common Information:

• Subnet: 255.255.255.0
• Gateway: 10.0.0.1
• DNS1: 10.0.0.11
• DNS2: 10.0.0.12
• NTP: 10.0.0.11
• SysLog: 10.0.0.10

Manual Links:

• IBM Installation Manual: https://www-304.ibm.com/support/docview.wss?uid=isg3T7000299&aid=1
• IBM RackSwitch I-S CLI: http://www-01.ibm.com/support/docview.wss?uid=isg3T7000676

What you can tell from above, is that ports 23/24 are linked together with a pair of Cisco passive DAC SFP+ TwinAx cables. Port 1 on the top switch is connected to an unused 10GbE port on an ESXi host so we can do some basic testing. Both switches have their MGTA ports connected to my current Dell PowerConnect 6248 switches, on ports {Top/Bottom}/g39 respectively, with no VLAN trunking. This won’t really matter for the basic configuration we’re doing now, but it will once we start configuring data ports vs simply management interfaces.

1) Initial Login:

I was going to use my Digi CM32 and an RJ45 cable and converter to connect to the DB9, however, both the cable and my converters are both female and I have no serial gender benders on hand. So instead, I opted to use two serial ports on two ESXi hosts, and connect the COM port to a VM. Note, you will have to power down the VM to do so, and it will prevent vMotion, etc. I’m using disposable VM’s I use for benchmarking and testing, so this isn’t a concern. Port speeds are whatever the default PuTTY assumes – 9600,8,N,1, I’m sure.

First, the hard part. The default password is “admin” with no password.

2) Enter configuration:

The first thing you’ll notice, is that so far, this feels very Cisco like. To get started, we enter the “enable” mode and then “conf t” to configure from the terminal.

Command:

enable

configure terminal

3) Let’s confirm our running configuration:

Yup. That’s pretty reset to factory.

Command:

show running-config

4) As per the manual, we’ll set up the management IP’s on both switches:

Page 44 suggests the following commands:

interface ip-mgmt address 10.0.0.94

interface ip-mgmt netmask 255.255.255.0

interface ip-mgmt enable

interface ip-mgmt gateway 10.0.0.1

interface ip-mgmt gateway enable

However, as you can see above, it appears that the version of the firmware I’m running has two options for “interface ip-mgmt gateway” – address w.x.y.z and enable. So the actual commands are:

Commands:

interface ip-mgmt address 10.0.0.94

interface ip-mgmt netmask 255.255.255.0

interface ip-mgmt enable

interface ip-mgmt gateway address 10.0.0.1

interface ip-mgmt gateway enable

You can expect to see a message like the above when the link comes up. In my case, this was because I didn’t configure the Dell PC6248’s until after doing this step.

5) Set the hostname:

Command:

hostname NW-IBMG8124B

We can set the hostname. Note that it changes immediately.

6) Now would be a good time to save our work:

Just like on a Cisco, we can use:

wr mem

or

copy running-config startup-config

Note the prompt above – because the switch is restored to factory defaults, it is booting in a special mode that bypasses any existing configurations. This is why it confirming if you want your next boot to go to the current running/startup config.

7) Set NTP server(s):

You will need to configure at least the “primary-server” if not also the “secondary-server” with an IP address as well as the PORT on the switch that will do the communication. In my case, I’ll be letting the mgta-port connect out, but this could easily be a data port on the switch as well. Do note that it requires an IP address, so you won’t be able to use DNS names such as “ntp1.netwise.ca”, unfortunately. Then, enable the NTP functionality.

Command:

ntp primary-server 10.0.0.11 mgta-port

ntp enable

You’ll note I made a typo, and used the wrong IP. That actually worked out well for the documentation:

When I changed the IP, you can see console immediately displays that it has updated the time.

This is also a good time (pun intended) to set up your timezone. You can use the “system timezone” command to be prompted via menus to select your numbered timezone. As I had no clue what my number might be for Alberta (DST-7?), I ran through the wizard – then checked the running config:

There we go. Command to set America/Canada/Mountain-Alberta as your timezeone:

system timezone 93

8) Setup an admin user:

User access is a little different from a Cisco switch. Here we need to set the name, enter a password, give the user a level, and then enable the user. Note that you cannot enter the password at the command line – it will interactively prompt you. So there’s no point entering any password in the config

Commands:

access user 10 name nwadmin

access user 10 password

access user 10 level administrator

access user 10 enable

The running-config shows the password command as:

access user 10 password "f2cbfe00a240aa00b396b7e361f009f2402cfac143ff32cb09efa7212f92cef2"

Which suggests you must be able to provide the password at the command line, non-interactively.

It is worth noting the built in “administrator” account has some specialty to it. To change this password you would use:

Access user administrator-password <password>

Setting the password to blank (null) will disable the account. Similar also exists for “operator-password” for the “oper” account, but it is disabled by default.

9) Setup SSH:

At this point, the switches are on the network, but I’m still configuring them via serial console. If we attempt to connect to them, we’ll realize that SSH doesn’t work but Telnet does – which is generally expected.

Commands:

ssh port 22

ssh enable

You should now be able to connect as the user you just created, AS WELL AS the default user – admin with a password of admin.

10) Disable Telnet

Now that we’ve configured SSH, let’s get rid of telnet. There is no equivalent “telnet disable”, but you can use “no …” commands.

Commands:

no access telnet enable

Note that my active Telnet configurations has their configurations closed, and indicated on the console.

11) Set SNMP:

My SNMP needs are basic – I largely use it for testing monitoring and management products. So we’ll just set a basic Read Only and Read Write community, and we’ll set it for SNMP v2 which is the most common:

Commands:

snmp location "NetWise Lab"

snmp name NW-IBMG8124B

snmp read-community "nw-ro"

snmp write-community "nw-rw"

snmp version v1v2v3

access snmp read-only

access snmp read-write

NOTE: The SNMP name will change the HOSTNAME, and should not include quotes. This makes me believe it would ASSUME the hostname, which is what most people set to anyway.

12) Configure HTTPS access:

Some people like HTTPS configuration access, some see it as a security risk. I’ll enable it so I have the option of seeing what it looks like

Commands:

access https enable

If there is no self signed certificate, it will generate one.

13) Configure DNS

It would be nice if we could get DNS for hostname resolution. Nothing is worse than having to remember IP’s.

Commands:

ip dns primary-server 10.0.0.11 mgta-port

ip dns secondary-server 10.0.0.12 mgta-port

ip dns domain-name netwise.ca

14) Configure Spanning Tree

Any good switch should do some manner of Spanning Tree. As these will be my storage switches, we’ll ensure these are set to protect against loops and also set as Rapid Spanning Tree (RSTP)

Command:

spanning-tree loopguard

spanning-tree mode rstp

15) Configure SysLog:

This is pretty simple, we simply point it at the IP and tell it to use the mgta-port.

Command:

logging host 1 address 10.0.0.10 mgta-port

logging host 1 severity 7

logging log all

What is nice is you can define two of them, by specifying “host 2”

16) Backup the running config:

Configuring the switch isn’t a lot of good if you don’t back up the configuration. So we’ll make a copy of the config to our TFTP server.

Command:

copy running-config tftp address 10.0.0.48 filename NW-IBMG8124B_orig.cfg mgta-port

It is worth noting that it does support standard FTP as well, if you desire.

So if we take all of the above and put the commands together, we get:

enable

conf t

interface ip-mgmt address 10.0.0.94

interface ip-mgmt netmask 255.255.255.0

interface ip-mgmt enable

interface ip-mgmt gateway address 10.0.0.1

interface ip-mgmt gateway enable

hostname NW-IBMG8124A

copy running-config startup-config

ntp primary-server 10.0.0.11 mgta-port

ntp enable

access user 10 name nwadmin

access user 10 password "f2cbfe00a240aa00b396b7e361f009f2402cfac143ff32cb09efa7212f92cef2"

access user 10 level administrator

access user 10 enable

#access user administrator-password <ChangeMe>

ssh port 22

ssh enable

no access telnet enable

snmp location "NetWise Lab"

snmp name NW-IBMG8124A

snmp read-community "nw-ro"

snmp write-community "nw-rw"

snmp version v1v2v3

access snmp read-only

access snmp read-write

access https enable

ip dns primary-server 10.0.0.11 mgta-port

ip dns secondary-server 10.0.0.12 mgta-port

ip dns domain-name netwise.ca

spanning-tree loopguard

spanning-tree mode rstp

logging host 1 address 10.0.0.10 mgta-port

logging host 1 severity 7

logging log all

We now have a basically working switch, from a management perspective.  Next will be to get it passing some actual data!

 

Some other interesting command:

While poking around in the (conf t) “list” command, which will show you all the command options, I found some interesting ones:

boot cli-mode ibmnos-cli

boot cli-mode iscli

boot cli-mode prompt

The ISCLI is the “Is Cisco Like” which is why it seems familiar. The other option is IBMNOS-CLI, which is… probably painful

 

boot configuration-block active

boot configuration-block backup

boot configuration-block factory

Here is how we can tell the switch to reset itself or boot clean. It’s not immediately clear to me how this would be better than “erase startup-config”, “reload”, but it’s there.

 

boot schedule friday hh:mm

boot schedule monday hh:mm

boot schedule saturday hh:mm

boot schedule sunday hh:mm

boot schedule thursday hh:mm

boot schedule tuesday hh:mm

boot schedule wednesday hh:mm

I can’t think of a lot of times I’ve wanted to schedule the reboot of switches on a weekly basis. Or reasons why I’d need to, on a good switch. But… maybe it’s to know that it WILL reboot when the time comes? If you reboot it weekly, then you might not be so timid to do so after the uptime is 300+ days and no one remembers if this is the switch that has startup issues?

 

interface ip-mgta address A.B.C.D A.B.C.D A.B.C.D enable

Not sure why I’d want multiple IP’s on the management interface – but you can.

interface ip-mgta dhcp

In case you want to set your management IP’s to DHCP. Which sounds like a fun way to have a bad day someday…

 

ldap-server backdoor

Not sure what on earth this does

 

ldap-server domain WORD

ldap-server enable

ldap-server primary-host A.B.C.D mgta-port

ldap-server secondary-host A.B.C.D mgta-port

Need to look into what LDAP supports

 

logging console severify <0-7>

logging console

Sets up how much is logged to the console

 

logging host 1 address A.B.C.D mgta-port

Configures syslog via the mgta-port

 

logging log all

Logs everything, but you can do very granular enablement.

 

radius-server backdoor

Not sure what on earth this does

radius-server domain WORD

radius-server enable

radius-server primary-host A.B.C.D mgta-port

radius-server secondary-host A.B.C.D mgta-port

I’ll need to find the appropriate commands for both the switches as well as the RADIUS server to enable groups.

 

virt vmware dpg update WORD WORD <1-4094>

virt vmware dpg vmac WORD WORD

virt vmware dvswitch add WORD WORD WORD

virt vmware dvswitch add WORD WORD

virt vmware dvswitch addhost WORD WORD

virt vmware dvswitch adduplnk WORD WORD WORD

virt vmware dvswitch del WORD WORD

virt vmware dvswitch remhost WORD WORD

virt vmware dvswitch remuplnk WORD WORD WORD

virt vmware export WORD WORD WORD

I understood the switch was virtualization aware – but this is going to need some deeper investigation!

IBM RackSwitch–10GbE comes to the lab.

I’ve recently come into some IBM RackSwitch switches in both the 1Gbe and 10GbE varieties, that I’m hoping will work out well for the home lab. While they’re neither the Dell 8024F’s or Cisco Nexus 5548’s that I have the most experience with – they’re significantly cheaper. I’m going to be doing a series of posts on getting these units running over the next week or two, but wanted to collect some notes first for those who might like some background on these switches.

First, let’s get some information links out there:

IBM G8000 48 port 1GbE 1U Switch with dual uplink module options:

• RedBook – IBM System Networking RackSwitch G8000
• IBM RackSwitch G8000 Installation Guide
• Firmware – Fix Central
• 48 Port 1GbE – 44x RJ45, 4x SFP
• Optional: 2pt CX4 / 2pt SFP+ 10GbE networking, one front, one rear.
• Stacking, Layer 3, 120w power draw
• “IBM VMready”

IBM G8124 24 port 10GbE SFP+ 1U Switch

• Product Documentation – 7.9.0
• Firmware – Fix Central
• RedBook – IBM System Networking RackSwitch G8124E
• RedBook – Implementing IBM System Networking 10Gb Ethernet Switches
• 24 Port 10GbE SFP+ – apparently works well without requiring “signed” or “branded” SFP+’s, or Active TwinAx.
• Stacking, Layer 3, 120w power draw
• “IBM VMready”

First a little background on these switches, as you’ve probably never heard of them. BLADE Networks was the name of a company that came from the remnants of Nortel’s Blade Chassis Networking division. This would eventually get purchased by IBM and become IBM RackSwitch and BNT products, often seen used in IBM BladeCenter chassis’. So when you go looking for information, you may have to dig and switch names around to find what you’re looking for. The links above should certainly help you out.

So why would someone want some non-Cisco, HP, Juniper, Dell switching? Cost. They’re not very popular, and that works for us. Normally the more popular enterprise equipment was, the lower their resale as off-lease or on eBay. However, when no one has ever heard of it – it tends to make them cheap. Like:

$1000 USD for the G8124 24pt 10GbE switch

$160 USD for the G8000 48pt 1GbE switch

What made me prefer these over the Dell’s I’ve been using? Well I can find a Dell PowerConnect 8024F 24pt 10GbE switch for about $1500 if I get lucky. I do like them as they have 4xRJ45 ports for 1000/10000 ports, and standard RJ45 based console ports. But the power supplies are often $350 and the $1500 switches seldom come with two. The G8124 comes with both power supplies, and supports stacking for $1000. How can I pass that up?

The PowerConnect 6248’s have always treated me well, and support CX4 based stacking in the rear, which removes the need to tie up any front based ports. Also support a 2pt 10GbE SFP+ module that I can never find cheap. With the CX4 stacking module and cables, they’re worth about $600 typically, each. Understandably, the G8000 at $160 was a steal. Especially when you consider that they come with dual power supplies.

So the hope is to take:

1x PC8024F working, single power supply (~$1300)

1x PC8024F non-working, single power supply (~$500)

2x PC6248 working, single power supply, CX4 stacking (~$500)

Or about $2800 in switching

And turn it into:

2x IBM G8124 (~$1300 landed in CAD)

2x IBM G8000 (~$250 landed in CAD)

For about $3100 in switching.

The hope, of course, was to get dual redundant 10GbE switching, and have everything have dual power supplies, for a near wash.

Then I ran into a snag. Turns out these stupid switches have some “mini USB B to DB9” console cable required. It’s the same as the IBM VirtualFabric BladeChassis switches, I could tell based on some units one of my clients had. However, they manage theirs via the chassis/AMM and not the console and didn’t have cables available. So I had to dig some up. If you’re looking for them, you might find this PDF – http://bladenetwork.net/userfiles/file/PDFs/Blade-Console-Cables.pdf – which will help. It’s even so kind as to list some part numbers:

· Blade Part: BN-SB-SRL-CBL

· IBM FRU/CRU: 43X0510

Except, it turns out that FRU isn’t an orderable number, and you’ll get bounced all over IBM if you’re looking for it. You do have two orderable numbers you can try:

· 90Y9338 – mini-USB B to DB9 only

· 90Y9462 – mini-USB B to DB9 and mini-USB B to RJ45 for use with a Cisco console, this is the kit that comes with the VirtualFabric switches.

You’ll find out that no one stocks this cable, and no one is even sure where to order it from. I randomly googled for nights while watching TV until I found a place that said they could. And 3 weeks later I had some cables. They work J

So where am I at now? I have completed the basic configuration and documentation of the G8124’s, and verified they work. I need to get them configured for some networking, primarily stacking and then simple 10GbE so I can use them as vMotion and PernixData interconnects on my ESXi hosts. ISCSI is coming, but right now all my iSCSI is 1GbE. SO I’d have connect the SAN’s to an SFP+ to 1GbE RJ45 transciever. I DO have XFP HBA’s for my NetApp FAS3040’s, so with any luck, and some fibre cables, I can make that work out. What I’d really like to do is find some of the 2pt SFP+ modules for the G8000 and then I think I could get fancy.

If nothing else, I’m learning IBM Networking, and having some fun.

More to come soon!

HOWTO: VMware Horizon View 6 – Overview

I’m going to be doing a series of documents on how to build a general VMware Horizon View 6 Proof of Concept. This is largely for my own benefit and lab, to better understand the environment and concepts, as well as having a sample environment to test in.

We’ll need to review some basic design plans for this environment. The following questions will need answers:

· Will you be using a unified Single Sign On environment?

· Will your View vCenter be Linked-More or completely segregated?

· Will your installation utilize a service account? If so, your installations should all be run logged in AS that account

· What AD group(s) will be added to vCenter for administration? Small environments might utilize “Domain Admins” or “Enterprise Admins”. You may wish to consider creating a group like “VMware vCenter Admin – View”.

o Consider your naming, and how it interacts with your existing Server/Infrastructure environment. For example if that group is “VMware vCenter Admins” will it now become “VMware vCenter Admins – Infrastructure”?

· You are going to require a number of supporting Horizon Suite infrastructure servers. You’ll need to consider where these are going to live. Your options are:

o In the View cluster/vCenter. If you do this, consider that you’ll basically be configuring this environment as a “greenfield”. You’ll need to have a working host(s), with connectivity to storage, etc. If you’re planning on utilizing vDS, you can’t simply configure the hosts prior to the vCenter. You’ll need to configure the vCenter somewhere first – perhaps on local storage, perhaps on NAS/SAN utilizing vSS, etc. You can then import and migrate the vSS to vDS configuration. But you really shouldn’t do this if you’re on a single host, if your vCenter server is running there. So you’ll want to have at least 2 hosts. You can then upgrade the second host to a vDS, get everything working, migrate the vCenter VM to that host and configuration, and then upgrade the first and other hosts.

o In an existing vCenter. Doing this allows you to spin up the vCenter server in parallel with your hosts, and makes your host/cluster configuration much simpler. Also from a licencing standpoint, this makes the most sense. Most sites will be utilizing Windows Server Data Center with unlimited Windows Server VOSI (Virtual Operating System Instances). If you place the supporting Horizon servers in the Horizon cluster, those host will require either a number of Windows Server Standard licences or additional Windows Server Datacenter licences. In the event that choose to place these servers in your standard cluster, you almost certainly have existing Datacenter licences. Equally, if you are going to utilize VMware vSphere for Desktops, which allows you to run a named amount of VM’s on any number of hosts – you are not licenced to run any server based workloads on those hosts.

· Where will you be storing your VDI VM’s?

o I can’t say that I’ve ever heard of anyone who’s been all that successful deploying on an existing SAN used for other workloads. Your mileage may vary. My lab certainly has no other options.

o Ensure your storage supports VAAI and where possible VCSI. In the case of NetApp, if you happen to be utilizing NFS, remember that VAAI for NFS is an add-in host extension that has to be configured. Also, ensure that the NetApp Virtual Storage Console (VSC) is configured and has optimized your hosts, adapters, MPIO, and connection settings.

o Remember that VDI is “response” oriented, vs “transaction” oriented. Because people are sitting and watching their clicks, and animations and videos, they see and feel every moment of what’s going on. A mail/web/DB server that takes a half second longer doesn’t seem slow. A client window that maximizes with “stutter” gives users an ‘impression’ that you can’t take away.

· Where and how will you be keeping your required DB’s for this?

o Almost certainly you shouldn’t use the SQL Express instances, unless it is a very small environment.

o If your VDI is critical to you, your DB’s should be as well. This means some manner of clustering – and hopefully not something based on LUN’s. Go for SQL Always On or similar.

o You may need to look into specific requirements and limitations. For example, vCenter Server DOES support SQL AlwaysOn using shared disks, but does NOT yet support AlwaysOn Availability Groups – http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1024051. They are clear, they will support it as a “best effort”. So likely you won’t have any issues, but it may be worth calling out.

· What are you doing about your desktop management strategy? If your current deployment process takes days and is largely manual with PFY’s doing a large number of hands on touches to deploy desktops, one by one – VDI is very likely not going to improve anything for you. To have a successful VDI implementation you need to have answers to things like:

o Profile management – where and how are you dealing with users and their personal data? Are you redirecting folders now? Locking out saving to C: drives? Making their experience uniform across desktops? Or do you give them a H:/P:/U: (Home/Personal/User, typically) drive mapped and “tell” them to save files there, and “hope” that they do. Does their mobility experience consist of having that drive mapped on a bare desktop?

o Application management – do you still do application installations manually? Are they one-offs? Are they packaged and done via a tool like SCCM or KACE or anything else? If you’re doing it by hand, VDI is leaps and bounds beyond where you are currently.

o Will you be using Dynamic Floating Pools or Static Fixed Pools? Most companies really want to get into Dynamic pools, and they work great for a call center or something where everyone truly is identical. As you start getting into more complex environments, then you start looking at being forced to play with Static fixed desktops. Once you’ve done that, a lot of the power that comes from VDI (eg: rapid deployment, common platform, DR, etc) all start going away.

· What is your long term goal?

o Are you doing this to “save money”? Is your goal that broadly defined? You’ll quite likely find you’re not saving money on the upfront costs. Infrastructure, licencing, installation design – this is all likely to cost you near or more what you’re paying now for traditional desktops. However, you’re likely to save back end management and operational costs.

o Are you doing this for better management? Remember, if all you’re going to do is create 1:1 fixed VM versions of your desktop, housed centrally, manually administered, with users with local admin rights – you’re not going to find this to be true.

o Are you doing this to improve performance? You certainly could be if you’re putting the user closer to the data. Having the user experience in the datacenter, vs over a WAN or ROBO file server, and at 10GbE vs 1GbE, can drastically improve the user experience. However, if you put 500 desktops on traditional storage with no acceleration, when these users are all used to $500 home computers coming with SSD’s and instant response time… you’re going to have a hard time winning over the hearts of those users.

· What are your DR plans?

o One of the great things about a VDI environment is that you can bring it up in a secondary site quickly and allow users to access it from any old BYOD. As long as you CAN in fact, bring it up easily and quickly somewhere else. Remember, now that you’ve centralized this infrastructure, you’ve centralized your possibility for failure. This doesn’t have to be a bad thing, however, as you can now tackle it in one place vs many. But you do need to consider how you’re going to deal with this. It shouldn’t be a bolt on, after the fact.

If you’re go through all of the above, please, above all else, do an actual Proof of Concept. I’ve seen a few deployments go through where clearly the software was purchased and someone “liked” vendor X, and a product was bought. Then afterwards, the attempt was made to make it fit the needs, after finding out it may not fit the requirements. Also, test your failure points. If you’re going to do a PoC and your live deployment will have load balancers – your PoC should. You’re going to want to ACTUALLY see what happens when you “fail” a host. How long does it take to come back? Does everything you expect come back? Does it behave the way you expect? It’s very easy for things to look good when they’re going good. You also need to know how they fail, so you can ensure it works well even when the going gets rough…  There’s nothing worse than finding out the product’s “core” features are great, but it’s ability to be managed, backed up, recovered, and deal with failures are all nightmares.  You’re better off with something that is an 80% across the board, than 100% in one area, and handful of 40%, in my mind.

As I go forward, I’ll be updating this post with links to the individual posts with the details. Otherwise, I fear I’ll keep all these as drafts, and never post anything anywhere.

 

Horizon View 6 Lab Guide – Table of Contents:

Placeholder 1

Placeholder 2

Placeholder 3

Lab Learning Lessons–01

So I figured I’d start a new theme, which the title represents.  This is “Lab Learning Lessons” or things you learn in the lab, that are better learned here than in Production somewhere.  Hopefully this will help you out in the future, or if nothing else will reinforce for me that I’ve done this before.  So with that in mind – this week’s lessons!

 

1) I can’t find the stupid ISCSI Target!

Ever have one of those days?  Setup a new SAN, configure the NIC’s, configure ISCSI, make some LUN’s, configure your Initiator Groups, and… nothing?  Add the ISCSI target to the Dynamic Initiators in the ESXi Software ISCSI Initiator and…. it never finds any Static Initiators like it should?  So you try to do a “vmkping <target_IP>” and sure enough, THAT works.  Worse yet, you do the SAME thing on the secondary NetApp (in this case) controller in that chassis, and IT works.  So you’re doing the right thing.  So you check against the OLD controllers – and your settings are similar as they should be.

So you change the IP addresses on the Targets and… boom.  It works.

Lesson Learned:  IP Address conflicts don’t tell you if the thing that is responding to your test pings is the device you WANT it to be.

 

2) Can’t vMotion VM’s.  Or create a new one.  Or create objects on a new datastore.

Sounds strange right?  The error includes “pbm.fault.pbm fault.summary” for everything you do.  VM’s are otherwise working and doing what they should be.  You can start, restart, reboot, etc.  You just can’t move them around.  A little Google-fu will suggest that you restart the vCenter Inventory and/or Profile Driven Storage service(s).  Sounds reasonable.  Except those take forever to do so.  So you reboot the vCenter server, hoping that’ll help.  No go.

Then you open Explorer…. and realize your vCenter is out of space.  Except all the services are quite happily started.  No “Automatic” services are unstarted or failing to start.  Nothing is tripping an error.  It’s just “not working”.

Lesson Learned:  Maybe if you’re not going to use a 3rd party monitoring solution (eg: Nagios, ZenOS, PRTG, SolarWinds, etc), then you should configure basic Windows Scheduled Tasks to send e-mails when a drive gets to a certain used amount.  Might save some stress.

 

3) IP Address Planning.

I’m big on having “predictable” IP Address standards.  If you can, have “Primary” addresses be a x.y.z.1# and “Secondary” be x.y.z.2#, or some other system that works for you.  Also if you have 4 NIC’s maybe the #’s in the previous examples should be the NIC #.  So on a NetApp, e0c and e0d would be 3 and 4, so your IP would be x.y.z.13, x.y.z.14, x.y.z.23, x.y.z.24, or something else.

The downside is you really need to be able to look at the final configuration, and work backwards.  Are you going to do one IP per NIC?  One per LACP/PortChannel?  (Not so much for ISCSI, but for NFS/CIFS).  if you do one for a virtual interface like an LACP vif – what # is it?  It’s none of NIC 1-4 (e0a/e0b/e0c/e0d).  Would you make it .10 and .20?  Maybe.  Or maybe .19 and .29, as it’s ‘odd’.

What if you have a second unit in the same place?   Is your solution scalable?  The if your first pair of controllers is NW-SAN1 and NW-SAN2, and is .1x and .2x, then NW-SAN3 and NW-SAN4 could easily be .3x and .4x – but are you chewing up a lot of IP’s?  Maybe.  But in my opinion, it’s so worth it.  Reading logs and troubleshooting becomes amazingly simpler, as you can now logically map one device to another by IP, hostname, port and NIC.

Lesson Learned:  Plan out as much as you can in advance.   But if you can’t, try it in a simulator and work through your options.  This is why they exist, and why we have labs.