Lab Learning Lessons–01

So I figured I’d start a new theme, which the title represents.  This is “Lab Learning Lessons” or things you learn in the lab, that are better learned here than in Production somewhere.  Hopefully this will help you out in the future, or if nothing else will reinforce for me that I’ve done this before.  So with that in mind – this week’s lessons!


1) I can’t find the stupid ISCSI Target!

Ever have one of those days?  Setup a new SAN, configure the NIC’s, configure ISCSI, make some LUN’s, configure your Initiator Groups, and… nothing?  Add the ISCSI target to the Dynamic Initiators in the ESXi Software ISCSI Initiator and…. it never finds any Static Initiators like it should?  So you try to do a “vmkping <target_IP>” and sure enough, THAT works.  Worse yet, you do the SAME thing on the secondary NetApp (in this case) controller in that chassis, and IT works.  So you’re doing the right thing.  So you check against the OLD controllers – and your settings are similar as they should be.

So you change the IP addresses on the Targets and… boom.  It works.

Lesson Learned:  IP Address conflicts don’t tell you if the thing that is responding to your test pings is the device you WANT it to be.


2) Can’t vMotion VM’s.  Or create a new one.  Or create objects on a new datastore.

Sounds strange right?  The error includes “pbm.fault.pbm fault.summary” for everything you do.  VM’s are otherwise working and doing what they should be.  You can start, restart, reboot, etc.  You just can’t move them around.  A little Google-fu will suggest that you restart the vCenter Inventory and/or Profile Driven Storage service(s).  Sounds reasonable.  Except those take forever to do so.  So you reboot the vCenter server, hoping that’ll help.  No go.

Then you open Explorer…. and realize your vCenter is out of space.  Except all the services are quite happily started.  No “Automatic” services are unstarted or failing to start.  Nothing is tripping an error.  It’s just “not working”.

Lesson Learned:  Maybe if you’re not going to use a 3rd party monitoring solution (eg: Nagios, ZenOS, PRTG, SolarWinds, etc), then you should configure basic Windows Scheduled Tasks to send e-mails when a drive gets to a certain used amount.  Might save some stress.


3) IP Address Planning.

I’m big on having “predictable” IP Address standards.  If you can, have “Primary” addresses be a x.y.z.1# and “Secondary” be x.y.z.2#, or some other system that works for you.  Also if you have 4 NIC’s maybe the #’s in the previous examples should be the NIC #.  So on a NetApp, e0c and e0d would be 3 and 4, so your IP would be x.y.z.13, x.y.z.14, x.y.z.23, x.y.z.24, or something else.

The downside is you really need to be able to look at the final configuration, and work backwards.  Are you going to do one IP per NIC?  One per LACP/PortChannel?  (Not so much for ISCSI, but for NFS/CIFS).  if you do one for a virtual interface like an LACP vif – what # is it?  It’s none of NIC 1-4 (e0a/e0b/e0c/e0d).  Would you make it .10 and .20?  Maybe.  Or maybe .19 and .29, as it’s ‘odd’.

What if you have a second unit in the same place?   Is your solution scalable?  The if your first pair of controllers is NW-SAN1 and NW-SAN2, and is .1x and .2x, then NW-SAN3 and NW-SAN4 could easily be .3x and .4x – but are you chewing up a lot of IP’s?  Maybe.  But in my opinion, it’s so worth it.  Reading logs and troubleshooting becomes amazingly simpler, as you can now logically map one device to another by IP, hostname, port and NIC.

Lesson Learned:  Plan out as much as you can in advance.   But if you can’t, try it in a simulator and work through your options.  This is why they exist, and why we have labs.

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: