Home > FAS2020, FAS3040, Hardware, Home Lab, NetApp > NetApp: How to fix BAD RAID LABEL on OnTAP v8.x disks.

NetApp: How to fix BAD RAID LABEL on OnTAP v8.x disks.

So let’s suppose for a moment, that you have a surplus NetApp controller.  Maybe it can only run OnTAP v7.x (eg: a FAS2020).  You also have some disks/shelves removed from an OnTAP v8.x system.  You go to hook them up, and everything you try says “BAD RAID LABEL” or “RAID label v10 found, expecting v5-9”.   This of course, makes you all sad panda as you realize that your lab isn’t going to get disks any time soon.

But let’s also suppose you’ve got access to ANOTHER NetApp controller – one that *IS* OnTAP v8.x capable.  Say, a FAS3040.  Let’s also suggest that you have some “SPARE” NetApp disks in a shelf that this FAS3040 can use.  Then you have some options.  Here, we can do the following:

  • Get the FAS3040 up to OnTAP v8.x
  • From OnTAP v8.x, fix the ownership, RAIDLABEL and spare status of the disks
  • Connect them back to the OnTAP v7.x system
  • Take ownership of the disks in maintenance mode, then reboot and assign them to the system and create a file system.

The right answer, of course, is BEFORE removing the disks from the original OnTAP v8.x system, ensure you perform the required maintenance and spare them THEN.  You’ll save yourself a lot of grief, be more certain that the disks are unrecoverable, and make the removed disks of far more value should you with to reuse, sell, etc.

In my research I found some similar blog posts that got me in the right direction but not quite where I needed to be:

http://markkulacz.wordpress.com/2011/06/03/re-using-netapp-ontap-8-drives-with-a-ontap-7-x-system/
Here the suggestion is to put the disks onto an OnTAP v8.x system, then downgrade the whole system to v7.x to fix the disks.  That sounds like it would work, but what if the only OnTAP v8.x system you can find, isn’t yours and you can get some ‘flexibility’ with it (eg: maybe it’s a backup/SnapVault unit and can tolerate 8 hour downtime without too much issue, but you certainly can’t effect the existing data or configuration).

 

My process is as follows:

 

· Attach a DS14MK2 FC-ATX shelf with 3x SATA 250GB disks previously running OnTAP v7.3.7

· Boot to the LOADER> prompt and choose “boot_primary”. Press “CTRL+C”

Please choose one of the following:

(1) Normal Boot.

(2) Boot without /etc/rc.

(3) Change password.

(4) Clean configuration and initialize all disks.

(4a) Clean configuration and initialize all disks, and install a new filesystem

(5) Maintenance mode boot.

Choose Option #4a

· Let the disks zero – this will take a few hours

· Reboot when complete

· We don’t care about the extra disk shelves just yet – only getting an actual OnTAP v8.x system available.

· Boot to the LOADER> prompt.

o Give the system an IP address: “ifconfig e0a -addr=10.0.0.77 -mask=255.255.255.0 -gw=10.0.0.1 -dns=10.0.0.11 -domain=netwise.ca”

o Run “netboot http://10.0.0.3/737P1_netboot.q.exe” to install OnTAP v7.3.7P1 from a web server on the local network. This will install it to the 3 local disks.

o Reboot when prompted

· Boot to the OnTAP prompt, which will run through the first time setup, where you assign a hostname, IP, etc to the system.

· Upgrade to OnTAP v8.0.5

o Run “software get http://10.0.0.3/805_q_image.zip” to get the v8.0.5 software

o Run “software update 805_q_image.zip” and follow the prompts

o Reboot when prompted, as OnTAP v8.0.5 will not be loaded until it reboots.

· Again, boot to the LOADER> prompt and then “boot_primary”. Press “CTRL+C” and then choose Option 5

· Run “disk show -a” to see all the disks.

*> disk show -a

Local System ID: 118050772

DISK OWNER POOL SERIAL NUMBER HOME

———— ————- —– ————- ————-

0d.51 (118056620) Pool0 3KR0YYDV00007608NHC7 (118056620)

0d.16 NW-FAS3040B(118050772) Pool0 9QE3ZTT4 NW-FAS3040B(118050772)

0d.18 NW-FAS3040B(118050772) Pool0 9QE3ZV10 NW-FAS3040B(118050772)

0d.17 NW-FAS3040B(118050772) Pool0 9QE3ZXLA NW-FAS3040B(118050772)

0d.59 (118056620) Pool0 VAW3ENWA (118056620)

0d.53 (118056620) Pool0 VAW2ZZJA (118056620)

0b.38 NW-FAS3040B(118050772) Pool0 VAW3JMVA NW-FAS3040B(118050772)

0d.57 (118056620) Pool0 VAW3N4ZA (118056620)

0b.44 NW-FAS3040B(118050772) Pool0 VAW3YGLA NW-FAS3040B(118050772)

0d.60 (118056620) Pool0 VAW3J87A (118056620)

0d.55 (118056620) Pool0 VAW3RP5A (118056620)

0d.61 (118056620) Pool0 VAW3HWZA (118056620)

0b.39 NW-FAS3040B(118050772) Pool0 VAW418NA NW-FAS3040B(118050772)

0b.41 NW-FAS3040B(118050772) Pool0 VAW3395A NW-FAS3040B(118050772)

0b.33 NW-FAS3040B(118050772) Pool0 VAW0BHLA NW-FAS3040B(118050772)

0b.36 NW-FAS3040B(118050772) Pool0 VAW0BR8A NW-FAS3040B(118050772)

0d.58 (118056620) Pool0 VAW33N0A (118056620)

0d.56 (118056620) Pool0 VAW3DGRA (118056620)

0b.40 NW-FAS3040B(118050772) Pool0 VAW0BHVA NW-FAS3040B(118050772)

0b.42 NW-FAS3040B(118050772) Pool0 VAW3Y5MA NW-FAS3040B(118050772)

0b.43 NW-FAS3040B(118050772) Pool0 VAW419GA NW-FAS3040B(118050772)

0b.35 NW-FAS3040B(118050772) Pool0 VAW3Z2NA NW-FAS3040B(118050772)

0d.50 (118056620) Pool0 VAW3RKGA (118056620)

0b.32 NW-FAS3040B(118050772) Pool0 VAW3HABA NW-FAS3040B(118050772)

0b.37 NW-FAS3040B(118050772) Pool0 VAW3WG5A NW-FAS3040B(118050772)

0d.48 (118056620) Pool0 VAW3LBDA (118056620)

0b.45 NW-FAS3040B(118050772) Pool0 VAW0BNNA NW-FAS3040B(118050772)

0d.49 (118056620) Pool0 VAW3M81A (118056620)

0b.34 NW-FAS3040B(118050772) Pool0 VAW0AGXA NW-FAS3040B(118050772)

0d.52 (118056620) Pool0 VAW3RKPA (118056620)

0d.54 (118056620) Pool0 VAW2WV4A (118056620)

This shows a system where disks 0d.16-0d.18 are the 3 SATA disks. 0b.32-0b.45 are already brought over. 0d.48-0d.61 is a second shelf recently added. You can see that they are not owned by the node with the NAME=NW-FAS3040B and SYSID=118050772.

· Run “disk reassign -s 118056620 -d 118050772” to take all disks assigned to 118056620 and assign them to this node.

Partner node must not be in Takeover mode during disk reassignment from maintenance mode.

Serious problems could result!!

Do not proceed with reassignment if the partner is in takeover mode. Abort reassignment (y/n)? n

Proceeding with disk reassignment.

Disk ownership will be updated on all disks previously belonging to Filer with sysid 118056620.

Would you like to continue (y/n)? y

Sun Aug 4 03:26:09 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.51 (S/N 3KR0YYDV00007608NHC7) from (ID 118056620) to (ID 118050772)

Sun Aug 4 03:26:09 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.59 (S/N VAW3ENWA) from (ID 118056620) to (ID 118050772)

Sun Aug 4 03:26:09 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.53 (S/N VAW2ZZJA) from (ID 118056620) to (ID 118050772)

Sun Aug 4 03:26:09 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.57 (S/N VAW3N4ZA) from (ID 118056620) to (ID 118050772)

Sun Aug 4 03:26:09 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.60 (S/N VAW3J87A) from (ID 118056620) to (ID 118050772)

Sun Aug 4 03:26:09 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.55 (S/N VAW3RP5A) from (ID 118056620) to (ID 118050772)

Sun Aug 4 03:26:09 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.61 (S/N VAW3HWZA) from (ID 118056620) to (ID 118050772)

Sun Aug 4 03:26:09 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.58 (S/N VAW33N0A) from (ID 118056620) to (ID 118050772)

Sun Aug 4 03:26:09 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.56 (S/N VAW3DGRA) from (ID 118056620) to (ID 118050772)

Sun Aug 4 03:26:09 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.50 (S/N VAW3RKGA) from (ID 118056620) to (ID 118050772)

Sun Aug 4 03:26:09 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.48 (S/N VAW3LBDA) from (ID 118056620) to (ID 118050772)

Sun Aug 4 03:26:09 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.49 (S/N VAW3M81A) from (ID 118056620) to (ID 118050772)

Sun Aug 4 03:26:09 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.52 (S/N VAW3RKPA) from (ID 118056620) to (ID 118050772)

Sun Aug 4 03:26:09 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.54 (S/N VAW2WV4A) from (ID 118056620) to (ID 118050772)

You will be prompted that this CAN BE VERY DANGEROUS if there is a partner attached and CF FAILOVER in place.

· Now if we run a “disk show -a” again:

*> disk show -a

Local System ID: 118050772

DISK OWNER POOL SERIAL NUMBER HOME

———— ————- —– ————- ————-

0d.49 NW-FAS3040B(118050772) Pool0 VAW3M81A NW-FAS3040B(118050772)

0d.16 NW-FAS3040B(118050772) Pool0 9QE3ZTT4 NW-FAS3040B(118050772)

0d.18 NW-FAS3040B(118050772) Pool0 9QE3ZV10 NW-FAS3040B(118050772)

0d.17 NW-FAS3040B(118050772) Pool0 9QE3ZXLA NW-FAS3040B(118050772)

0d.53 NW-FAS3040B(118050772) Pool0 VAW2ZZJA NW-FAS3040B(118050772)

0d.48 NW-FAS3040B(118050772) Pool0 VAW3LBDA NW-FAS3040B(118050772)

0b.38 NW-FAS3040B(118050772) Pool0 VAW3JMVA NW-FAS3040B(118050772)

0d.52 NW-FAS3040B(118050772) Pool0 VAW3RKPA NW-FAS3040B(118050772)

0b.44 NW-FAS3040B(118050772) Pool0 VAW3YGLA NW-FAS3040B(118050772)

0d.50 NW-FAS3040B(118050772) Pool0 VAW3RKGA NW-FAS3040B(118050772)

0d.55 NW-FAS3040B(118050772) Pool0 VAW3RP5A NW-FAS3040B(118050772)

0b.39 NW-FAS3040B(118050772) Pool0 VAW418NA NW-FAS3040B(118050772)

0b.41 NW-FAS3040B(118050772) Pool0 VAW3395A NW-FAS3040B(118050772)

0b.33 NW-FAS3040B(118050772) Pool0 VAW0BHLA NW-FAS3040B(118050772)

0b.36 NW-FAS3040B(118050772) Pool0 VAW0BR8A NW-FAS3040B(118050772)

0d.60 NW-FAS3040B(118050772) Pool0 VAW3J87A NW-FAS3040B(118050772)

0d.61 NW-FAS3040B(118050772) Pool0 VAW3HWZA NW-FAS3040B(118050772)

0b.40 NW-FAS3040B(118050772) Pool0 VAW0BHVA NW-FAS3040B(118050772)

0b.42 NW-FAS3040B(118050772) Pool0 VAW3Y5MA NW-FAS3040B(118050772)

0b.43 NW-FAS3040B(118050772) Pool0 VAW419GA NW-FAS3040B(118050772)

0b.35 NW-FAS3040B(118050772) Pool0 VAW3Z2NA NW-FAS3040B(118050772)

0d.57 NW-FAS3040B(118050772) Pool0 VAW3N4ZA NW-FAS3040B(118050772)

0b.32 NW-FAS3040B(118050772) Pool0 VAW3HABA NW-FAS3040B(118050772)

0b.37 NW-FAS3040B(118050772) Pool0 VAW3WG5A NW-FAS3040B(118050772)

0d.54 NW-FAS3040B(118050772) Pool0 VAW2WV4A NW-FAS3040B(118050772)

0b.45 NW-FAS3040B(118050772) Pool0 VAW0BNNA NW-FAS3040B(118050772)

0d.58 NW-FAS3040B(118050772) Pool0 VAW33N0A NW-FAS3040B(118050772)

0b.34 NW-FAS3040B(118050772) Pool0 VAW0AGXA NW-FAS3040B(118050772)

0d.59 NW-FAS3040B(118050772) Pool0 VAW3ENWA NW-FAS3040B(118050772)

0d.51 NW-FAS3040B(118050772) Pool0 3KR0YYDV00007608NHC7 NW-FAS3040B(118050772)

0d.56 NW-FAS3040B(118050772) Pool0 VAW3DGRA NW-FAS3040B(118050772)

· Now that the disks are *OWNED* by this controller, reboot it, and login to the OnTAP normal console after a normal boot.

· There is a good chance that the system is not AUTOBOOTing like you might expect. If so, run “setenv autoboot true”. Then ENSURE you use “bye” to save the setenv options.

· Boot to the LOADER> prompt and choose “boot_primary”. Press “CTRL+C”

Please choose one of the following:

(1) Normal Boot.

(2) Boot without /etc/rc.

(3) Change password.

(4) Clean configuration and initialize all disks.

(4a) Clean configuration and initialize all disks, and install a new filesystem

(5) Maintenance mode boot.

Choose Option #5

· Run “label makespare ##.##” to make the disk spares.

label makespare: Disk 0b.16 forced to be a SPARE disk

*> label makespare 0b.17

label makespare: Disk 0b.17 forced to be a SPARE disk

*> label makespare 0b.18

label makespare: Disk 0b.18 forced to be a SPARE disk

· Once all the disks are done, reboot

· Disconnect the disks/shelf from the OnTAP v8.x system used to make them “SPARE” and connect them to the OnTAP v7.x system

· Boot to Option 5 in Maintenance mode

· Run “disk show -a” to confirm the disks are seen, but are owned by a different node

*> disk show -a

Local System ID: 118056620

DISK OWNER POOL SERIAL NUMBER

———— ————- —– ————-

0d.19 NW-FAS3040B(118050772) Pool0 3KR0YYDV00007608NHC7

0b.23 NW-FAS3040B(118050772) Pool0 VAW3RP5A

· Run “disk reassign -s 118050772” to remove the ownership from the old system

*> disk reassign -s 118050772

Partner node must not be in Takeover mode during disk reassignment

from maintenance mode. Serious problems could result!!

Are you sure partner is NOT in takeover mode (y/n)? yes

Disk ownership will be updated on all disks previously belonging to Filer with sysid 118050772.

Would you like to continue (y/n)? yes

Sun Aug 4 19:39:19 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.19 (S/N 3KR0YYDV00007608NHC7) from NW-FAS3040B (ID 118050772) to (ID 118056620)

Sun Aug 4 19:39:19 GMT [diskown.changingOwner:info]: changing ownership for disk 0b.23 (S/N VAW3RP5A) from NW-FAS3040B (ID 118050772) to (ID 118056620)

Sun Aug 4 19:39:19 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.29 (S/N VAW3HWZA) from NW-FAS3040B (ID 118050772) to (ID 118056620)

Sun Aug 4 19:39:19 GMT [diskown.changingOwner:info]: changing ownership for disk 0b.22 (S/N VAW2WV4A) from NW-FAS3040B (ID 118050772) to (ID 118056620)

Sun Aug 4 19:39:19 GMT [diskown.changingOwner:info]: changing ownership for disk 0b.26 (S/N VAW33N0A) from NW-FAS3040B (ID 118050772) to (ID 118056620)

Sun Aug 4 19:39:19 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.28 (S/N VAW3J87A) from NW-FAS3040B (ID 118050772) to (ID 118056620)

Sun Aug 4 19:39:19 GMT [diskown.changingOwner:info]: changing ownership for disk 0b.20 (S/N VAW3RKPA) from NW-FAS3040B (ID 118050772) to (ID 118056620)

Sun Aug 4 19:39:19 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.25 (S/N VAW3N4ZA) from NW-FAS3040B (ID 118050772) to (ID 118056620)

Sun Aug 4 19:39:19 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.24 (S/N VAW3DGRA) from NW-FAS3040B (ID 118050772) to (ID 118056620)

Sun Aug 4 19:39:19 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.21 (S/N VAW2ZZJA) from NW-FAS3040B (ID 118050772) to (ID 118056620)

Sun Aug 4 19:39:19 GMT [diskown.changingOwner:info]: changing ownership for disk 0b.18 (S/N VAW3RKGA) from NW-FAS3040B (ID 118050772) to (ID 118056620)

Sun Aug 4 19:39:19 GMT [diskown.changingOwner:info]: changing ownership for disk 0d.16 (S/N VAW3LBDA) from NW-FAS3040B (ID 118050772) to (ID 118056620)

Sun Aug 4 19:39:19 GMT [diskown.changingOwner:info]: changing ownership for disk 0b.27 (S/N VAW3ENWA) from NW-FAS3040B (ID 118050772) to (ID 118056620)

Sun Aug 4 19:39:19 GMT [diskown.changingOwner:info]: changing ownership for disk 0b.17 (S/N VAW3M81A) from NW-FAS3040B (ID 118050772) to (ID 118056620)

· Run “disk show” to confirm the disks show this unit’s ID and no owner

*> disk show

Local System ID: 118056620

DISK OWNER POOL SERIAL NUMBER

———— ————- —– ————-

0d.19 (118056620) Pool0 3KR0YYDV00007608NHC7

0d.28 (118056620) Pool0 VAW3J87A

· Issue a “halt” and reboot the system.

· At the LOADER> prompt, boot with “autoboot” or “boot_primary” and press CTRL+C for special boot mode.

(1) Normal boot.

(2) Boot without /etc/rc.

(3) Change password.

(4) Initialize owned disks (14 disks are owned by this filer).

(4a) Same as option 4, but create a flexible root volume.

(5) Maintenance mode boot.

Selection (1-5)? 4a

· Choose option “4a”. This will take the 14 disks assigned to this system, and build an aggregate and file system out of them. You’ll be asked to confirm twice.

The system has 14 disks assigned whereas it needs 3 to boot, will try to assign the required number.

Zero disks and install a new file system? yes

This will erase all the data on the disks, are you sure? yes

Zeroing disks takes about 80 minutes.

Sun Aug 4 19:43:19 GMT [coredump.spare.none:info]: No sparecore disk was found.

…………………………………………….

And you can see, we are not getting any “RAID Label” errors.

· Let the disks zero – this will take a few hours

· Reboot when complete

· Boot to the LOADER> prompt.

o Give the system an IP address: “ifconfig e0a -addr=10.0.0.78 -mask=255.255.255.0 -gw=10.0.0.1 -dns=10.0.0.11 -domain=netwise.ca”

o Run “netboot http://10.0.0.3/737P1_netboot.q.exe” to install OnTAP v7.3.7P1 from a web server on the local network. This will install it to the local disks.

o Reboot when prompted

At this point you can now continue to run OnTAP v7.3.7P1 on the system (if it is a system that cannot run v8.x) or you can continue to upgrade as appropriate.

 

There you go. What we’ve done:

· Use ‘spare’ v7.x capable disks and created a new aggregate on a FAS3040 running OnTAP v7.3.2.1

· Use that OnTAP v7.3.2.1 system to attach the OnTAP v8.x disk shelf(ves)

· Upgraded the OnTAP v7.3.2.1 to v7.3.7P4 and then to v8.0.5, so that the disks can be used at all.

· Used various “maintenance mode” and “priv set diag” and “priv set advanced” modes to:

o Remove/reset ownership of the disks

o Mark the disks as unowned and maintained

o Change the RAID label of the disks, by using the “label makespare” command

· Attach those still OnTAP v8.x disks, but marked as spare, to another FAS3040 running OnTAP v7.3.2.1

· Boot the OnTAP v7.3.2.1 system into maintenance mode and take ownership.

· Reboot the OnTAP v7.3.2.1 system into maintenance mode and choose option 4a to recreate a filesystem on the disk(s) now available to the system

· Upgrade the OnTAP v7.3.2.1 to v7.3.7P4 and then to v8.0.5, so that the disks can be used at all.

The assumption/requirements:

· That you can locate at least 3 “SPARE” disks. They could be OnTAP v8.x already, as long as they’re spare they should be able to be used by the OnTAP v7.x system to create the aggregate required to build that system up to v8.x

Is there an easier way to do this? With less steps? Perhaps. This worked out for me.

Advertisements
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: