Blog Relocation

Folks,

This is my last post on this blog as NetApp has graciously offered several of us the opportunity to use Typepad as the hosting service.

So, starting today I will be blogging using the new hosting service. The new Blog is titled Storage Nuts N' Bolts and so I hope to see you there.

Solaris 10 iSCSI configured with Dynamic Discovery

Recently we went thru re-IPing all of our servers and storage arrays in our office. For the most part everything went fine with the exception of a Solaris 10 U3 server I was running iSCSI on.

After I got thru the steps of changing the server's IP address, gateway and DNS entries I rebooted the server. Upon reboot, I noticed a flurry of non-stop error messages at the server's console:

Sep 30 18:37:37 longhorn iscsi: [ID 286457 kern.notice] NOTICE: iscsi connection(8) unable to connect to target SENDTARGETS_DISCOVERY (errno:128)Sep 30 18:37:37 longhorn iscsi: [ID 114404 kern.notice] NOTICE: iscsi discovery failure - SendTargets (0xx.0xx.0xx.0xx)

As a result of this, I was never able to get a login prompt either at the console or via telnet even though I could succesfuly ping the server's new IP address. What the message above indicates is that the initiator issues a SendTargets and waits for the Target to respond with its Targets. To my surprise there's NO timeout and the initiator will try this process indefinately. In fact, just for kicks, I left it trying for an hour and 45'.

That also means that you will be locked out of the server, as attempting to boot into single user mode results in the exact same behavior.

To get around this problem you have 2 options even though option #2, for some, may not be an option.

Option 1
--------
a) Boot from a Solaris cdrom
b) mount /dev/dsk/c#t#d#s0 /a
c) cd /a/etc/iscsi
d) Remove or rename *.dbc and *.dbp files (iscsi not configured any longer)
e) Reboot the server
f) Use iscsiadm and configure the Solaris server with Static discovery (static-config) so you don't get into this situation again

Option 2
---------
a) Change back to the old Target IP address
b) That will enable you to reboot the server
c) Reconfigure the server to use static-config by specifying the target-name, new Target-ip-address and port-number
d) Change the Target IP address to the new one

I followed Option #1 because #2 was not really not an option for us. So the morale of the story is that you may want to consider static-discovery on Solaris with iSCSI.

VMware over NFS: Backup tricks...continued

There have been a couple of questions on how to do file level backups of a Linux vmdk over NFS. I described the process for a Windows vmdk in a previous article here

In order to do this for a Linux vmdks you need to do the following:

1. Create a Flexclone of the NFS volume from a snapshot
2. Mount the flexclone to your linux server
3. Do not use the read-only mount option as linux requires read-write access
4. Specify -t ext3 as the mount option (you can get the FS type per partition by "df -T")
5. Remember to use fdisk -lu to get the starting sector for each partition
   Multiply the starting sector x 512 bytes and specify the result in the "offset" field of the mount command

Here's an example to mount and explor a copy of the /boot partition of a Red Hat 4 U4 vmdk using a flexcloning:

One reader asked a good question regarding Windows. The question was how to do file level backups of partitioned windows vmdks? The answer to this lays in the offset parameter of the mount option

What you need to do in a scenario like this is:

1. Run msinfo32.exe in your Windows vm
2. Go to Components -> Storage -> Disks
3. Note the Partition Starting offsets and specify them as part of the mount option.

Demos from VMworld

I promised last week to post some links to some of the demos we ran after VMworld was over. So for those who have not seen them here they are. There's audio as well so plug in your headsets.

1) VDI on Netapp over NFS


2) Eliminate duplicate data with A-SIS in a VMware environment



There are also several presentations and technical whitepapers at TechONTAP site which you may find very useful.

VMware on NFS: Backup Tricks

Ok, so if you've decided to use VMware over NFS. Then there's always some guy who's find something to neatpick about and so he'll say "Well, can't run VCB on NFS". He's right but I don't see this as an issue? Sometimes it takes imagination to find a solution to a challenge.

Using NFS as a protocol on VMware you have similar choices and flexibility as with VCB and you can mount the NFS volume or a snapshot of the volume on a server other an ESX...Other = Linux in this case.

So if you are deploying VMware on NFS here's a way to backup whole VMDK images or files within VMDKs using Netapp Snapshots given that the Snapshots are accessible to the NFS client.

Mind you that with this approach you do all kinds of cool things and not just backups without impacting the ESX host. You can also restore, or you could also provision...

So here's the process:

1) Install the Linux NTFS driver if it's not already in your Linux built.

Note: For RHEL and Fedora installs click on the About RedHat/FC RPMs

2) Mount the export onto your linux server
# mount xx.xx.xx.xx:/vol/nfstest /mnt/vmnfs

So now you can backup VMDK images or you can drill into the .snapshot directory and back them up from there.

Next step is to backup files within VMDKs by accessing the snapshot...and you get to pick from which one. For this test, I select from the hourly.3 the snapshot named testsnap

3) Mount the VMDK as a loopback mount specifying the starting offset (32256) and NTFS file system type

# mount /mnt/nfstest/.snapshot/hourly.3/testsnap/nfs-flat.vmdk /mnt/vmdk -o ro,loop=/dev/loop2,offset=32256 -t ntfs

Here's your NTFS disk as seen from Linux:

# cd /mnt/vmdk
# ls -l

total 786844
dr-x------ 1 root root 0 Dec 19 03:03 013067c550e7cf93cc24
-r-------- 1 root root 0 Sep 11 2006 AUTOEXEC.BAT-
r-------- 1 root root 210 Dec 18 21:00 boot.ini
-r-------- 1 root root 0 Sep 11 2006 CONFIG.SYS
dr-x------ 1 root root 4096 Dec 18 21:10 Documents and Settings
-r-------- 1 root root 0 Sep 11 2006 IO.SYS
-r-------- 1 root root 0 Sep 11 2006 MSDOS.SYS
-r-------- 1 root root 47772 Mar 25 2005 NTDETECT.COM
-r-------- 1 root root 295536 Mar 25 2005 ntldr
-r-------- 1 root root 805306368 Mar 13 16:42 pagefile.sys
dr-x------ 1 root root 4096 Sep 11 2006 Program Files
dr-x------ 1 root root 0 Sep 11 2006 RECYCLER
dr-x------ 1 root root 0 Sep 11 2006 System Volume Information
dr-x------ 1 root root 0 Dec 19 00:35 tempd
r-x------ 1 root root 65536 Mar 13 17:41 WINDOWS
dr-x------ 1 root root 0 Sep 11 2006 wmpub

The nice thing about the loopback mount is that Linux will see a VMDK's content for any filesystem it recognizes...so now you can backup Windows and Linux VMs.

Here's a more indepth presentation on VMware over NFS including the backup trick from Peter Learmonth as well as a customer presentation from the VMworld breakout sessions. Login and passwords are proivided below:

user name: cbv_rep
password: cbvfor9v9r

Cheers