Please note that I will no longer be posting blogs at this location.  However, I have openned a website dedicated to this topic.  Its location is here:

http://www.oneandonemakesthree.com

In addition to all previous blogs, I will be hosting discussion topics, posting interviews, architecture documents, surveys, etc.

I hope you enjoy it,

Chris

This entry I'll talk about an overview of Snapmirror and how VMware ESX Server allows clean replication of an entire environment from one location to another.  This post will be very specific to Netapp.

Snapmirror is a feature that allows synchronous or asynchronous replication from one Netapp to another.  You can replicate an entire volume or an individual QTREE.  Snapmirror (as with all things Netapp) is an additional licensed product on top of your base Data Ontap license.

Using Snapshots, Snapmirror only replicates the block level changes after the initial "seeding" process is complete.  The seeding process is the initial load where all the contents of the volume are replicated from the source Netapp Filer to the destination.  Because of the volume of data possible, I highly recommend setting up the Snapmirror relationship as early as possible as you don't want to have your initial seeding process replicating 2TB of data over a 10Mb link!

Netapp does have a utility called lrep that allows you to use portable media for the intial seeding process, but that only works for QTREE Snapmirror (QSM) and not for Volume Snapmirror (VSM).  Because you would typically place your luns in a Volume and not a QTREE, this doesn't help us.  So talk to your Netapp Sales Rep and complain, they should fix this!

Snapmirror can not be used for data retention.  Due to the fact that it's a mirror and not a vault, whatever exists on the source, exists on the destination.  If you delete a snapshot on the source, it gets deleted on destination at the next replication schedule.  Snapmirror allows you to have semi-current data at two locations in the event of a disaster.

And speaking of scheduling, Snapmirror partnerships are configured at the destination, and the schedule is enacted at the destination.  So if you want to replicate data at 5pm EST everyday and the destination resides in Germany, then you should configure it so that the replication starts at 10pm local German time. 

From a firewall perspective, connections are initiated from destination to the source on port 10566.  Also, you have to grant permission for a destination filer to grab the data from the source.  You can see who currently has access to perform this operation by typing:

options snapmirror.access

If you are setting up Snapmirror for the first time ensure that add the destination system's IP address onto the source filer by typing a command similar to the following:

options snapmirror.access host=destinationIPaddress

If you would like more information, check the Now website, they will have additional command line information as well as details on Snapmirror.

When you couple VMware and Netapp using Snapmirror, you receive a great many benefits.  VMware of course abstracts the hardware from the operating system, it also encapsulates a virtual machine and its configuration files (in ESX 3.0) in a single location.  By Snapmirror'ing all of this information to a remote site you have a very simple disaster recovery and business resumption plan.  Basically, once you stop the replication and the destination volume is brought online, you can rescan the SAN, import the virtual machine configuration into Virtual Center, and turn it on!

Now granted, there's a little more to it then that, for example, how does your virtual machine IP address become routable out of the new site?  We'll talk about that next post.

The beauty of this is that you don't have to worry about having identical hardware at both sides of the replication pair.  You could have Intel processors at the source site, and AMD at the secondary.  And Snapmirror works between different models of Netapp Filers as well.  Just make sure that your destination filer has and equal to or later version of Ontap otherwise they won't be able to replicate.

Over the next few blogs, we'll take this information and apply it in a more global sense. We've talked about replicating from one site to another, so next we'll evolve that into an enterprise environment and illustrate a few different examples and scenarios.  Keep in mind that the cheapest time to buy Netapp software is when you buy the hardware, so if you support multiple environments and are buying a new filer, get the license, you'll see the great benefits over the next couple of weeks!

Cheers,

Chris

Hello everyone,

Netapp yesterday released an upgrade to their FAS3020 filer.  The following link from Computer World outlines the new release.

http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9012341&source=rss_topic19

Cheers,

Chris

VMware released patches for March addressing several issues.  I looked through it and nothing terribly interesting was fixed, but if you are agressive in your updates you can find more information here:

http://www.vmware.com/support/vi3/doc/esx-6075798-patch.html

Thanks,

Chris

Hey Everyone,

VMware has released a new patch for Virtual Center 2.  The key updates that most of you have experienced are better database usage (no more larege log files or tempdb files) and more integrity in the performance logging.  The VMware community took note of the fact that Virtual Center has gaps in the performance statistics and VMware was working to resolve the issue.  This should now be fixed in this release.

If you have performed the update, please post a comment on anything you've seen or experienced that may help out others.  We'll be putting this into production this week. 

The database issues were easily fixed with some DBA skills, just perform a "dump transaction VCDatabaseName with no_log" to clean the transaction log and then shrink the transaction log.  That would typically get you back some disk space. 

The perfomance data was a real inconvenience.  Because of our aggressive use of VMware, we constantly battle the non-believers that say any little problem is because their application runs on VMware.  So basic approach is to review the performance stats to see if there was an abnormal spike at the time they indicated the performance was poor.  This wouldn't work if the data was gone for the period of time you were looking for.

Just as a side bar, it is very very rare that the application issues that the app owner has brought to us has anything to do with VMware.  Almost always its an application issue that the person was too lazy to investigate!

Details to the upgrade can be found here:

http://www.vmware.com/support/vi3/doc/vc-201-200702-patch.html

Cheers,

Chris

Hey Guys,

Netapp posted a new Snapmirror Best Practices Guide on their site.  You can find it here:

http://www.netapp.com/library/tr/3446.pdf

This is quite topical for my blog as I'm writing up VM replication in a multi-part post that will start coming out this week.  Also, another post I'm working on in the background is "where did my disk space go?".  This will be specific to Netapp and I'm talking to technicians to make sure that I have my facts and naming correct.

Thanks,

Chris

Just to clarify, thanks to Peter who messaged me about iSCSI MPIO on the Netapp.  As per Peter, its not really needed because when a head in a clustered unit fails, the good head will take over the IP addresses for the failed head and will service subsequent requests. 

When we were setting up one of our Netapps we experienced really slow network performance.  Initial connections took about 5 seconds to initiate and after the connection was initiated, through-put was fine.  

After some packet captures we found out that our HP switch was putting the MAC address of the wrong gateway in the packet.  Netapp has an option called ip.fastpath.enable that tells the IP stack to respond using the source MAC address instead of the default gateway.  This allows the Netapp to circumvent bouncing around a subnet in case the packet came into the network on a link other then the default gateway.

If you experience this, first fix your switches!  If that's not an option, then submit the following command in a Telnet or SSH session to your Netapp filer:

options ip.fastpath.enable off

Cheers for now, more next week,

Chris

I've seen a lot of new traffic over the last few days.  Please feel free to comment on what you've seen, give me Kudos, or ideas of areas you would like to see explored. 

Also, I'm looking at the possibility of setting up a web site specific to this topic.  This would allow me to display architecture documents, interviews, links to knowledge base articles specific to this topic and continue my blog in that arena. 

So if you would like more information about Netapp and VMware integration please keep coming back and driving up those numbers. 

Cheers,

Chris

In the training last week I was surprised by how many people want to use ESX Server on a series of Blade servers.  I'll apologize ahead of time for this post being VMware specific and slanting towards traditional servers.

When blades first came out I was really excited about them.  I'm a freak when it comes to cabling, different colors, different sizes, predefined cable runs, etc.  I still think that a good interview test for a sysadmin is to bring them in, give them a bowl of cooked spaghetti noodles and ask them to organize them.  So having a system that allows 14 servers and significantly reduces cabling requirements was something that I strongly related to.

The first versions of the blades were in my opinion unusable.  They were basically laptops with redundant hard drives.  And some didn't even have hot swap hard drives.

Current versions are more powerful, are comparable to 1U servers and continue in the reduction of cabling which reduces the cable clutter some server rooms and data centers experience.

Now here's the thing, what if I get a quad socket, dual core server with lots of memory in a 3U rack space.  If I run ESX Server on it, I can conceivably run 20 – 40 virtual machines.  This is not marketing numbers, this is what I personally have witnessed.  And I am using a total of two power cables, seven network cables (including one for out of band management), two fibre ports and one more network cable for IP KVM access.  Which is better?

So of course some of you will say, "Chris, I'm going to run ESX Server on my blades."  Yes, good choice, however, how is your redundancy going to work?  I recommend that you have at least four network ports (six if you are going to do any iSCSI) and two separate Fibre Channel cards to prevent against a FC card failure and completely losing disk based access. 

In order to attain this you now must (from what I know based upon previous research, sorry if I'm wrong) use the double width blade server, so instead of a possible 14 servers, you have a maximum of 7 blade servers in a single blade enclosure.  If you also use one of your blades to run Virtual Center (which will take one blade), you have a maximum of 6 double width blades.  Now granted, this is probably in a 7U enclosure but what's the maximum amount of RAM you can have, do you have dual cores, can the double width blade system match performance of a similar spec'd physical server?

Keep in mind that a dedicated server would run about 5-15% utilization without virtualization.  With virtualization you can run 50 – 80% comfortably so you have to expect more from your server.  If you can afford it you should also have memory redundancy because you will be putting a lot more memory sims in your server as it will be the weakest link and will cause ESX to purple screen of death (same as a Windows blue screen of death) if there's a failure.  

Now if you go with a physical server and install ESX, you get high density of operating systems, as much redundancy as you need, better physical hardware (as a result of buying big), and the same cable savings you would have with a blade system.   

In my opinion, it makes more sense to go with the physical servers.  And vendors like Sun have incredibly powerful servers in a very small form factor.  So you can get everything you need in a 2U unit furthering my argument.

I guess in the grand scheme of things, it doesn't really matter.  It is more about what you are comfortable with as opposed to what someone else says.  So the take away from this post is more that you think about the different options when you design your architecture and if you are going with VMware then that's a very good start.

Cheers,

Chris

During my training I found out why for iSCSI you need to have a Service Console connection and a VMKernel connection.  Apparently, the Authentication occurs over the service console port and the actual iSCSI communication happens on the VMKernel port.

Also, there is multipathing available on the iSCSI Software Initiator on the ESX 3.0.1 server.  However, it looks like it doesn't work that well on the Netapp.  I think this is because of Netapp's Active Passive clustering, but I'll dig into that more when I get done with Training.

The one thing that struct me during training was a general lack of hands on experience with ESX, and companies treading very lightly into it.  I guess I'm just fortunate that a couple of years ago we were mandated that ESX was the way to go and that we do everything we could to make it work.

Many were scared but I thought it was great and I could see where management was coming from.  While the vision was not complete for a while, we saw the ease of disaster recovery, business resumption, and business continuity with this product.  All you needed to do was invest in the software that replicated your VMs from one site to another and because of the hardware abstraction, as long as the disk files were there, you could run the virtual machine.

In future posts I'll get more into the replication, fall over mechinisms and our implementation of Build Anywhere, Run Anywhere.  Its a good story and I look like a genius.  Just don't tell anyone how easy VMware and Netapp make it!

Cheers,

Chris

Hey there,

I'm in VMware 3 Deploy, Secure and Analyze this week.  If anything interesting comes out of it, I'll post it here.

Cheers,

Chris

With ESX 3, there was a lot of excitement regarding support for iSCSI.  This was especially hard to believe seeing that EMC owns VMware and they have a very small portion of the iSCSI market.

The question I get asked most is won't my virtual machines be slow if the storage is via iSCSI?  My response is show me a server that uses more then 1Gbps of storage bandwidth and then maybe it might be slow.  The reality is that you don't even use 100Mbps 95% of the time, even during boot up which is the most disk intensive operation. 

So, in any VMware implementation, Fibre Channel provided the best fail over.  It was instant.  Network failover was pretty good, you'd look 1-3 packets depending on the switches you used, but FC was flawless everytime. 

So now you have a smaller site with a smaller budget and they want the benefits of virtualization and the ever important VMotion.  So you buy a couple of servers for ESX, a virtual center server, and nice little Netapp, say a FAS250 or a 270.  Could actually be some other iSCSI provider, but we'll stick to Netapp.

The key requirement is that you need to provide redundancy so that the storage for your ESX servers doesn't disappear and all your servers crash.  In this post we'll talk about protecting against port failure and switch failure. 

From a Netapp perspective the key to failure protection is to ensure that you have two network ports available for iSCSI traffic.  If possible, these ports should only do iSCSI, but on the smaller devices you only have two ports per head so you don't have this option.  For this I would recommend a non-routable VLAN, one that doesn't have anything else communicating on it. 

When you have the two network ports, define a Single Mode Virtual Interace (VIF).  This means that of the two ports, only one communicates at a time and requires no configuration from a switch perspective.  Otherwise known as Network Failover.  As a result, you can connect each port to a different switch and protect against both switch failover and port failover. 

Another option is that if you have four ports available, you can create two Multimode VIFs that trunks/aggregates the two ports together for Network Load Balancing.  With Multimode VIFs, you can then create a Single Mode VIF that spans them so that you have Network Failover with Network Load Balancing.  Of the four ports, only two will be active at a time.  The two ports in a Mutlimode VIF must be connected to the same switch and you'll have to enable LACP on those two ports.  However, the other Multimode VIF can be connected to another network switch so you are still protecting against a switch failure.  Hope that makes sense, its easier then it sounds.

Now most people would say that I have a 2Gbps interface into my Netapp, this is great.  However, keep in mind that between two devices, the fastest you can communicate is 1Gbps even if both stations have multiple network cards trunked together.  Where the benefit lies is when multiple servers communicate to one Netapp consuming more then 1Gbps of traffic.  However, you'll rarely see it and if you do then you probably already have Fibre Channel anyways.

Our network redundancy is now complete from a Netapp perspective, so we will now switch to the ESX server.

ESX 3.0 allows you to do something similar to the Netapp in that you can configure two ports to either operate together in Network Load Balancing (requiring switch configuration) or in Network Failover allowing you to span the connection across multiple switches. 

In the configuration of the ESX server, define a new virtual switch and assign two adapters to it.  For a connection that will use the ESX iSCSI software intiator, you need a service console port and a VMkernel port assigned to the virtual switch.  I haven't really figured out why yet, but to make it work both of these are required.  Ensure as well that one of the network adapters that you assigned to the virtual switch is set to standby mode.  This sets it up as Network Failover. 

Also, you need to enable iSCSI Client traffic from the ESX Server by modifying the Security Profile of the ESX Server. 

You now have network failover from a Netapp perspective and an ESX Server.

However....you do not have head failover if you are running clustering on your Netapp.  This is because the ESX iSCSI software initiator does not provide mutli-pathing Input Output (MPIO).  When connecting to the Netapp with iSCSI you should always use MPIO by providing the IP addresses associated with both Netapp heads.  So that if the primary head fails it uses the IP address of the second head to get to the disks. 

I hope this helps.  There is a few gotchas as mentioned, but the value of iSCSI is definately worth trying it out.

Cheers

Netapp has just come out with a new document regarding VMware and Netapp.  At first glance, its pretty good.  Download it here:

http://www.netapp.com/library/tr/3428.pdf

I'll keep this document in mind when I write future posts so that I don't duplicate content.

Cheers,

Chris

Hey Guys,

Sorry its been a while.  Today I'll talk about the benefits of LUN cloning, an interesting feature that's included in with Netapp ONtap version 7 or later. 

I first used lun clone when I wanted to create multiple iSCSI LUNs that contained the same original data.  We had a requirement to create 16 training servers that would be contain the exact same starting data, but be independant so that the data entered during training wouldn't spread to the other servers. 

So how we did this is we created the pristine image, and then duplicated it 15 times.  For the C: drive, I basically just did a copy of the vmdk file 15 times.  The data drive I was lucky enough to setup an iSCSI LUN on the Netapp and use the software based initiator in the guest OS (it was ESX 2.5 at the time). 

What I could do to duplicate the drive is create 15 copies of the iSCSI LUN using NDMPCOPY but then I would be consuming a great deal of space.  Instead, I used lun clone to clone the iSCSI LUN 15 times.  And the cool thing is that the lun clone does not use any additional space from the original lun.  So say my iSCSI LUN was originally 10GB, I lun clone 15 times, and the space used is only 10GB, because there has been no changes to the files.  The only time you use space is when there are block level changes to the cloned luns. 

Taking this one step further, when I setup each server by changing IP, computer name, iSCSI initiator name, and a couple of other things, I took a snapshot and every week I roll back to day 0 so that the trainers don't have any old day at the start of the week.

So we were really happy with this, it worked out well and didn't take up a lot of storage because of the lun clone feature.  So how else can you use this technology.

Say you have data on a LUN and part of it gets corrupted for whatever reason, OS issue, you name it.  You don't want to do a snap restore or maybe you don't have snap restore.  Doing a snap restore would revert the entire volume.  So, a great way is to do a lun clone from a snapshot.  You can now present the lun clone as a secondary drive to the server and drag and drop the files that you need to recover.

Command structure is similar to the following:
lun clone create /vol/serverdatavol/backuplun -o noreserve -b /vol/serverdatavol/currentlun hourly.3

So details from the above would be lun clone create is the first part required, /vol/serverdatavol/backuplun would be name/location of the lun to be created from the clone.  -o noreserve is required because you don't want to consume any space.  -b says that you are cloning from a snapshot and /vol/serverdatavol/currentlun indicates the current active lun.  hourly.3 tells the filer what snapshot to create the clone from. 

A couple of key things here, make sure that if you are using Windows, you are using W2k3, it does drive resignaturing automatically because the new drive that is presented will have the exact same drive "serial number" and the OS won't use it unless it gets resignatured.  Also, ESX 3 will do lun resignaturing for you if you tell it to do that, do a search in the VMware forums for lun resignature and you'll find the exact syntax on how to permit that. 

Also, even though you won't consume storage by doing this, the volume wants to be able to handle the additional size requirements of the LUN.  So I've had to increase the size of the volume substantially to accomodate it, even though it doesn't use the space. 

If this sounds similar to flex clones, it is, but flex clones deals with cloning volumes, not luns, and flex clone is a separate license, so of course, extra money.

Hope this helps you guys out in the future.

Cheers,

Chris

Hello, its Friday again.

Each virtual machine needs to place their virtual disk files somewhere.  These typically have the extension vmdk or dsk (older versions or imported from VM Workstation or VM Server).  With Netapp, there's a certain art to where you place your disk files.  In this blog I'll go through four different options, and there may be more but I think these are the most common and reflect our evolution.

When we got our first Netapp, due to ease of use, we created one great big LUN and had all of our vmdk files on that one LUN.  The LUN was presented to three ESX Servers and everything ran harmoniously.  And then one day, someone (not me, I'd admit if it was), ended up fdisk'ing (formatting) our great big lun.  So, strange thing, most of the machines continued to operate and one by one they started dropping off.  At this point I was very new to Netapp and the main administrator of it.  I got the call, and after laughing, started looking at what could be done.  Of course, Netapp with its snapshots left us thinking this wouldn't be a big deal.  I did a snap restore and all the vmdk files came back, no one was the wiser.  Short note that these servers held no data, just configuration and would do things like serve up web pages and route mail.

As I started to gain experience with the Netapp, and looking at the benefits of it we started to wonder what would happen if we had to restore a single disk file?  All of our disk files were in one LUN, that LUN was contained in our Volume, and snapshots encompass entire volumes.  So if we wanted to restore one disk file, we would have to restore all the disk files.  Not very practical. 

So the next step was to create a separate volume for each virtual machine we hosted.  Each volume would have a LUN that would be dedicated to a virtual machine.  For example we would have a volume called vmmssql01 representing a Virtual Machine that was MS SQL, and it was the first MS SQL server in that location.  This way, we could snap restore an individual virtual machine without effecting the others.  Perfect. 

One issue with this is that Netapp has a limit on the number of volumes per filer, 200 comes to mind for some reason.  So is there another way of doing things in case we  have more then 200 virtual machines?  We weren't going to hit that but it was possible, on to the next method.

QTREEs in Netapp allow you to host pretty much any kind of data.  And they can contain LUNs as well.  So why don't we create a QTREE that contains LUNs, and we can do a snap restore on individual files, i.e. the LUN files.  We went down this path very briefly.  And a little more then a year later when we had to restore a LUN, it took forever, we ended up having to do a LUN Clone from a snapshot to get it up and running, more on LUN Clones in a future blog.  So LUNs in a QTREE were completely out of the picture.

At this point, we've settled on virtual machines having their own volume.  This has worked out the best but we are looking at future directions.  If you are in an organization that has dedicated storage administrators, then each time you want to create a virtual machine you have to interface with the storage department, get them to provision a Volume, LUN, do the rescan SAN on all the ESX Servers you are running, and then you can start your build.  Can be time consuming and a hinderance when you have an immediate need.

So, lets look at QTREEs again!  This time however, instead of presenting LUNs, lets present an NFS file system to the ESX 3 server.  This would only be for the Operating System drives of the servers.  This way, the NFS file system is hosted on the Netapp, it has snapshots for individual file restoration, and to create a new virtual machine, you don't have to talk to the Storage group.  The data drives for each of the servers would either be other NFS mount points for Linux boxes, a CIFS share for Windows boxes, and an iSCSI or LUN for an application like SQL that requries a block device. 

We are experimenting with this at present but it looks very viable.  Restores from a snapshot will not take that long, and its only the OS anyways, more restores happen on the data drives anyways.  Performance won't be an issue because you don't use the OS drive that frequently, and its not like you need more then 1Gbps anyways.  I'll write more about this as we get further into our testing process.

Each solution can provide different areas of opportunity for you depending on your own environment.  Keep in mind that Netapp says you shouldn't do snapshots on volumes containing LUNs, thats B.S., don't buy into it.  I can fail but it hasn't for us yet.  Also, don't be scared of Network Attached Storage just because you think it can't perform.  Show me where you are actually using more then 50% of the bandwidth of a Fibre Channel connected port, most of you won't be able to!

That's it for now, have a good weekend.