vElemental

I just completed the install of VPLEX and thought I would post a bit about preparing the FC infrastructure for a specific type of install.  I am not going to go into what VPLEX is here, other than it is a killer storage virtualization technology from EMC (who I work for) that allows for running stretched VMware clusters in an Active-Active storage array configuration.  This means we have multiple storage arrays under the covers and we can perform vMotions in order to move workloads between blocks of infrastructure (Network+Storage+Compute).  This helps for workload mobility, disaster avoidance, and can be complimentary to disaster restart scenarios (VMware SRM).  See teammate, Scott Lowe’s blog for some information on VPLEX.

A Quick Review of VPLEX Storage Objects – blog.scottlowe.org – The

A Deeper Look at VPLEX – blog.scottlowe.org – The weblog of an IT

Using VPLEX and Data Replication Together – blog.scottlowe.org

VPLEX – Virtualization and Private Cloud – EMC

Hardware Specs

• Hardware Version: VPLEX VS2
• Operating Mode: 2 x Single Engine (Metro), each engine attached to its own storage array
• FC Switches: 4x Cisco MDS 9148 5.0(4d)
• VPLEX Engine Communication: FC (IP is an option for module or FCIP through an ISL I believe)
• Storage Array: EMC VNX 5700

WARNING: These notes are my own from the install that I performed.  Please use this information simply as a reference.  If you find any information from EMC that conflicts with this, EMC’s information is of course correct.

I am assuming before you look through the following that you have unboxed, racked, stacked, and followed the directions from the awesome VPLEX manuals.  The rest of what you see will be focused on the FC infrastructure and ensuring that it is prepared zoned and configured appropriately to allow for communication between the VPLEX engines.  Zoning and FC configuration from the ESX hosts to VPLEX is very easy (when possible zone to less engines per host).  See the Implementation and Planning Best Practices for EMC VPLEX regarding VPLEX to storage array zoning since it depends on the storage array.

There are four categories of FC connectivity that occurs within and among the VPLEX engines.

• LocalCOM – Connectivity between directors among a cluster of VPLEX engines.  In a single engine case we are doing direct connect between VPLEX directors.  In a multi-engine cluster you would use an internal SAN switch (different from customer connected SAN switch) to provide connectivity in the backend between the directors of the cluster.
• WANCOM – FC/Ethernet connectivity between directors of different clusters (in our case FC, only in Metro and Geo)
• BE – Connectivity between VPLEX directors (initiators) and back-end storage (targets) that is to be virtualized
• FE – Connectivity between hosts (initiators) and VPLEX directors (targets)

An overview of what is to be covered

• Configuring WANCOM VSANs
• Creating an ISL
• Zoning WANCOM ports

Useful commands on MDS

• sh flogi databse – List the logged in ports and their associated WWN’s
• sh zoneset active – List the active zonesets and which ports are logged in
• sh fcalias – List the FCALIAS database to show device name to WWN associations
• sh fcdomain – Show the active FCDOMAIN’s for the configured VSANs

1) Configuring WANCOM VSANs

The below is assuming you will be switching out my VSAN numbers for your own.  What we are doing is pretty simple.  We are creating a VSAN on each of the switches and aligning this new VSAN between the primary switches of the blocks and the secondary switches of the blocks.  Once we create an ISL between said switches we will then be able to zone initiators and targets among the distinct blocks of infrastructure.  The only initiators and targets that will be zones are for the VPLEX unit (WANCOM ports for VPLEX metro FC communication).  We will not be zoning ESX hosts to adjacent infrastructure.  Make sure the VSAN numbers do not align with existing VSAN numbers in use!

“fcdoman 1 static vsan 101” – This line creates an FCDOMAIN of id 1 within the VSAN 101.  An FCDOMAIN is like a Layer-2 subnet in the FC world.  In order for an initiator to talk to a target on an opposite switch it needs to be in the same VSAN (if not Inter-VSAN routing) and it needs to exist in an FCDOMAIN in that switch.  In this case we join switches together by way of a VSAN and ISL (Inter Switch Link), and the initators talk to the targets via different FCDOMAIN (where the FC services handle what we would consider in Ethernet L3 routing to get between FCDOMAINS).  FCDOMAIN id’s are unique per VSAN.

“zoneset distribute full vsan 101” – To save on configuration we are telling the principal switch (lowest priority number) to distribute the zonesets to it’s subordinates

From BLOCK01 Switch-A

vsan database

vsan 101 name "VPLEX_WANCON_VSAN101"

fcdomain priority 1 vsan 101

fcdomain domain 1 static vsan 101

zoneset distribute full vsan 101

fcdomain restart vsan 101

From BLOCK01 Switch-B

vsan database

vsan 102 name "VPLEX_WANCON_VSAN102"

fcdomain priority 1 vsan 102

fcdomain domain 1 static vsan 102

zoneset distribute full vsan 102

fcdomain restart vsan 102

From BLOCK02 Switch-A

vsan database

vsan 101 name "VPLEX_WANCON_VSAN101"

fcdomain priority 2 vsan 101

fcdomain domain 2 static vsan 101

fcdomain restart vsan 101

From BLOCK02 Switch-B

vsan database

vsan 102 name "VPLEX_WANCON_VSAN102"

fcdomain priority 2 vsan 102

fcdomain domain 2 static vsan 102

fcdomain restart vsan 102

2) Creating an ISL

The below is assuming you will be switching out my VSAN number and FC ports for your own.  Here we are creating an Inter Switch Link between the primary/primary switches as well as the secondary/secondary.  This ISL will carry the VSAN traffic for the WANCOM connectivity as described above.

“switchport trunk allowed vsan 101” – Only allow advertise the 101 VSAN to the other switch.  This is critical to ensure we isolate production VSANs from each other across the ISL and prevent VSAN 101 isolation in the case of configuration problems.

From BLOCK01 Switch-A and BLOCK02 Switch-A

interface fc1/39

port-license acquire

channel-group 1

 

interface fc1/40

port-license acquire

channel-group 1

interface port-channel 1

switchport trunk allowed vsan 101

From BLOCK02 Switch-A and BLOCK02 Switch-A

interface fc1/39

port-license acquire

channel-group 1

 

 

interface fc1/40

port-license acquire

channel-group 1

interface port-channel 1

switchport trunk allowed vsan 101

Bring up all of the interfaces and port-channels on all switches

int fc1/39

no shut

int fc1/40

no shut

int port-channel 1

no shut

Check your port connectivity with a “sh int brief” command.  You should see the status of the physical ports and the port-channel at the bottom of the page as trunking.

Check your FCDOMAIN status to see if the switches have properly communicated and agreed on their priorities and roles.  As the below describes, you should see Switch-A and Switch-B in the first block being reported at the “Principal Switch” with the interfaces for ISL connectivity listed at the bottom showing the role as Downstream.

Switch-A of BLOCK01

As you can see below the switch in the adjacent block is showing his role as Subordinated Switch.”  This means that he will rely on the Principal switch to carry out domain services.

Switch-A of BLOCK02

3) Zoning WANCOM Ports

At this point we are assuming you have gone through the above and have Primary->Primary switch communicating and the FCDOMAINs working as described above.  We are also assuming that you have followed the VPLEX Configuration Guide up to the point where it has enabled the WANCOM ports.  The next step is then to create zones and zonesets on the primary switches that will then get distributed to the secondaries that allow for the WANCOM ports to communicate between VPLEX clusters.

Since VPLEX has enabled the WANCOM ports for FLOGI’s we can go to the switch and view the WWNs of those ports.  There are other means to get the WANCOM ports but this is how I typically find WWN port name WWN that are to be used for zoning.

When doing zoning I always use FCALIAS’s to simplify and make the zoneset’s a bit more dynamic and easier to read and understand.  By doing this I can easily issue a command like “sh zoneset active” and see the names of the WWN’s that are logged in or out.

fcalias name BLOCK2_VPLEX_WANCON_E1D1_p2_1 vsan 101  pwwn 50:00:14:42:70:02:59:20

fcalias name BLOCK2_VPLEX_WANCON_E1D2_p2_1 vsan 101  pwwn 50:00:14:42:60:02:59:20

fcalias name BLOCK1_VPLEX_WANCON_E1D1_p2_1 vsan 101  pwwn 50:00:14:42:70:4a:5e:20

fcalias name BLOCK1_VPLEX_WANCON_E1D2_p2_1 vsan 101  pwwn 50:00:14:42:60:4a:5e:20

And then I use those aliases in the zone’s.

zone name BLOCK1_VPLEX_WANCON_E1D1_p2_1_to_BLOCK2_E1D1_p2_1_E1D2_p2_1 vsan 101    member fcalias BLOCK2_VPLEX_WANCON_E1D1_p2_1    member fcalias BLOCK2_VPLEX_WANCON_E1D2_p2_1    member fcalias BLOCK1_VPLEX_WANCON_E1D1_p2_1

zone name BLOCK1_VPLEX_WANCON_E1D2_p2_1_to_BLOCK2_E1D1_p2_1_E1D2_p2_1 vsan 101    member fcalias BLOCK2_VPLEX_WANCON_E1D1_p2_1    member fcalias BLOCK2_VPLEX_WANCON_E1D2_p2_1    member fcalias BLOCK1_VPLEX_WANCON_E1D2_p2_1

zone name BLOCK2_VPLEX_WANCON_E1D2_p2_1_to_BLOCK1_E1D2_p2_1_E1D1_p2_1 vsan 101    member fcalias BLOCK2_VPLEX_WANCON_E1D2_p2_1    member fcalias BLOCK1_VPLEX_WANCON_E1D1_p2_1    member fcalias BLOCK1_VPLEX_WANCON_E1D2_p2_1

zone name BLOCK2_VPLEX_WANCON_E1D1_p2_1_to_BLOCK1_E1D2_p2_1_E1D1_p2_1 vsan 101    member fcalias BLOCK2_VPLEX_WANCON_E1D2_p2_1    member fcalias BLOCK1_VPLEX_WANCON_E1D1_p2_1    member fcalias BLOCK1_VPLEX_WANCON_E1D2_p2_1

We then take these zones and include them in a single zoneset.

zoneset name VSAN_101 vsan 101    member BLOCK1_VPLEX_WANCON_E1D1_p2_1_to_BERT_E1D1_p2_1_E1D2_p2_1    member BLOCK1_VPLEX_WANCON_E1D2_p2_1_to_BERT_E1D1_p2_1_E1D2_p2_1    member BLOCK2_VPLEX_WANCON_E1D2_p2_1_to_ELMO_E1D2_p2_1_E1D1_p2_1    member BLOCK2_VPLEX_WANCON_E1D1_p2_1_to_ELMO_E1D2_p2_1_E1D1_p2_1

We then take that zoneset and activate it to the VSAN with the following command “zoneset activate name VSAN_101 vsan 101”.  I am going to paste the same code for the other SWITCH-B of the BLOCK1 below.

fcalias name BLOCK2_VPLEX_WANCON_E1D1_p2_0 vsan 102    member pwwn 50:00:14:42:60:02:59:21

fcalias name BLOCK2_VPLEX_WANCON_E1D2_p2_0 vsan 102    member pwwn 50:00:14:42:70:02:59:21

fcalias name BLOCK1_VPLEX_WANCON_E1D1_p2_0 vsan 102    member pwwn 50:00:14:42:70:4a:5e:21

fcalias name BLOCK1_VPLEX_WANCON_E1D2_p2_0 vsan 102    member pwwn 50:00:14:42:60:4a:5e:21zone name BLOCK1_VPLEX_WANCON_E1D1_p2_0_to_BLOCK2_E1D1_p2_0_E1D2_p2_0 vsan 102    member fcalias BLOCK2_VPLEX_WANCON_E1D1_p2_0    member fcalias BLOCK2_VPLEX_WANCON_E1D2_p2_0    member fcalias BLOCK1_VPLEX_WANCON_E1D1_p2_0

zone name BLOCK1_VPLEX_WANCON_E1D2_p2_0_to_BLOCK2_E1D1_p2_0_E1D2_p2_0 vsan 102    member fcalias BLOCK2_VPLEX_WANCON_E1D1_p2_0    member fcalias BLOCK2_VPLEX_WANCON_E1D2_p2_0    member fcalias BLOCK1_VPLEX_WANCON_E1D2_p2_0

zone name BLOCK2_VPLEX_WANCON_E1D2_p2_0_to_BLOCK1_E1D2_p2_0_E1D1_p2_0 vsan 102    member fcalias BLOCK2_VPLEX_WANCON_E1D2_p2_0    member fcalias BLOCK1_VPLEX_WANCON_E1D1_p2_0    member fcalias BLOCK1_VPLEX_WANCON_E1D2_p2_0

zone name BLOCK2_VPLEX_WANCON_E1D1_p2_0_to_BLOCK1_E1D2_p2_0_E1D1_p2_0 vsan 102    member fcalias BLOCK2_VPLEX_WANCON_E1D2_p2_0    member fcalias BLOCK1_VPLEX_WANCON_E1D1_p2_0    member fcalias BLOCK1_VPLEX_WANCON_E1D2_p2_0zoneset name VSAN_102 vsan 102    member BLOCK1_VPLEX_WANCON_E1D1_p2_0_to_BLOCK2_E1D1_p2_0_E1D2_p2_0    member BLOCK1_VPLEX_WANCON_E1D2_p2_0_to_BLOCK2_E1D1_p2_0_E1D2_p2_0    member BLOCK2_VPLEX_WANCON_E1D2_p2_0_to_BLOCK1_E1D2_p2_0_E1D1_p2_0    member BLOCK2_VPLEX_WANCON_E1D1_p2_0_to_BLOCK1_E1D2_p2_0_E1D1_p2_0

zoneset activate name VSAN_102 vsan 102

Now let’s see if our handwork paid off? Issue a “sh zoneset active” command.

It should look like the following on both switches!  The star represents that the port performed a FLOGI and is currently logged into the fabric.  If you are not seeing stars then you need to go back to the basics and ensure the you know the proper WWN (look it up with a sh flogi database command to verify port-> wwn association) and the port is active.

And that’s it for zoning the WANCOM ports.  The information here might help you through the zoning of the BE/FE ports as well which is pretty straight forward from an FC perspective.  Feel free to shoot me any comments you may have and hopefully this helps in some way with any of your FC work!