vElemental

Isn’t it awesome when you need something, know it will eventually arrive, but need it now anyways?  That’s a bit of what the information in the post is all about, but here goes anyhow.

First the apology =) Unfortunately I wrote way too many cmdlets recently for vCD, so I really don’t have enough chapters or cycles to get into the details around each cmdlet.  I tried to include examples and descriptions for the cmdlets, but didn’t quite make it through all of them.  It should be very self explanatory when looking at the name and the run-time example.

The goal of the cmdlets was to build the framework necessary to complete a whole bunch of automation with vCD, satisfying a primary use case for labs (actual lab cmdlets not here).

There is a lot of pretty cool stuff in here, some of the highlights are as follows.

 

Network

There are a handful of network related cmdlets.  An important one for any VApp deployment is to be able to specify the External Network correctly (Update-CIVAppParentNetwork).  In addition when a VApp is deployed you probably want to see the accessible remote IP (Get-CIRouterInfo).  A new twist on the VApp networking has to do with the fencing that can take place.  There is a cmdlet to view the fencing information (Get-CIFencing) for CIVMs to show their inside and outside mac addresses.  Another useful one is (Get-CIVAppNetworkingFirewallRules) which displays the firewall rules associated to a VApp.

Disk

One of the great things about vCD is the use of thin and linked clones (fast provisioning).  However, the reporting on a per object basis isn’t quite there.  For this we leverage the Get-CIVCVM to cross-link the CIVMs to their respective VC VM’s to get details per virtual disk (Get-CIVAppDiskGroupings, Get-CIVAppDiskUsage).  This is all fine and great, but I wanted to take a step even further than this and describe the relationship among VMs in the fast provisioning world (Get-CIVMDiskFileChain,Get-CIVMDiskFiles).  These commands will list out the relevant virtual disks and show the amount of VM’s sharing those virtual disks and the chain length.

VApp/VAppTemplates and Catalogs

Note that due to the complexity of deployment the Instantiate-VApp cmdlet has been changed.  In a previous post I demonstrated how you might be able to specify all of the relevant details (NAT, Firewall, etc) on the fly.  This however is a bit tough to get going correctly and for most not necessary.  So instead I opted to go for the “Clone” route which takes an existing configuration contained in a VAppTemplate and duplicates as much of it as possible versus specifying these things manually.  I have also included another goodie (DeployLab-FromOrgCatalogToAnotherOrgCatalog) which demonstrates the VC to VC scalability of vCD without using the OVF export routine (future post to explain).  Another good one is the ability to add VApps to catalogs and duplicate VAppTemplates among catalogs (Add-CIVAppTemplateToCatalog).  Last but not least, in order to leverage VC to VC transfers correctly we have included the ability to consolidate VAppTemplates in the catalog (Consolidate-CIVAppTemplate).  A new concept for vCD are shadow VMs.  These are used to track back to original VApp Templates and can be difficult to track down at times so there is a cmdlet for this as well (Get-CIVMShadow).

WARNING: The VApp/VAppTemplate cmdlets allow you to do things that might bring you outside of your “workspace” which could cause OVF related activities (not good for most cases, lose VM identities, etc).  Watch VC to see what is happening during these processes to ensure this is not the case.

VC Related

When running large scale environments there are always key things that need to be aligned.  I created a couple of easy ones (Get-VMHostDateTime and Get-VMHostSoftwareList) to help out with ensuring the hosts are aligned from a time perspective, and verifying all hosts were running the same vcloud agent version.  You will notice that there are a handful of cmdlets that leverage a VC connection (Connect-VIServer) in order to work correctly.  These are where I lookup the underlying VI information for an object to give a more complete view of something.  These hybrid views have turned out to be very powerful.

 

Anyways, have fun with the 43 cmdlets!  The following is unsupported and probably soon to be replaced =)

 

• Install PowerCLI 5.0.1, with Cloud
• Download the Cmdlet module here.
• Import-Module .\custom_vcd_cmdlets.psm1
• Connect-CIServer
• Connect-VIServer on some cmdlets

 

Remove-Catalog

Get-Catalog -name Catalog01 -org myOrg | Remove-Catalog -Confirm:$False

New-Catalog

New-Catalog -name Catalog01 -Org myOrg -isPublished:$true

Remove-CICustomProperties

Get-CIVApp Google* | Get-CIVM name | Remove-CICustomProperties

Get-CIDatastore

Get-ProviderVdc Google* | Get-CIDatastore

Get-CIMetadata

Get-CIVApp Google* | Get-CIMetadata

New-CIMetadata

Get-CIVApp Google* | Set-CIMetadata -key "test" -value "testvalue"

Remove-CIMetadata

Get-CIVApp Google* | where{$_.key -eq "test"} | Remove-CIMetadata

Update-CIMetadata

Get-CIVApp Google* | where{$_.key -eq "test"} | Update-CIMetadata -value "updated"

Get-CIRouterInfo

Get-CIVApp Google* | Get-CIRouterInfo

Update-CIUser

Get-CIUser labsorg | Update-CIUser -description 1

Remove-CIVApp

Get-Org Labs | Get-CIVApp Base-ePod-Basic-v8 | Remove-CIVApp

Start-CIVApp

Get-CIVApp Google* | Start-CIVApp

Get-CIVAppDiskGroupings

Get-CIVApp VApptest* | Get-CIVAppDiskGroupings

Get-CIVAppDiskUsage

Get-CIVApp test* | Get-CIVAppDiskUsage

Update-CIVAppParentNetwork

Get-CIVApp Google* | Update-CIVAppParentNetwork -parentnetwork (Get-Org (Get-OrgVdc ovdc_labs_c03).org | Get-OrgNetwork orgnetwork_external01_c03)

Get-CIVAppResourceUsage

Get-CIVApp test* | Get-CIVAppResourceUsage

Consolidate-CIVAppTemplate

Get-CIVAppTemplate Base-ePod-Basic-v8 -Catalog (Get-Org HOLDEV | Get-Catalog HOL_BASE_CAT01) | Consolidate-CIVAppTemplate

Remove-CIVAppTemplate

Get-CIVAppTemplate Base-ePod-Basic-v8 -Catalog (Get-Org HOLDEV | Get-Catalog HOL_BASE_CAT01) | Remove-CIVAppTemplate

Update-CIUser

Get-CIUser labsorg | Update-CIUser -description 1

Get-CIVAppTemplateDiskUsage

Get-CIVAppTemplate test* | Get-CIVAppTemplateDiskUsage

Get-CIVAppTemplateResourceUsage

Get-CIVAppTemplate test* | Get-CIVAppTemplateResourceUsage

Add-CIVAppTemplateToCatalog

Get-CIVAppTemplate Google* | Add-CIVAppTemplateToCatalog -VAppTemplateName test1 -Catalog (Get-Org labs | Get-Catalog test) -ToOrgVdc (Get-OrgVdc test)

Get-CIVAppTemplateTotalUsage

Get-CIVAppTemplate test* | Get-CIVAppTemplateTotalUsage

Get-CIVAppTemplateVM

Get-CIVAppTemplate Google* | Get-CIVAppTempalteVM

Add-CIVAppToCatalog

Get-CIVApp Google* | Add-CIVAppToCatalog -VAppTemplateName test13 -Catalog (Get-Org labs | Get-Catalog test) -OrgVdc (Get-OrgVdc OvDC_LABS_C03)

Get-CIVAppNetworkingFirewallRules

Get-CIVApp Base-ePod-Basic-v8 | Get-CIVAppNetworkingFirewallRules

Get-CIVAppTotalUsage

Get-CIVApp test* | Get-CIVAppTotalUsage

Get-CIVCVM

Get-CIVM test* | Get-CIVCVM | Get-VMDiskFiles

Get-CIVMDiskFileChain

Get-CIVM "vpodrouter v3" | Get-CIVMDiskFileChain

Get-CIVMFencing

Get-CIVApp | Get-CIVM | Get-CIVMFencing

Get-CIVMHardDisk

Get-ProviderVdc Google* | Get-CIDatastore

Get-CIVMShadow

Get-CIVMShadow

DeployLab-FromOrgCatalogToAnotherOrgCatalog

Get-CIVAppTemplate HOL12-HyperV-SPwithAvamar-v9-Clint -Catalog (Get-Catalog HOLDEV_HOL12_CAT01) | DeployLab-FromOrgCatalogToAnotherOrgCatalog -ToOrgVdc @("OvDC_LABS_C01","OvDC_LABS_C02","OvDC_LABS_C03","OvDC_LABS_C04") -ToOrgVdcMC OvDC_HOLDEV_HOL12_C01_MC01 -prefix "MC_" -fromCatalog "HOLDEV_HOL12_CAT01" -ToOrg "Labs" -verbose

New-Org

New-Org -Name name -FullName fullname -Enabled:$true -PublishCatalogs:$true

New-OrgGroup

New-OrgGroup -Name name -GroupName GroupName -RoleName RoleName

Set-OrgLDAP

Set-OrgLDAP -name  orgName –orgLDAPSystem:$true

New-OrgVdc2

New-OrgVdc2 –name fromOrg –FromOrgVdcName fromOrgVdc –orgVdcName newOrgVdc

Instantiate-VApp

Instantiate-VApp -name "test" -catalogname HOLDEV_HOL22_CAT01 -VAppTemplateName HOL22-Test-v1 -orgvdcname OvDC_HOLDEV_HOL22_C01 -mode Clone -poweron:$false -deploy:$false

Get-VMDiskFiles

Get-CIVM test* | Get-CIVCVM | Get-VMDiskFiles

Get-VMDiskUsage

Get-CIVM test* | Get-CIVCVM | Get-VMDiskUsage

Get-VMHostSoftware List

Get-VMHost | Get-VMHostSoftwareList | where {$_.name -match "vcloud"}

Get-VMHostDateTime

Get-VMHost -server vc01 | Get-VMHostDateTime

During the past couple of weeks we have been going full tilt building labs for EMC World 2012.  So far this has been great on all sides and I have enjoyed getting stick time in the weeds with a whole lot of technology that I don’t typically get extremely deep with.  In addition to getting deep and mostly fixing stuff, I am also in a position where I can pull back the onion a bit and build cmdlets and put information out there that might be useful to the general VMware community.  This post is going to dive a bit into the concept of an OVF file in vCD and what you need to know with vCD 1.5.

What the heck is an OVF?  An OVF is an open standard for distributing virtual machines.  The open standard should help standardize the formatting of a virtual disk, describe how a virtual machine needs to be configured to be re-assembled in other systems, and possibly describe an application configuration or custom properties that can be configured to make the VMs and vApp unique and/or function.  This open standard should make it a snap to transfer in between different hypervisors, and hopefully first off within the same family of virtualization technology.

So if you’re like me then you probably expected that an OVF is the same to vCenter/vSphere as it is to vCD.  Well, at the moment it’s not so and a vApp is currently (v1.5) technically not the same =( 

This post should provide an option for navigating this mess and at least being able to get OVF’s constructed as packaged vApps to function in vCD. 

vCloud itself has native functionality to import OVFs as vApps.  The following screenshot was taken by navigating to a catalog, right clicking anywhere inside of the open catalog inventory and selecting upload.  You can see that a Java window opens with the following questions around  he OVF package location, name, and description.  Upon selecting the appropriate options and saying upload, the transfer process begins where the virtual disks from the OVF are copied to the transfer directory of the vCD cell, configuration is consumed,  followed by another copy up to a provider vDC datastore, and a final configuration of the vApp.

In comes the problem with OVFs!  In order to understand this we need to see how OVFs are generated.  I am going to present this in two export methods from VMware technologies.  First though, the native vCD “Download” functionality where you can simply just right click a vApp Template and save the descriptor.ovf and associated files to a directory using the Java applet from vCD.  This *should* be the most native way to reimport a vApp to vCD.  The second way is more the legacy method where you export a vApp/VM from vCenter and/or possibly use OVFTool to construct a vApp where you assign custom properties that a user should answer during the import process.  The custom properties I am referring to may be to accept a license agreement, the hostname, IP address, subnet, gateway, etc. of VMs being deployed.  These custom properties are where the problem comes into play. 

Warning: What I am describing is something that I ran into so there are probably more ways to get around this and I probably am not presenting all the information correctly, but here goes..

The custom properties that are associated with an OVF seem to somewhat be incompatible (v1.5) with a standard “Upload”.  If there are custom properties, or internally called the “ProductSection,” the upload may fail with the following errors and screenshots.

“Error: Validation failed for the OVF file you provided”

“complex-type2.4.a: Invalid content was found starting with element”

“Error: The manifest of the OVF package is invalid”

 

So what do you do in this case?  The easy way that I have gotten around this is to first import the OVFs into vCenter and then utilize the “Import from vSphere” functionality to get the individual VMs into a vCD vApp.  This will allow you enter the OVF custom properties via vCenter as you normally would.  However, keep in mind that the custom properties will still be associated to this VM, but you shouldn’t have any problem getting the VM imported this way.  I suggest booting the VM up and then stopping it in vSphere following the standard “Deploy OVF Template” process.

Now you may be thinking, hey that isn’t so bad?  I got my VMs into the vCD vApp! Well it gets better.  As mentioned before the custom properties may still persist so the problem can now effect the startup of the VM in vCD since it doesn’t play nice with some of the custom properties from VMs brought into vCD from vSphere and previously imported as OVFs.  So what to do?  Pop open your vApp in vCD, right click and go to properties.  Do you see custom properties listed?  If so then you might have a problem.  Try filling these in and try and boot the VM.  Like Below?  Well for this of course we have Powershell and PowerCLI to the rescue!  Below is an error that you may run into if there is invalid custom properties.  Notice the “Invalid vApp Properties: Unknown property” error.  This is followed by other errors where the process then tries to roll itself back and runs into errors doing this.  Trying to start a vApp unsuccessfully may lead to having to go to vSphere and manually stop the other VMs in the vApp and/or delete a portgroup manually.

Here is a quick look at what custom properties or “ProductSection” could possibly cause problems inside of the OVF file.  Can you modify the OVF file ahead of time to make this easy?  Probably, but I decided to go about it a different way.

 

So as usual I have built a custom cmdlet using the PowerCLI cloud methods.  Following are the requirements.

• PowerCLI 5.0.1 (Cloud selected on install)
• Download the new custom powershell module here.
• Run Connect-CIServer vCDInstance
• Run Import-Module vcd_custom5.psm1

 

*New Remove-CICustomProperties* This cmdlet will remove custom properties other than a single “ProductSection” relevant to the VM itself.  Use at your own risk!

Example- Remove Custom Properties from a Virtual Machine named “name” in a vApp starting with “Google”.

Get-CIVApp Google* | Get-CIVM name | Remove-CICustomProperties

The following screenshot is what a VM in a vApp inside of vCD should look like from the “Custom Properties” tab.

 

 

And that’s it!  From there you should have no problem starting this VM.  If you are running into errors around the dvSwitch then you can either manually remove the associated dvPortGroup, create a new vApp and move the VMs into this vapp, or check the vApp into the catalog and redeploy it from the catalog as a new vApp.

 

 

 

 

 

 

 

I came across a post from William Lam today based on working with Metadata for VMware vCloud Director objects referenced here.  This lit a bit of a spark which got me thinking about how mature the underlying assemblies were with vCD in PowerCLI.  The jist of the debate here is that William is leveraging the RESTful (simply web POST/GET/PUT) capabilities of the vCD API, whereas the Cloud portion of PowerCLI is built to do all of the RESTful calls for you as well as setting easy to use types.  The downside to this approach is that you lose flexibility with PowerCLI whereas if there is a bug or something didn’t get implemented in PowerCLI then it will still need to be called from the REST interface.  So far in my recent endeavors I have only run into a single situation (around a bug in NAT for vApps), other than that it’s been very mature from a method perspective.

What does this mean?  Well I’ll have you read through his post, and even give it a shot =) Then give what you see below a try and it might shed some light on how effective good frameworks can be on top of APIs.  This goes back to one of my philosophies around APIs.. they’re great, but if you truly want adoption you need to write frameworks or command line functionality on top of it to make it scalable.  Without this it is only for the 1%’ers who have the time to read the docs and can do those types of calls. =)

What you’re going to find below are four new cmdlets for PowerCLI.  As usual see the requirements.

• Downloaded the module here.
• PowerCLI 5.0.1 with Cloud selected on install
• Import-Module vcd_custom4.psm1
• Connect-CIserver vCDInstance

 

*New New-CIMetadata* Create new metadata associated to whichever vCD object is passed.  See William’s post as mentioned at the top for other links around which objects are actually valid to tag Metadata to.

*New Get-CIMetadata* Retrieve metadata for a specific vCD object.

 

*New Remove-CIMetadata* Remove metata assocaited to whichever vCD object is passed.

 

*New Update-CIMetadata* Updates metadata for a vCD object’s own metadata.

 

 

Now if you’re interested in how we made this happen keep on following on.  For those that just want the unofficial metadata cmdlets you should be good to go.

When exploring for unadvertised features in PowerCLI the first thing I always do is look through the views or ExtensionData from the object.  You can see this below where I issue a “| gm *” after the view which is short for Get-Members *.  The returned information is based on available methods and properties.  The methods are all callable based on the definitions.  This is the first hint that there is a lot of beef behind the Cloud assemblies for PowerCLI.  Looking through the list you can see the GetMetadata methods, perfect!

With that method found, let’s start to use it.  The following screenshot is a call to the method.  You can see that we then have the metadata keys returned under the MetadataEntry parameter.

However, what’s important here is that we are looking for something that might say “UpdateServerData.”  If this was there I would be able to set the parameters directly and then call this method to actually post the changes.  In this case that method doesn’t exist and instead there is actually one called SetMetadataValue.  Wallah, there you go.

Here are the specific Metadata entries that were stored in the MetadataEntry parameter.

And the next section is actually a sample of what I would do to create a new metadata item.  Notice how when we call SetMetadataValue for this first time it tells us the required parameters (key,value).  In the next line we call it as it required and then check to see if it exists.  Perfecto, all done.

 

 

All in all I want to thank William and the PowerCLI team for implementing a lot of the vCD API.  Although the cmdlets aren’t quite fully baked yet it looks like we will have a pretty rocking Powershell interface to vCD in the short term!

As you can tell from the title this post is another (short) one in the easy button series for simple repeatable tasks in VMware vCloud Director leveraging PowerCLI and methods available with the default views.  This one specifically is important if you are a service provider and wishing to create catalogs for customers on the fly or other use cases that are similar.  Also, if you have the need to simply automate a whole bunch of monotonous tasks for setting up development areas for labs then this could help as well =)  This is the 4th post in the series, and yes I will be combining all of the automation work together for vCD into a single module.  I may even go as far as creating a module specifically dialed in around the vCD Lab use case!

So here we go, as usual see requirements below. 

• PowerCLI 5.0.1
• Download the module here.
• Import-Module vcd_custom3.psm1
• Connect-CIServer vCDinstance

*New New-Catalog*

Creates a new catalog by specifying the name of the catalog, the name of the organization and whether to publish (not in screenshot).

*New Remove-Catalog*

Removes an existing catalog from an organization.

*New Add-CIVAppToCatalog*

Adds an existing VApp to a catalog.

 

That’s all for now.

My most recent project has been a pretty fun one.  As you might have been able tell so far it has to do with automating VMware vCloud Director.  I have been focusing pretty heavily on some of the more common use cases, see this post around possibly signing up new organizations here, and here where we deploy a vApp from a catalog.  This post is going to continue the deployment route by making the functions a bit more advanced in the networking area.

When deploying applications in vCD the combinations of external, organizational, and vApp networks are limitless.  Depending on what the target operating environment is you might need to apply NAT at multiple levels or simply just bridge the networks together which would mitigate any networking multi-tenancy concepts.  Other specific use cases may specify that you not create multi-tenant organizational network divisions, but more specifically a NAT per deployed vApps.  This use case may be aligned towards a lab (hmmm… what’s coming up in May for EMC? And what do we offer there in massive scale?).  Thanks Clair Roberts, Curtis Pope, Everett Golden @VMware Integration Engineering for providing some guidance here. 

In proceeding down this path of leveraging vCD for possibly a lab environment I was able to look at a successful lab scenario (VMworld 2011) that leveraged vCD and looked into what it might take to achieve the same automation in Powershell.  To be successful it is hard to communicate the amount of detail that goes into the backend equipment and vCD design.  As far as vCD we break this into two areas, one for how we support the development of the labs and one for the operational aspects of active labs.

This post is going to demonstrate how we can provision from a catalog including the vApp Networking details that apply NAT for obfuscating vApp VM details, and firewall policies for allowing RDP connections into the vApp.

So let’s dig into it, get the next version of the functions for Instantiation-VApp cmdlet here. ***

The new options that are included in this version have to do with the vApp networking for DHCP, Firewall, and NAT services.  See the example parameters below when deploying a vApp.  If you are not concerned about the details around how the vApp networking is configured, or you are going to leverage NAT possibly at the organizational level then you can skip everything at the –fencemode and below lines which aligns more with the previous post.

# Instantiate-VApp -name "test" -catalogname HOLDEV_HOL22_CAT01 -vappname HOL22-Test-v1 -orgvdcname OvDC_HOLDEV_HOL22_C01 -hashNetworks @{0=@{sourceNet="orgNetwork_EXTERNAL01_HOL22_C01";targetNet="orgNetwork_EXTERNAL01_HOL22_C01"}} -fencemode "bridged"

# Or

# Instantiate-VApp -name "test" -catalogname HOLDEV_HOL22_CAT01 -vappname HOL22-Test-v1 -orgvdcname OvDC_HOLDEV_HOL22_C01 -hashNetworks @{0=@{sourceNet="orgNetwork_EXTERNAL01_HOL22_C01";targetNet="orgNetwork_EXTERNAL01_HOL22_C01"}}

#    -fencemode "natRouted" -natRoutedIpScope @{IsInherited = $False;Gateway = "192.168.100.1";Netmask="255.255.255.0";Dns1=$null;Dns2=$null;DnsSuffix=$null}

#    -IpScopeIpRanges @(@{startAddress="192.168.100.100";endAddress="192.168.100.199"})

#    -DhcpEnabled $False

#    -DhcpIpRange @{startAddress="192.168.100.2";endAddress="192.168.100.99"}

#    -DhcpDefaultLeaseTime 3600

#    -DhcpMaxLeaseTime 7200

#    -FirewallEnabled $True

#    -FirewallRule @(

#       @{IsEnabled=$True;Description="RDP";policy="allow";protocols=@{any=$null;icmp=$null;tcp=$True;udp=$null};port=-1;destinationip="Any";sourceport=3389;sourceip="Any";direction="in";enableLogging=$False},

#       @{IsEnabled=$True;Description="Allow all outbound traffic";policy="allow";protocols=@{any=$True;icmp=$null;tcp=$null;udp=$null};port=-1;destinationip="Any";sourceport=-1;sourceip="Any";direction="out";enableLogging=$False}

#       )

#    -Nat @{NatType="ipTranslation";Policy="allowTraffic"}

So in summary, if you use the parameters above when deploying from a catalog you will be deploying a vApp with a vApp networking leveraging the “natRouted” mode to provide separation from other vApps.  All connectivity to the outside of the vApp will be provided by the “orgNetwork_EXTERNAL01_HOL22_C01” network which can be anything.  See the previous post listed above if you need more details on the parameters before “-fencemode”.

This post is a follow up to yesterday’s post where I continue to explore the vCD PowerCLI cmdlets and generate new one’s to fit my use cases.  See yesterday’s post around requirements as they are the same.  Attached here you will find another Powershell module with a new function in it. 

*New Instantiate-VApp* This function looks for a specific VApp Template that resides in a catalog of an organization and creates a new VApp in a specific Organizational vDC.  The following is what the parameter section looks like for runtime options.

    Param (

        $Name=$(throw "need -name"),

        $CatalogName=$(throw "need -catalogname"),

        $VAppName=$(throw "need -vappname"),

        $orgVdcName=$(throw "need -orgVdcName"),

        $hashNetworks = @{},

        $LinkedClone=$true,

        $AllEULAsAccepted=$true,

        $IsSourceDelete=$false,

        $Deploy=$true,

        $PowerOn=$true

    )

The parameters are pretty self explanatory, but if needed you can get some more information from the official vCD REST API guide here on page 33.  The –name parameter is for the new vApp, and the VAppName is the template name.  The piece that needs some expanding on is the $hashNetworks portion.  Here we are using a hash table that is issues as follows.

-hashNetworks @{0=@{sourceNet="ALL_PURPOSE";targetNet="BBS_external01"}}

The important thing here is that we need to map the VApp template network connections to the proper network in the destination Org vDC.  For this we create a hash table based on an integer index (unique network names, not network adapters), and sourceNet/targetNet parameters under those indexes.  So in essence, an array of hashTables with sourceNet/targetNet specified for mapping the two.  A two index hash table would look like the following.

-hashNetworks @{0=@{sourceNet="ALL_PURPOSE";targetNet="BBS_external01"};1=@{sourceNet="Internal-NoUplink";targetNet="BBS_internal01"}}

The following is a screenshot of the script in action where we actually can choose not to specify –hashNetworks if we want to specify this manually after deployment or we are deploying to a vDC that has the network names that the vApp template currently has.

 

Otherwise we would run the command like follows.

 

In the good spirit of leaving bread crumbs for all of those fellow Powershellers out there, the function that I built leverages the calls that are a part of the OrgVdc object.  This functionality is exposed via issuing a Get-OrgVdc cmdlet follwed by calling the Get-CIView cmdlet and then a method that is available called InstantiateVappTemplate.  Previous to the method being called, we also built out the $paramsVAppT object to contain necessary required fields.  These required fields were pulled and transcoded pretty much line for line from the official vCD REST API guide here on page 33.   Hopefully following this function you can either make this cooler, or implement completely different use case.

(Get-OrgVdc $orgVdcName | Get-CIView).InstantiateVappTemplate($paramsVappT)

I just recently had a chance to play around a bit with the vCD PowerCLI cmdlets.  So far I have mixed reviews about what I have found, but it is a good start.  There is a basic level of functionality exposed through cmdlets and a bit more that takes some slightly advanced Powershelling to expose.  As usual the more advanced features can be accessed by using the Get-CIView cmdlet or $object.ExtensionData and using methods directly exposed from these vCD objects.  

Alan has a good post on some basic functions that can help to achieve some basic things all leveraging exposed methods under the covers.

http://blogs.vmware.com/vipowershell/2012/03/automating-creation-of-vcd-organizations-users-and-org-vdcs.html

My intent for this post is to add a few more cmdlets to the public that may be useful for certain circumstances and might even be more useful with a bit of tweaking for yourself.  My use case here was to be able to create an organization, an organizational vDC, create groups for the organization, and set LDAP appropriately to allow an LDAP security group access to that organization.

Requirements are as follows:

• PowerCLI 5.0.1 (with vCD cmdlets selected on install)
• The unofficial vCD module I created here.
• Run “Connect-CIServer”
• Run “Import-Module vcd_custom.psm1”

 

*As Is* New-Org The following screenshot is based on work from Alan Renouf from the link listed above where we simply create an Org.

 

*New* Set-OrgLDAP The following cmdlet will set the SYSTEM LDAP setting for an Org.  Before running this we are expecting that you have LDAP configured on the main portion of the vCD Administrator page (outside of an Org) and that you have used the test LDAP functionality in the setup area.  The SYSTEM setting leverages the existing LDAP information from the main vCD instance for the child Org.

 

*New* New-OrgGroup The next portion here will create a new organization group based on a certain LDAP security group name and an existing vCD role name.  I haven’t had a whole lot of time to test this part, but I did find that in order to get the LDAP sync’d for the org created I had to restart vCD.  Unfortunately I am using 1.5 and not 1.5.1 for specific reasons so am not sure if this is a bug or if I need to wait an indicator saying LDAP group adds are ready.

 

*Modified* New-OrgVdc2 The following cmdlet is a spin off of another one that Alan created except I am using a different parameter, –FromOrgVdcName which will allow us to specify a model OrgVdc.  This simplifies the parameters specified since we copy the important stuff from an existing OrgVdc.

 

 

That’s it!  Pretty quick and dirty examples of how to automate some more detailed setup tasks.  I would assume we will see a lot of this stuff get rolled out in the near term in official cmdlets, so this should only be temporary.  Stay tuned to the blog as I have a good amount of vCD automation work that might be popping up here shortly.

In this post I am going to demonstrate how you would go about multi-threading the execution of a PowerCLI commands.  As usual I want to make sure that I properly set the stage and describe my use case because this might not be applicable in all situations. 

When running PowerCLI cmdlets, some include the ability to asynchronously run which means that they get sent to the background and processed while allowing to run more commands.  This parameter can be invoked by specifying “-runasync:$true”.  This can be very handy if I have a single PowerCLI cmdlet that I want to execute many times with a predetermined set of data, for example to stop all VMs as quicky as possible  “get-vm | Stop-VM -runasync:$true -confirm:$false”.  But what about a situation where I might have a set of commands that need to be run asynchronously?

This is where leveraging the jobs portion of Powershell to launch multiple PowerCLI sessions helps.  With this method we are reusing the existing PowerCLI session token and launching new instances of PowerCLI  to accomplish many tasks in sequence that are all tied together possibly similar to a workflow.

So if you’re interested in seeing this in action, continue on.  Firstly the below is a summary of some of the useful tidbits you will see in this example.

• Optimizing Powershell and PowerCLI for speed
• Reusing VIServer credentials based on session token
• Executing differently if PowerCLI is loaded and/or credentials specified
• Multi-threaded PowerCLI

 

One of the goals of multi-threading is to be able to perform more actions than executing in a serial fashion.  For this reason I assume that you are looking to minimize the average time per execution of PowerCLI.  So to begin with I highly suggest reading a previous post of mine which walks you through how to optimize Powershell and PowerCLI execution times here.  This also has a huge effect on the CPU as if you do not optimize, when PowerCLI starts it will perform compilation operations that eat a whole core of your CPU for 10+ seconds.

Now with that behind us let’s get into the details of credentials.  When logging into a VIServer in PowerCLI the session creates and stores a specific token secret token.  See the screenshot below for retrieving this token.

We will be reusing this token by passing it to the script that is the target of our multi-threaded execution (in the job) which replaces the need to re-login to create multiple sessions under the same username or pass credentials to the job.

The next important demonstration is pretty common and others may have different approaches.  It involves being able to load/not-load PowerCLI and Connect/Not-Connect based on whether it is needed or not.  For example, if we launch the child job script then we will need to load the PowerCLI snappin on the fly.  But if we want to keep the script the same even if PowerCLI is already loaded (for execution outside job)  then we want to do a simple detection for the existence of the cmdlets. 

The following is a re-useable section of code that you can use to simplify detection of PowerCLI and existence of a login.

if(!$global:DefaultVIServer) {

      if(!(get-pssnapin | where {$_.name -eq "vmware.vimautomation.core"})) {

        try {

            add-pssnapin VMware.VimAutomation.Core -ea 0| out-null

        } catch {

            throw "Could not load PowerCLI snapin"

        }

    }

    try {

        connect-viserver $vcserver -username $vcusername -password $vcpassword -wa 0

    } catch {

        throw "Failed to connect to VI server"

    }

}

 

Ok so now let’s move on to the more exciting stuff around multi-threading.  The first portion of what we do is set our current path variable.  This is necessary because when loading a new Powershell instance the current path changes to your profile’s set path.  What we want is to set the path based on where the script is being executed from.  For this we issues to following command in the control script and then pass it to the job.

 

$currentPath = (Split-Path -parent $MyInvocation.MyCommand.Definition)

 

So you may be asking, why would we not just use the invoke-item cmdlet or have the job kick off the script without the new Powershell session?  In testing I found that I was getting random errors from PowerCLI cmdlets and methods.  I am not 100% sure why but I believe it has to do with the jobs executing with a console.  Creating a new Powershell instance that PowerCLI can execute under solves this.  Keep in mind the memory usage per thread which can be 60+MB.

Anyways, the next portion is the scriptblock that will be executed when we kick off the job.  Notice the first line, “set-location $args[0]”.  This takes the path that we got from above and has the child kob instance reset its current working directory.  We are passing two parameters to the script, the first being the session to be reused, and the second being the  destination vcserver.

 

$job = {

    set-location $args[0];

    powershell -command ".\powercli_example_cmd.ps1 -session $($args[1]) -vcserver $($args[2])"

}

The following line is how we kick off the job.  Notice the “-argumentlist” where we specify the parameters to be passed into the $job scriptblock.

start-job -argumentlist $currentPath, $session, $vcserver -scriptblock $job | out-null

Before kicking off the job we need to build our powercli_example_cmd script.  This script will be responsible for loading the PowerCLI snapin and logging into the VC server with the existing session.  Notice the line for connecting with the “-session” parameter.  This script also includes our workflow, serial steps of commands, or in this case a single command that returns the first VM found to the job output.  An important note here is that the output of the job sent from receive-job comes through in arrays of strings.  So you can either send the output in something like JSON or simply grab what you want like the VM names (| %{ $_.name }) which is easily parseable.  If using this simply to execute methods then this warning may not be valid.

param($session=$(throw "missing -session parameter"),$vcserver)

 

if(!$global:DefaultVIServer) {

      if(!(get-pssnapin | where {$_.name -eq "vmware.vimautomation.core"})) {

        try {

            add-pssnapin VMware.VimAutomation.Core -ea 0| out-null

        } catch {

            throw "Could not load PowerCLI snapin"

        }

    }

    try {

        connect-viserver $vcserver -session $session -wa 0| out-null

    } catch {

        throw "Failed to connect to VI server"

    }

}

 

get-vm | select -first 1

 

Ok, so let’s move into actually running the jobs.  In order to execute the job we are leveraging a foreach loop as “1..2 | %{ .\powrcli_mt_control.ps1 –vcserver ip”.  This means run this command twice.  We can then leverage the get-job cmdlet to see the status of the job, as you can see below it is running and then turns to completed.  Once it is completed we can run “get-job | receive-job” to see the output.  Review the help for these cmdlets as there are some gotchas.  For example if you run the get-job | receive-job it will trip the “HasMoreData” to $false until more data comes in.  Long story, just play with it a bit to get comfortable.  When done with the jobs issues a “get-job | remove-job” to remove the existing jobs.

 

I’m afraid that’s all for now!  I am using this very successfully in order to multi-thread some pretty massive workflows all built within Powershell leveraging PowerCLI.  Here are the example scripts.

Have you ever had a situation where you need to do things that weren’t necessarily baked into PowerCLI for VMware vSphere?  As requirements change and we continue to reach for more features and functionality that becomes available we find ourselves a bit ahead of the curve and thus inevitably needing to create new ways of doing things.  I recently demonstrated with a new communication method via Adobe AMF to work in a programmatic fashion with VMware View.  And now I find myself in a new with a whole new bag of tricks for those that want to stay ahead of the curve.

If you’re a big fan of Powershell, I highly suggest following Luc Dekens and Alan Renouf.  They have some great examples of how to do things that are outside of the PowerCLI framework.  For example, see the following link for functions written by Luc that allow you to create, update, and remove datastore clusters.

http://www.lucd.info/2011/12/11/vsphere-5-top-10-storage-drs/

So this got me thinking about PowerCLI as a whole.  As we know it is a framework of cmdlets that leverage .NET assemblies which then communicates via SOAP to the vSphere API.  Now a bit of a warning here.. this is about the point where I stray beyond my normal zones so please correct me for any inaccuracies in what I’m saying so in comes my challenge.  I wanted to leverage the .NET assemblies (vmware-vim.dll) in order to get closer to the API.  Remember, cmdlets for PowerCLI leverage the underlying assemblies so working with the assemblies directly means that you have absolutely unfettered access to whatever is baked into the assemblies.  Now the gap between PowerCLI and the VMware VIM .NET assemblies for me is unknown, I will let a commenter fill me in there..  But what I do know is that it was a pretty cool project.  The rest of what you are going to see is basically a dive into the .NET vSphere assemblies (SDK) along with the vSphere MOB.

So you may ask.. what cool have you done with this?  So far nothing new *I think*, nothing catastrophic.. but it was definitely a great learning experience so if you have a few minutes to follow this then I guarantee it will be well worth your time for a vSphere API perspective.

By the end of what you see you should be able to start churning through the folder organizations and identify all methods and types that can be used.  Now let’s start with the basics.  In order to walk the available methods and types within the vSphere API there is a MOB url that you can leverage from the web.  The screenshot below is just this, give it a shot yourself by opening a web request to https://ipofvcenter/mob

 

 

Notice at the top how it lists the ManagedObjectID as “ServiceInstance.”  This is important for the organization of how we traverse the information available from the MOB.  The picture below is a good view of exactly how objects are categorized.  The structure begins with the ServiceInstance, followed up the rootFolder.  The root folder has a childEntity property which describes it as a datacenter object.  This datacenter object then has a handful of properties, and four of which are the core of how vSphere files away the different types of objects such as datastores, vms, networks, dvswitches, etc.  See this in the yellow below under the “ha-datacenter” table listed as datastoreFolder, hostFolder, networkFolder, and vmFolder.  In order to find any specific object, we have the option to turn to this parent folders first.

 

 

http://pubs.vmware.com/vsphere-50/index.jsp?topic=/com.vmware.wssdk.apiref.doc_50/right-pane.html

 

Now the next step is to actually start traversing the hierachy.  Click the rootFolder item which is listed as “group-d1 (Datacenters).”  This may look familiar since it is the ManagedObjectReference (moref of type Folder).  When working with the API it is important to keep in mind that we always reference moref’s in the methods or when performing actions. 

 

The next view below is once we have enetered the rootFolder.  It then lists the “childEntity” as being the “datacenter-2 (BigData).” 

 

So very cool so far.. we have now gotten to the bowels of how vSphere stores the object information.  So let’s take a quick break from the MOB traversing fun and turn to some Powershell fun!  Firstly, download this powershell module.  See the following screenshots for loading it properly.  The module itself loads the vmware-vim assemblies that comes with PowerCLI and leverage methods directly from the assemblies to operate.  I have basically written my own *unsupported* cmdlets that run completely independently of any *supported* PowerCLI cmdlets.  Again this was for fun and is educational in nature.  And there may not be a whole lot of value based on the cmdlets I wrote, but I leave that to you to go deeper than you had been able to previously!

 

So here we go, I am loading the module and issuing a Connect-VIServerVIM command.  This establishes an active session with the vCenter server which gets used by the other cmdlets.

 

Here is the first cmdlet that is generating useful data for us.  Notice how we are demonstrating a similarity between what you saw in the MOB screenshots based on the main datacenter moref?

 

The next screenshot of the MOB environment then shows us inside of the datacenter-2 object.  Now go to the next section…

 

Notice how we run the GetDatacenterVIM | Get-ItemProperties ?  The Get-ItemProperties is a cmdlet that I wrote which does a lookup on the moref object that is piped in.  This is essentially the same as when we cliked a link in the MOB browser, where we now have the details.  The information in these two screenshots are identical.

 

 

Ok so that’s pretty cool stuff.  Now let’s start to peel back the onion a bit in regards to what you might be able to take advantage of from the vmware-vim .NET assemblies.  In the following commands I load the assemblies and then issue a GetTypes() method call in order to show all of the available types.  The list is huge, 2726 items to be specific.  These all represent different object types that can be used and in most cases expected when working with the vim methods. 

 

 

Let’s go ahead and run the following command to take a peak at the methods, 2814 of them to be exact!  Not sure but that sounds like a bit more than is available by way of standard PowerCLI cmdlets. 

 

At this point I highly suggest bringing up vSphere 5 Documentation Center for online API docs that will explain some of what you just saw.  http://pubs.vmware.com/vsphere-50/index.jsp

Go ahead and drive the list to vSphere API –> VMware vSphere API Reference –> All Methods –> C –> CreateStoragePod.  Now what you notice here is this method is going to perform a specific action which is to create a new datastore cluster.  The parameters window listed below specifies exactly what this method is expecting to successfully be called.  Under the _this name, you can see it is expecting a moref object based on a parent folder.  So think about to what you saw in the beginning.  As we showed there the organization structure includes something called datastoreFolder.  This is exactly the moref that it is expecting in order to create this new folder (StoragePod/Storage Cluster).  And the second part is simply the name.  Done, these two things passed to this method would accomplish creating a new storage cluster.

So in summary, this example is showing new ways for Poshers to leverage .NET assemblies directly to achieve things that may not be published via PowerCLI as cmdlets or methods.  So next time you get your hands on a alpha copy of vSphere and you want to stear the new functionality with Powershell, don’t fret all you need is the vmware-vim .NET assemblies and you’re off to the races pretending to live a .NET life =) 

The following is showing some of the basic cmdlets that I created as examples.  Again the module is available here! (vSphere/PowerCLI 5+ only)

 

I am going to be a bit brief on the following examples since they are embedded in the functions within the module. 

This is an example of how you would issue a login request.  Once you have created a new VMware.Vim.VimClient object, we simply call a Connect method with the url followed by a Login method with the credentials.  We then have the session saved as part of the $global:vimClient variable which we later use in our functions.  Another important point again is the ServiceInstance, notice how we call it on the next line.  This allows us to look at the rootFolder which we need in order to start traversing the hierachy of folders exactly as you saw in the beginning of this post.

function Connect-VIServerVIM {

    param($username=$(throw "need -username"),$password=$(throw "need -password"),$viserver=$(throw "need -viserver"))

    $global:vimClient = New-Object Vmware.Vim.VimClient

    $t2  = $global:vimClient.Connect("https://$viserver/sdk")

    $global:vimClient.login($username,$password)

    $global:vimServiceContent = $global:vimClient.VimService.RetrieveServiceContent((New-Object "VimApi_50.ManagedObjectReference" -property @{type="ServiceInstance";Value="ServiceInstance"}))

}

 

This second function gets repeated a few times for the most part.  What we are doing here is issuing a RetrieveProperties method call based on the PropertyFilerSpec object that we created with some special filtering requirements.

function Get-Folder {

param([array]$pathSet=@("vmFolder"),$psType="Datacenter",$morefType="Datacenter",$morefValue="datacenter-2",

      [Parameter(Mandatory=$false, Position=1, ValueFromPipeline=$true)]

      [PSObject[]]$InputObject)

BEGIN { $objects = @() }

PROCESS { $objects += $InputObject }

END {

 

    if($objects[0]) {

        $morefType = $objects[0].type

        $morefValue = $objects[0].value

    }

    $ps = New-Object VimApi_50.propertyspec[] 1

    $ps[0] = New-Object VimApi_50.propertyspec -property @{type=$psType;pathSet=$pathSet}

 

    $os = New-Object VimApi_50.objectspec[] 1

    $os[0] = New-Object VimApi_50.objectspec -property @{obj=(New-Object "VImApi_50.ManagedObjectReference" -property @{type=$morefType;Value=$morefValue});skip=$false}

 

    $pfs=New-Object VimApi_50.PropertyFilterSpec[] 1

    $pfs[0]=New-Object VimApi_50.PropertyFilterSpec -property @{propSet=$ps;objectSet=$os}

    [VimApi_50.ObjectContent[]]$vmFoldersObjectContent = $global:vimClient.VimService.retrieveproperties($global:vimServiceContent.propertycollector,$pfs)

 

    $vmFoldersObjectContent[0].propSet[0].val

    }

}

 

That’s it for now! Looking forward to hearing what I was wrong about =) and possibly if this has any merit whatsoever =).

 

 

 

Some of the work here was modeled from Permalink to VMWare SDK HowTo- Determine if you are talking to a ESX Host or Virtual Center Serv (John McBride).

Who ever said VDI Admins can’t script?  Well at this point I can understand why they haven’t even tried.  Recently we (EMC vSpecialist rockstars @andreleibovici @chrishorn), and others have been focused on creating a next-gen lab environment and View has been the focal point for what I would term demand management.  The core concept that we are working to deliver is essentially “desktop as a service” for lab environments.  DaaS in our case is an extreme version where we are second by second removing and adding View desktops into already existing pools linked in a way to an active lab vApp.  With this is mind our requirements are pushing the limits of VMware View’s capabilities out of the shoot, so we really had to work overtime on developing methods that allow for the automation of a product with little to no operational hooks.

In previous posts that I have made around VMware View you can see some of this work progressing.  The good news is that we have accomplished exactly what we set out for.  The bad news is that we have had to go outside the normal acceptable methods that View provides in order to do so.  With this as always, what we post if meant for educational purposes and not meant for production environments as it may affect your supportability. 

Powershell and Adobe AMF3

Automating VMware View Pool Membership

Permanent Link to Installing VMware View Agent remotely using PowerCLI

How to Access Windows 2K8 Server via PCoIP with VMware View (@andreleibovici)

Please read on!

 

The most recent challenge for us has been around how to get proper information for capacity reporting out of View.  For example, what we wanted was to be able to see the current desktops in use and available in a certain pool along with the status of agents.  Unfortunately, using the current GA VMware View cmdlets there was only information about desktops that hadn’t been added to pools yet.  The idea that I sort of got what I wanted provided me some hope so I decided to start digging..

If you are looking for what is currently supported, see Alan Renouf’s site for some previous information that shows the GA cmdlets for View.  That native cmdlets have good information, but unfortunately are limited due to lack of remote capabilities (without remoting turned on), don’t receive piped objects in all cases, and are lacking the information we needed.

http://www.virtu-al.net/2010/09/14/vmware-view-4-5-more-powershell-cmdlets/

 

What I found was very interesting.  There are currently three methods that I know of to programatically interface with View.  The first being the public supported method of VMware View cmdlets included with a connection server install.  The second being an undocumented ability to leverage the message bus locally per VMware View connection server in order to inject commands.  And the third being my most favorite undocumented option being leveraging Adobe AMF (action message format) messaging.  I continued on this third method and developed a very cool scalable method for anyone to leverage AMF remoting with Powershell.

The AMF method is an awesome way to quickly profile a client GUI and transpose those calls into CLI.  See (Powershell and Adobe AMF3).  For scripts that I will be linking to at the bottom of this post I used the following method, which you don’t really need to know but cool enough to describe..

• Open Wireshark (configured to sniff SSL)
• Open VMware View Client GUI
• Do an operation
• Grab Hex from Wireshark of HTTP POST (AMF call)
• Deserialize the Hex into a powershell AMF psobject packet (Java Bean, Method, Parameters)
• Export this object as a xml file to be later used (check psobject.xml/ directory)
• Duplicate a generic function in my advanced Powershell functions
• Test and ensure parameters met what was built as expected from template
• Run cmdlet

 

Anyways, a very cool project that required about 99% of the time in creating an AMF serialization/deserialization process, and the other 1% enjoying the work and creating these “advanced” cmdlets.  Consider what is being displayed here as a v1 and a completely read-only version.  Who knows, you may see some of this work repeated for other cool things that are Flex/Flash based! And as always hopefully this becomes irrelevant sometimes soon =)

Here are some of the basic specs of what is currently available under the Avanced VMware View Powershell Cmdlets.

• Access View Connection servers remotely (without Powershell remoting)
• Built using Adobe AMF remoting
• All information displayed in View GUI is available
• PowerCLI style connection persistence (Connect-ViewConnServer)
• Currently Read-only access
• Powershell Advanced Functions to create cmdlets (@alanrenouf will be pleased) =)
• Plenty of stuff that isn’t available under the standard VMware View cmdlets

 

Without further ado, here are the scripts!  Most likely the best use case at this point is around generating reports of configuration and reporting on status for services and desktops.  Anything is possible since we are now hooked into AMF, but I am still debating outside of the reporting use case what else we need to be able to do via Cmdlets remotely.  Making them read-write is all about knowing the java beans, methods, and parameters that are required for posting.

 

• Connect-ViewConnServer
• Get-AccessSettings
• Get-Administrators
• Get-Desktops
• Get-EventConfiguration
• Get-Folders
• Get-ImageRepositoryDetails
• Get-LocalSessions
• Get-PersistentDisks
• Get-Policies
• Get-Pools
• Get-RemoteSessions
• Get-Roles
• Get-SecurityServers
• Get-Servers
• Get-ThinAppRepositories
• Get-ThinApps
• Get-TransferServers
• Get-UsageList
• Get-UsersGroups
• Get-VDMServers
• Get-ViewDashboard
• Get-ViewGlobalSettings
• Get-ViewStatus

Current Available Cmdlets

 

Instructions are very simple.  Download the zip, unzip ensuring that you keep the xml files in their respective directory, and then run an “import-module uadv_vmware_view.psm1”.  From there you have access to all of the cmdlets listed above.  You can issue get-help before cmdlets to see the available parameters.  For the most part the cmdlets are very bare since we are doing read-only operations and you can manipulate the data after the fact using Powershell standard methods (ft, sort, where, etc).

Begin by issuing a “Connect-ViewConnServer -user administrator -password xx -domain domain -viewConnServer 192.168.1.112”  From there you it should look like below, and you should be able to issue a command like “get-Desktops”.

A successful login, and then a get-desktops command formatted as a table.

 

 

Enjoy!!