Integrating Avamar VM Image Backup Reporting with VMware vCenter Operations

Posted by vele1655 | Posted in Avamar, vCOps | Posted on 25-02-2013

Tags:

7

Have you ever wondered how your backups affect performance and other things in your environment?  Are you used to manually hunting and pecking for this information?   Shouldn’t an operator know right away that a backup was taking place when troubleshooting or performing root cause analysis after the fact?  If a Virtual Machine or Datastores are underperforming wouldn’t it be nice to know how many backups were occurring during these times and what is affected? 

 

If you’re a VMware customer and are familiar with vCenter Operations Standard+ and utilize EMC Avamar for VM Image based backups these kinds of questions can be answered!  As usual with my stuff, I am providing an unofficial way to perform this integration, but if you find it valuable let VMware and EMC know!

 

First of all, the integration being shown was performed with vCenter Operations 5.6 Standard and above along with Avamar 6.1.

 

The integration that is occurring between Avamar and vCenter Operations is very simple.  We are taking backup events after they have finished and creating “change events” in vCOps for the start and finish of the backup operation.  You can see this from the Operational Events graph below. 

 

We are looking at the Health graph of a specific VM along with its related events.  Here we are hovering over the graph on an ME logo which corresponds to change events for this VM.  Notice mostly there are clustered events two at a time representing start and finish of backup jobs for this VM.  Also notice how the Health (graph) is lower in some cases during these events?  This represents the analytical capabilities of vCOps determining that there are a certain percentage of metrics or KPI’s that are above their “learned” dynamic threshold (DT) for this VM.  This can result from a Virtual Machine’s own metrics that it relies on.  The event is being shown here with the duration of backup and total bytes that were sent after changed blocks (CBT) were deduplicated within the Avamar VMware image proxy VM.

 

 

clip_image001[8]

 

 

Note!  The concept of the VMware image proxy is important for this use case!  Here a similar graph that is showing the workload instead of health for a Virtual Machine.  Notice how there is not extra workload being reported?  Well this makes sense since the VM is not doing anything during backup time, the proxy is!  This can however in certain cases spike since a VM workload may peak since it’s underlying dependent items such as storage may be hitting peak demand.  Anyhow, this offload is how VM image based backups occur with Avamar, but as you can see can cause lower health due to using underlying shared resources (such as storage) and thus increasing response times or reducing capable throughput.

 

 

clip_image002[7]

 

 

So how can we get to the bottom of this?  We showed the Health prior, but let’s now look at the anomalies view.  This summarizes possible reasons for the health being low and is somewhat inverse to the health graph.  Again we see backup events integrated and the anomalies being reported.

 

clip_image003[7]

 

 

Now let’s flip over to see related items such as Datastores that the VM is using.  You can see this easily from the Environment -> Members tab where the Datastores for the VM are listed.

 

clip_image004[7]

 

 

Here we have the same Operational Events view from the VM and are still looking at anomalies.  We start by reviewing the anomalies and selecting a workload badge on the graph on the left side.  You can see here that there is a time when vCOps determined that a Datastore was being limited by Disk IO and thus the demand was at 100% affecting all resources that relied on the Datastore. 

 

 

clip_image005[7]

 

 

Now let’s overlay the change events on the graph.  We can now see that the backup events can be shown to directly relate to the anomalies.  Notice how there are now multiple events being shown?  This view is from the Datastore and it is showing not only events for itself, but also events that occur from all of its children (VMs).  So here we can see an accumulation of backup events all occurring at the same time for different VMs!  This is a good thing of course since we are intentionally doing this.

 

clip_image006[8]

 

 

So you may be asking now, so what?  Don’t do backups?  The point here is to show that we have Avamar and vCenter Operations can be integrated to give an operator the information they need to properly react and plan for these type of events.  In this example we are doing image based VM backups with source based deduplcation from Avamar along with VMware’s changed block tracking (VADP integration).  These features offload and reduce infrastructure utilization during backups.  So what would your environment look like if you’re not taking advantage of CBT and source based dedupe?  You tell me!

 

 

 

So how can I get this integration going?  The integration used here is written in Powershell/.NET.  It is very simple, we consume events from the Avamar Postgres through a .NET provider and post them to the vCOps REST interface as change events.  Below is a basic outline of how I outline the areas that will be covered when it comes to installing and configuring the integration.

 

 

  • Installing the Avamar vCenter Operations Integration
  • Success!  Continue on with Standard Non-Interactive Usage..
  • Standard Interactive Usage
  • More details on Get-VMAvamarActivities

 

 

 

Installing the Avamar vCenter Operations Integration

Now in order to get the integration working there are a few requirements that I must call out.  First, what you can download from this site is not officially supported, take it as an early alpha release technical preview!  Your feedback can help shape future product integration so send it my way!

 

Requirements

 

Integration Points

  • Avamar Postgres Database
    • Review Avamar 6.1 Administration Guide, “Support for third-party reporting tools”
    • Review Avamar 6.1 Administration Guide, Appendix C “MCS and EMC Database Views”
  • Avamar
  • VMware vCenter
  • vCOps 5.6 REST Interface

 

 

The Powershell module contains cmdlets that can leveraged either in a continuous service oriented fashion, or interactively.  In order to make this easier, as part of the install process there is a step to create a login profile that stores authentication information (in a secure form) as well as a shortcut to launch the Avamar vCOps Service.  So let’s get started!

 

Download Powershell Module and Postgres Data Provider

If Powershell and PowerCLI is installed then the first step is to download the Avamar vCOps Powershell module along with proper Postgres .NET data provider assemblies.  The Powershell module can be downloaded from here.  The Postgres module must be downloaded from another place here.  Notice the screenshot below, from the Downloads page, download any of the version 2 ZIPs that match your .NET version.  In my example, since I am using Powershell 3 I downloaded the Npgsql2.0.12.0-bin-ms.net4.0.zip.  I highly suggest you use Powershell v3 to take advantage of Login Profiles.

clip_image007[8]

 

Unzip Downloads and Unblock

Now once downloaded, unzip the Powershell module to a non-network drive and directory of your choice.  This requirement is in place due to DLLs that don’t play nicely when living on network drives.  Limitation can be overcome with some simple hardcoding if you so please.  Unzip the Npgsql file and grab two files, Npgsql.dll and Mono.Security.Dll.  Copy them to the newly created directory with the Powershell module in it.  This is how the files should look in this directory.

clip_image008[7]

 

One more thing here for Powershell v2 users.  The DLLs may need to be Unblocked for security purposes (we do it automatically in v3).  Right click the DLLs individually and press Properties.  Press the Unblock button at the bottom.

 

 

Create Login Profile

With that done, you should be ready to create your first Login Profile which is used to store credential (as a Secure String) and IP/DNS information for each integration point.  This dramatically simplifies things and enhances the security a bit.  Important note! The Secure Strings are .NET based, and are based on keys that are generated per logged in User per Server to the Powershell instance.  Each User that opens Powershell will need to run this command themselves with the credential information or you will need to generate the proper Secure Strings for them with their unique Key (not shown here but possible).  This leverages the ConvertTo-SecureString cmdlet.

 

Ok, now that you’ve got it downloaded and proper files in place the next thing to do is to open a Powershell window.  Change directory to where you unzipped the Powershell module.

 

Run the following command to import the Powershell module.

Dir *.psm1 | Import-Module

 

Now check to ensure you have the necessary cmdlets from this module loaded.  See screenshot below.

Get-Command -Module avamar_vcops

 

And finally you can create the profile if the cmdlets are loaded as seen in the screenshot below.  Take the following command and copy it to notepad and edit the properties in include the proper IP/DNS, Username, and Password of Avamar, vCenter (VIServer) and vCenter Operations.  If you wish to not include vCenter Operations, remove the “vCOps=” line.  This will mean you can only use the script in an interactive form, no posting the backup job events to vCOps but possibly to a destination of your choice.  Also, ensure you include passwords in single quotes (‘) to ensure special characters are preserved.

New-AvamarLoginProfile -Shortcut -Name “profile1″ -LoginProfile  @{

            Avamar=@{server=”ave02″;username=”MCUser”;password=’pass’}

            AvamarDB=@{server=”ave02″;username=”viewuser”;password=’viewpass’}

            VIServer=@{server=”vcenter01″;username=”root”;password=’rootpass’}

            vCOps=@{server=”vcops01″;username=”admin”;password=’pass’}

        } 

 

 

You should see the following output where “Shortcut has been created” is shown at the bottom.  I will explain the shortcut in a second.

 

clip_image009[7]

 

 

 

Test Login with Profile

Now the next step is to test and use the login profile.  But first if you don’t have a PowerCLI window (you followed instructions, but PowerCLI window would work too), you need to load the PSSnapin from the Powershell window.  By not having you open PowerCLI I can ensure that we are not connecting to multiple vCenter instances.  During development I didn’t plan on this, so for now make sure you are only using 1 vCenter at a time per Powershell instance.

Add-PSSnapin vmware.vimautomation.core

 

From the same command prompt enter the following command, edit the “profile1″ name to whatever you specified in the above cmdlet for the -Name parameter.

Connect-AvamarLoginProfile -Name profile1

 

This command will connect to all of the items that were specified in the Login Profile.  If successful, you will see the output as follows without error.  If it is not successful, then it will show errors.  Most likely errors include password or IP/dns problems.  Please refer to Requirements and previous steps if you are encountering errors.

clip_image010[6]

 

 

 

Success!  Continue on with Standard Interactive Usage..

Since we have successfully created and are using a profile now we have everything we need to get the integration working.

 

Start a VM Image Backup

In the case of this demo, I am using the integration in a fresh lab environment with an unused Avamar Virtual Edition node installed.  So I am going to start a backup so I have a backup record to see.

clip_image011[6]

 

 

Get-VMAvamarActivities

There is a semi-advanced cmdlet that we are going to be using in its most basic fashion to correlate Virtual Machines as Avamar VM Clients and their corresponding backups.  Under the covers the cmdlet can be used in a handful of different scenarios that I will cover in the advanced section.  In this case since there are no parameters specified we are returning every backup job for this VM with the following command.

Get-VM wguest-01 | Get-VMAvamarActivities

 

clip_image012[6]

 

It is important to also call out that there is a much more efficient way to get all VMs (for larger environments) and their associated Avamar accounts.  The following command would return all VMs (-getAllVms) and cache all Avamar VM client accounts without individual lookup (-useBulkAvamarVMLookup) and specifies to only pull the last three days of jobs (-Start_Recorded_Date_Time). 

Get-VMAvamarActivities -getAllVms -useBulkAvamarVMLookup -Start_Recorded_Date_Time ((Get-Date).addDays(-3))

 

These are all important parameters for the next part.  We are working towards being able to prepare for Non-Interactive Usage (service) where we continuously update the backup jobs.  In order to do this however, there is one more concept that needs to be covered.

 

Bookmarking (-useBookmark)

In order to track which records have been sent to vCenter Operations, after a successful post if the (-vCOpsPost) parameter is specified , a bookmark entry is created and stored as a AvamarNameorIP.CliXml.  See the following screenshot where the file was created after a successful post.  The (-useBookmark) parameter if specified respects the bookmark as last record seen, and to retrieve only the newer records.  This way we can either run the integration on demand or run it continuously as a service and it keeps track after each individual successful Post per backup job where it left off and can recover from any error (as long as bookmark is there).  Important note:  If you have multiple vCenter’s that are talking to the same Avamar instance, then bookmarking will not work properly.  In the case you care to integrate multiple vCenters with a single Avamar instance and send the data to vCOps (using bookmarking) then create separate directories with the same files in them.  The separate directories will allow each instance of Powershell to maintain its own AvamarNameorIP.CliXml file.  This is possibly not a limitation in a future rev.

 

Bookmarking is required so that we don’t send duplicate data to vCenter Operations.

 

 

clip_image013[6]

 

 

So before running the Non-Interactive mode you have a choice to make.  You can either sync up the entire Avamar VM image job history to vCenter Operations for VMs that currently exist, or you can create a bookmark to begin at a certain point.  The next command gets all of the VMs, and starts from 3 days prior, posts the results to vCenter Operations and creates proper bookmarks.

 

Get-VMAvamarActivities -getAllVms -useBulkAvamarVMLookup -Start_Recorded_Date_Time ((Get-Date).addDays(-3)) -vCOpsPost

 

Or for pulling all backup jobs and creating bookmarks. (This is what is ran continuously in Non-Interactive Usage)

Get-VMAvamarActivities -getAllVms -useBulkAvamarVMLookup -vCOpsPost

 

clip_image014[6]

 

Important Note:  A bookmark will only be created with a successful post of a backup job!  You can create a bookmark manually by specifying the following command and changing the “AddDays” portion from the previous 30 days, to something of your choice.  This will set you up for running this as a service starting at this point, or interactive usage if specifying -(useBookmark) automatically from this point.  Note how we are specifying ToUniversalTime, may not be extremely important, but can be if you are trying to be exact since the Avamar database stores events in UTC.

New-Object -Type psobject -Property @{recorded_date_time=((Get-Date).AddDays(-30)).ToUniversalTime() } | Export-CliXml “$($DefaultAvamarServer.Server).CliXml”

 

Standard Non-Interactive Usage (service)

Open Created Shortcut

With Explorer open the directory that has the Powershell module.  Notice the “Avamar vCOps (profile1)” shortcut that exists.  This was created automatically for you when we created the Login Profile.  This shortcut can be used to launch the non-interactive “service” method of performing the integration.

clip_image015[6]

 

 

Once you double click the shortcut it will launch a Powershell window and title it appropriately for the profile name and Powershell PID.  You can see here that we perform a Login, Backup Job query and post, and Logout.  This sequence is iteratively ran with a 5-minute delay between runs.  A parameter of (-once) can be added to the shortcut in order to only run once and quit.

 

Below you can see the window with a couple of lines that begin with “Posting.”  Since we ran a backup job prior, it is actually posting this information to vCOps now.

 

clip_image016[6]

 

 

Verify Post of Change Event in vCOps

Now you can see that the event actually made it into vCOps by loading the standard vSphere oriented UI and navigating to the VM on the left side.  You can then click on Operations in the top, and Events below it.  Once there a health screen graph will appear, and you can select the Target above the graph in the “Related Events” portion of the graph toolbar.  If necessary you can modify the date range with the calendar button on the tool bar.  You should then see the appropriate change events marked ME (My Event) with the attached descriptions as you saw in the Powershell window above.

 

clip_image017[6]

 

 

 

All Done!  You now have the interactive and non-interactive modes working and are ready to push backup information continuously to vCOps and possibly do some other really cool stuff with Powershell and Avamar!

 

 

 

 

 

 

 

Comments (7)

This is awesome! Thanks so much for posting this, I’ve been looking for a way to integrate Avamar and vCops like this for a while. I’ve just spent the last 2 hours setting it up and it works a treat.

Any chance this could be used to add a metric to each VM with the amount of change data? This would really help to visualise change rates over time, give us a way of correlating production IO to backups and add a layer of vCops analytics to the relationship.

Yes this is possible. However it is a bit outside the normal metrics vcops tracks as it will only be reported per backup versus per 20 seconds. Will look at it and are what I can drum up!

Glad to hear you like it!

Fantastic, thanks

Very Cool! Thanks for the tips/help on getting this to work with VDP as well! Very slick!

Hello.
This is very nice. But I seem to have a problem when starting the profile and \ or trying to post data to vCOPS. I get this powershell error and I can`t pass it and no data gets inside VCOPs. I`m not expert in powershell so apologize for that. Any thoughts?!

error:

9/26/2014 10:08:03: Avamar server hostname change in Clients detected, updating cache.
xception calling “getClientInfoList” with “2” argument(s): “There is an error in XML document (1, 107985).”
t C:\Avamar-vCOPS\avamar_vcops.psm1:542 char:17
[array]$tmpOut = $global:DefaultAvamarServer.Proxy.getClientInfo …
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : InvalidOperationException

Try to re-download from the module from the link in the post or here. The WSDL file had to be updated for later versions of Avamar.

Thanks a lot. Worked like a charm! :)

Write a comment


× one = eight