Using Azure Backup Logs in Azure Log Analytics (aka OMS)

If you’ve followed along with some of my previous posts (i.e. Azure Backup Now Has Reports or How-To Create Custom Azure Backup Reports Using the Data Model, then you should already have Azure Backup configured to send Log data to an Azure Storage account, and are using the PowerBI content pack for visualization of that data.

As was mentioned in the first referenced article, in addition to using an Azure Storage account, you can also send Azure Backup log data into Log Analytics (aka OMS).

Azure Backup Reports –
OMS Integration

Today we’re going to look at this data as it is presented in OMS, and work with it to create a custom OMS solution.

 

Integration

To start, you need to integrate the Azure Backup logs into an OMS Workspace. To do this, follow along with the introduction and Step 1 of my Azure Backup Now Has Reports article, and select the Send to Log Analytics option.

Azure Backup Reports –
OMS Integration

After the integration is complete, to confirm that data is being sent to OMS, use the following search query: AzureDiagnostics | where (Category == “AzureBackupReport”) 

Provided sufficient time has passed (Note: The recommended wait time is 24 hours) since you integrated the Backup Logs with Azure Log Analytics, you should see some results returned.

Azure Backup Reports –
OMS Integration –
Query Results

Important: Please note that all examples and queries are based on the new OMS query language. If your Workspace has not been upgraded yet, the provided queries will not work for you.

For further information on the new Azure Log Analytics query language, please see the official documentation found here.

 

Create a Custom OMS Solution

Now that we’ve verified that we have data available in OMS, we can start to create our own custom solution to monitor and visualize our backup jobs. We will be using the OMS View Designer to create this solution.

Note: We are creating a custom OMS solution using the available Log data, since the current OMS Azure Backup solution only works with ASM-based Vaults, and not the new ARM-based Recovery Services Vaults.

If and/or when Microsoft officially releases an updated OMS Backup solution, our custom solution may become obsolete, or need further customization based on our organization’s specific needs (which may or may not be satisfied through an official solution).

 

Overview Tile

If we’re creating a custom OMS solution, the first “view” is the Overview tile, and it will be the first thing we see in the OMS (and/or Azure) portal. So we need to think about what is the most important piece of information we need to be able to check easily and quickly first thing. The answer (in my mind) is the Backup Job Status since we would want to see if any backups failed overnight.

So, in our query editor, we can retrieve this information via the following query: AzureDiagnostics | where (Category == “AzureBackupReport”) | where (OperationName == “Job”) | summarize count() by JobStatus_s

And we can use that as our query in our overview tile.

Azure Backup Reports –
OMS Integration –
OverviewTile

 

Drill Down Views

Now that we have an overview tile that quickly and easily shows us job statuses, we want to be able to drill down into this, especially if there are any failed jobs.

Within the View Designer, we will add a View Dashboard to list the individual protected systems and their backup status.

In the list query I used the following: AzureDiagnostics | where Category == “AzureBackupReport” | where OperationName == “Job” | project ProtectedServerUniqueId_s, JobStatus_s 

Azure Backup Reports –
OMS Integration –
View Dashboard

You may notice that the Computer field isn’t quite useful. If you run the query in the Log Search, this is what the results look like:

Azure Backup Reports –
OMS Integration –
View Dashboard – Query

You see that the ProtectedServerUniqueId_s field contains the system name, but also includes the region, a GUID, etc. 

Using the new query language guide (specifically the String Operators), I found several methods for retrieving just the element I want in the view (i.e. the Computer Name); namely the Extract, and Split operators. Unfortunately, I have not had any success getting either of those to work with the View Designer. When I do, I will come back and update this article.

Just take note that you can create custom views based on retrieving the fields you want to visualize/list as part of the custom solution.

 

Conclusion

In conclusion, I hope this shows you what you can do with the Backup log data being integrated with OMS, as well as how you can create custom views. If you’re interested in creating custom dashboards and reports, then check out my complimentary article: How-To Create Custom Azure Backup Reports Using the Data Model.

How-To Create Custom Azure Backup Reports Using the Data Model

With the recent release of the Azure Backup Reports by the Azure Backup team, we now are starting to gain insight and visibility into our backups.

Sure, we can navigate to the Recovery Services Vault (RSV) in the Azure portal to check for any errors, issues, etc. But we can gain far more value through the customization we can achieve with PowerBI.

 

Azure Backup Reports Data Model

As part of the release of the PowerBI content pack, the Azure Backup team also thoughtfully included documentation on the data model that is created and used.

By using this data model, not only can we modify the existing reports provided, but we can also create our own reports.

The Data Model currently contains 9 tables, and has 108 fields!

• Alert (11 fields)
• Backup Item (12 fields)
• Calendar (11 fields)
• Job (14 fields)
• Policy (28 fields)
• Protected Server (13 fields)
• Storage (5 fields)
• Time (6 fields)
• Vault (8 fields)

 

Creating a Custom Azure Backup Report

Using this data model, we can create a custom report, using the data that’s provided through the Recovery Service Vault.

After you have successfully configured and deployed the Azure Backup Reports, specifically, setting up the PowerBI component, you should now have an Azure Backup dataset.

The Azure Backup PowerBI content pack gives us a great start to see what we can achieve, but let’s see what else we can do with a custom report.

If you select the Azure Backup dataset, you will be presented with a blank canvas page. From there, you will need to find the various Fields you are interested in, and develop your own custom report.

Here is an example of a custom report I developed, to give you an idea. It may be a little plain, but it demonstrates how you can mix and match fields from different tables (i.e. Backup Item, Job, Protected Server, etc.) to be able to visualize and list exactly the information you are looking for.

You can additionally export these reports to Excel, PowerPoint etc. and also append it with specific fields to your organization for further use.

But wait, there’s more!

 

Using PowerBI Desktop

Using PowerBI online is fine, but I personally prefer to use PowerBI Desktop to develop my dashboards and reports, and then upload them to the online service. This provides me with more flexibility to achieve exactly what I want.

But how do we connect to the Azure Backup dataset in PowerBI Desktop? And, while we’re on the topic, if you are using the Log Analytics (aka OMS) integration (which, it itself can create/send datasets to PowerBI as well), there is no clear way to connect to an OMS-created dataset to PowerBI Desktop.

Ah, but there is.

In PowerBI Desktop, if you select Get Data, and navigate to Online Services. From there you should see an option for PowerBI Service. Click Connect.

You will be prompted to log into your PowerBI account, after which, you will be able to select any dataset that is attached to your service workspace.

Now you can work in PowerBI Desktop to develop your custom dashboards and reports. Also of note, is the fact that you can now view the hidden fields that are not exposed in PowerBI Online. If you right-click on one of the Field tables and choose ‘View hidden‘, you will then be able to see not only the hidden tables but also the hidden fields within the primary 9 data model tables.

 

PowerBI Online vs. PowerBI Desktop

One of the main differences with PowerBI Desktop is that it allows us to create new custom fields, measures, etc.

Normally, when you connect to a data source in PowerBI Desktop, you can access the data model relationship, preview the data contained in the tables, and create all the modifications you want.

Because we’re connecting to the PowerBI Online service, we do not get all these features and options. We can, however, create a new Measure to aid in our visuals.

For example, the Storage table has a field called ‘CloudStorageInMB’. But what if you have Gigabytes or Terabytes of data contained in the Recovery Services Vault? We don’t want some large Megabyte-based number, but something more relevant.

We can create a new Measure, and use the formula: Measure = [CloudStorageInMB] * 1024 to produce a number that represents Gigabytes, or you can add to the calculation to represent Terabytes.

Once you have completed your custom report, you can Publish it back to PowerBI Online. The custom field will remain, although it is unclear how this will affect subsequent data refreshes since no new dataset is created (I’m not a PowerBI guru).

 

Conclusion

The out-of-the-box Azure Backup Reports are great, and definitely get you started nicely. But, if you want further customizations, you can either modify the existing reports, make copies of the originals and modify them as needed, or create completely new custom reports and dashboards.

Nice work Azure Backup Team, we look forward to more to come.