How-To Create Custom Azure Backup Reports Using the Data Model

With the recent release of the Azure Backup Reports by the Azure Backup team, we now are starting to gain insight and visibility into our backups.

Sure, we can navigate to the Recovery Services Vault (RSV) in the Azure portal to check for any errors, issues, etc. But we can gain far more value through the customization we can achieve with PowerBI.

 

Azure Backup Reports Data Model

As part of the release of the PowerBI content pack, the Azure Backup team also thoughtfully included documentation on the data model that is created and used.

By using this data model, not only can we modify the existing reports provided, but we can also create our own reports.

The Data Model currently contains 9 tables, and has 108 fields!

• Alert (11 fields)
• Backup Item (12 fields)
• Calendar (11 fields)
• Job (14 fields)
• Policy (28 fields)
• Protected Server (13 fields)
• Storage (5 fields)
• Time (6 fields)
• Vault (8 fields)

 

Creating a Custom Azure Backup Report

Using this data model, we can create a custom report, using the data that’s provided through the Recovery Service Vault.

After you have successfully configured and deployed the Azure Backup Reports, specifically, setting up the PowerBI component, you should now have an Azure Backup dataset.

The Azure Backup PowerBI content pack gives us a great start to see what we can achieve, but let’s see what else we can do with a custom report.

If you select the Azure Backup dataset, you will be presented with a blank canvas page. From there, you will need to find the various Fields you are interested in, and develop your own custom report.

Here is an example of a custom report I developed, to give you an idea. It may be a little plain, but it demonstrates how you can mix and match fields from different tables (i.e. Backup Item, Job, Protected Server, etc.) to be able to visualize and list exactly the information you are looking for.

You can additionally export these reports to Excel, PowerPoint etc. and also append it with specific fields to your organization for further use.

But wait, there’s more!

 

Using PowerBI Desktop

Using PowerBI online is fine, but I personally prefer to use PowerBI Desktop to develop my dashboards and reports, and then upload them to the online service. This provides me with more flexibility to achieve exactly what I want.

But how do we connect to the Azure Backup dataset in PowerBI Desktop? And, while we’re on the topic, if you are using the Log Analytics (aka OMS) integration (which, it itself can create/send datasets to PowerBI as well), there is no clear way to connect to an OMS-created dataset to PowerBI Desktop.

Ah, but there is.

In PowerBI Desktop, if you select Get Data, and navigate to Online Services. From there you should see an option for PowerBI Service. Click Connect.

You will be prompted to log into your PowerBI account, after which, you will be able to select any dataset that is attached to your service workspace.

Now you can work in PowerBI Desktop to develop your custom dashboards and reports. Also of note, is the fact that you can now view the hidden fields that are not exposed in PowerBI Online. If you right-click on one of the Field tables and choose ‘View hidden‘, you will then be able to see not only the hidden tables but also the hidden fields within the primary 9 data model tables.

 

PowerBI Online vs. PowerBI Desktop

One of the main differences with PowerBI Desktop is that it allows us to create new custom fields, measures, etc.

Normally, when you connect to a data source in PowerBI Desktop, you can access the data model relationship, preview the data contained in the tables, and create all the modifications you want.

Because we’re connecting to the PowerBI Online service, we do not get all these features and options. We can, however, create a new Measure to aid in our visuals.

For example, the Storage table has a field called ‘CloudStorageInMB’. But what if you have Gigabytes or Terabytes of data contained in the Recovery Services Vault? We don’t want some large Megabyte-based number, but something more relevant.

We can create a new Measure, and use the formula: Measure = [CloudStorageInMB] * 1024 to produce a number that represents Gigabytes, or you can add to the calculation to represent Terabytes.

Once you have completed your custom report, you can Publish it back to PowerBI Online. The custom field will remain, although it is unclear how this will affect subsequent data refreshes since no new dataset is created (I’m not a PowerBI guru).

 

Conclusion

The out-of-the-box Azure Backup Reports are great, and definitely get you started nicely. But, if you want further customizations, you can either modify the existing reports, make copies of the originals and modify them as needed, or create completely new custom reports and dashboards.

Nice work Azure Backup Team, we look forward to more to come.

Azure Backup Now Has Reports

Recently, the Azure Backup team released a set of PowerBI reports for Azure Backup.

I had the privilege of being a part of the cognitive study while these reports were in development. Now, they are starting to become available for everyone.

 

New Azure Backup Reports

If you have a Recovery Services Vault (RSV), you will see a notification banner stating: “NEW! Azure Backup Reports are now available. Go to Settings blade > Backup Reports to configure them.”

Azure Backup Reports – Notification Banner

 

Let’s walk through the setup and configuration of this new feature.

The instructions state: “Go to Settings blade > Backup Reports to configure them.” In the Settings blade, the Backup Reports is listed under Monitoring and Reports.

Azure Backup Reports – Settings Blade

When you navigate to Backup Reports for the first time, you will see a list of instructions to follow.

Azure Backup Reports – Instructions

 

Step 1: Configure Diagnostics

Our first step is to configure diagnostics for our Backups. This will be an Azure Storage account to hold the data.

When you click the Configure button, you are prompted to enable the Diagnostics Settings.

Azure Backup Reports – Diagnostics Settings

When you set the Status to On, you will be prompted for the location to store your diagnostics data. Notice that we have 3 options:

• Azure Storage
• Event Hub
• Log Analytics (aka OMS)

I am very pleased and excited that Log Analytics (aka OMS) is an option since this will facilitate maintaining a central view for monitoring your environment.

Also, notice that the Log selection includes ‘Azure Backup Report’ logs, but also includes Site Recovery Jobs/Events, and data on Replicated Items. So obviously, this is expanding beyond just using it for Azure Backup reports, to also exposing Azure Site Recovery information as well. This is very good since it will allow us to more easily create custom OMS Solutions and Alerts for our Business Continuity and Disaster Recovery (BCDR) needs.

Azure Backup Reports – Diagnostics Settings Detailed

Notice, however, that in order to use the Azure Backup Reports PowerBI content pack, you must configure an Azure Storage account. Even though I have configured the data to be sent to my OMS Workspace, in which I could use OMS to create/send a dataset to PowerBI, the currently supported method is to use an Azure Storage account. Hopefully, in the future, we will have the option to forego the Azure Storage account altogether (as was my feedback to the Azure Backup team originally), and just use OMS directly.

Azure Backup Reports – Storage Account Required

Notice that when you select “Archive to a Storage Account” we have a few more options that we can configure. We now have the ability to set the Retention Days. The default is zero, which means the data will be retained forever.

That’s an important setting to be aware of because if you leave it as zero, you’ll have to keep an eye on your Storage Account in the event that it becomes full (not very likely, but still something to consider in case the target Storage Account serves multiple purposes).

Also, notice that the blade journey experience does not allow you to create a new Storage Account as part of this process. You will need to have already created your target Storage Account. Hopefully, this experience will be enhanced in the future, similar to the Virtual Machine creation process, where you can create a new Storage Account in the same journey (without having to leave the process and come back).

Additionally, there is no indication as to the recommended and/or required Storage Account settings and configuration (i.e. General Purpose vs Blob Storage, LRS vs GRS, Encryption, or if it is required to be in the same region as the Recovery Services Vault).

Update: According to the Supported Scenarios, the Storage account selected should be in the same region as Recovery Services Vault. However, it is unclear if this is only in reference to viewing reports across Vaults and across Subscriptions. I have not tested otherwise, as my Azure Storage account and Recovery Services Vault were located in the same Azure Region.

Update 2: According to the Product Group, the region limitation for the Azure Storage account is accurate for all scenarios.

Azure Backup Reports – Storage Account Configuration

 

Step 2: Deploy and Configure the PowerBI Azure Backup Content Pack

The second step is to sign into PowerBI, and setup the content pack.

After you have logged into PowerBI, under Services click Get.

Azure Backup Reports – PowerBI – Service Get Data

In the AppSource, search for “backup“, and click “Get it now” for Azure Backup.

Azure Backup Reports – PowerBI – AppSource

You will then be prompted for the Azure Storage Account that holds the data.

Azure Backup Reports – PowerBI – Storage Account

After you’ve specified the Storage Account, you will need to also provide the Storage Account Key to authorize access.

Azure Backup Reports – PowerBI – Storage Account Key

On the first attempt, I encountered an error.

Data Source Error: AzureBlobs failed to get contents from ‘https://azurelabstoragebckuprpts.blob.core.windows.net/insights-logs-azurebackupreport/’. Status code: 404, description: ‘The specified container does not exist.‘.. The exception was raised by the IDataReader interface.
Activity ID: 46c6c314-d9d1-4382-8af1-8b6c0b0187a4
Correlation ID: d53e62ab-d80c-bc44-19f1-f149fc790e5e
Request ID: 9e6c32d3-8bad-6291-9630-5f4ad4397ec8
Status Code: 400
Time: Wed Jun 28 2017 12:07:12 GMT-0400 (Eastern Daylight Time)
Version: 13.0.1721.165
Cluster URI: https://wabi-us-north-central-redirect.analysis.windows.net

Azure Backup Reports – PowerBI – Storage Connection Error

Notice that it is looking for a container within my Azure Storage account called “insights-logs-azurebackupreport“. It is uncertain at this time, if the log ingestion process will auto-create this Container (and if ti does, how/when this occurs), or if we are required to manually create it ourselves.

Update: According to the Frequently Asked Questions, the “insights-logs-azurebackupreport” container will be auto-created.

In order to continue with my exploration, I manually created the Container required.

Although I configured this against a Recovery Services Vault that actually has data (i.e. being used for Backups and also for Site Recovery), the reports and dashboard remained blank.

Update: According to the Frequently Asked Questions, it will take approximately 24 hours for data to be pushed into the target Azure Storage Account, and therefore available for the PowerBI dashboards and reports.

I’m assuming this is because this solution was just configured, and no logs exist, but it would be nice if it was configured to reach back in time (say, something configurable like 3, 6, or 12 months) to immediately gain valuable insight into the health and status of your backups.

This is what the Dashboard looks like:

Azure Backup Reports – PowerBI – Dashboard

And this is what the Reports look like. Notice that there are multiple tabs along the bottom, providing information on Storage, Backup Items, Job Health, Job Duration, and Alerts.

Azure Backup Reports – PowerBI – Reports

Azure Backup Reports –
PowerBI – Report Tabs

 

Caveats

A few quick notes or caveats about the Azure Backup Reports.

Firstly, currently the Azure Backup Reports are only supported for Azure Virtual Machine backups, and File/Folder backups using the Azure Recovery Services Agent (aka Direct-Agent).

This means that, although you can use Azure Backup (via the Recovery Services Vault) with Azure SQL, Microsoft Azure Backup Server (MABS), and System Center Data Protection Manager (SCDPM), reporting on the backup jobs, events, and statuses of and through those technologies, is not supported yet.

I highlight the word yet because it makes sense that the Azure Backup team work towards unifying the monitoring, reporting, and management of all backup technologies and uses through a single interface. We just have to be patient.

 

Conclusion

I am personally very excited about these Reports, as they can help in quickly getting an understanding of your Backup landscape across the environment.

They can help to answer questions like:

• Which protected servers use the most backup storage and have the highest impact on billing?
• What were the top causes of job failures yesterday?
• Which backup items had maximum data transferred last week?
• Which backup items had no successful backups yesterday?
• Which virtual machines had longest running jobs last week?
• Which folders take the longest time to restore?
• Which data sources generated the most critical alerts in last week?

You can get started and try out these new Azure Backup Reports now.

Another useful reference to be aware of is the data model for Azure Backup reports. This will help you get started on either customizing the existing PowerBI content pack dashboards and reports or start creating your own using the same data.

And if you have an idea, suggestion, or feedback, don’t forget to share it on the Azure Backup Uservoice.

Backup is NOT Disaster Recovery, Why You Need Both to be Fully Covered!

A few years ago I had the privilege to attend Azure Site Recovery Boot Camp training at Microsoft’s Silicon Valley in Mountain View, CA. The experience was amazing, and I enjoyed it. In fact, we received a first-hand preview of the (yet to be released) ASR VMware replication/protection through the newly acquired InMage technology.

The training was led by Vishal Mehrotra, and one thing that he mentioned has stuck with me every since.

Simply: Backup is NOT Disaster Recovery.

Although that is a simple statement and may seem obvious, it is powerful.

 

Business Continuity & Disaster Recovery (BCDR)

When most organization think about BCDR, they usually only think about Backup. But that’s only one piece of the puzzle. The other piece is Disaster Recovery.

Here’s a quick definition of each component:

Disaster Recovery

When your sites, servers or applications have a catastrophic failure and you are required to run them in Azure or a secondary datacenter.

 

 

Backup
When your data is corrupted or lost and you need to restore your data or infrastructure to the original location or a new location.

 

 

 

 

Azure Backup vs. Azure Site Recovery

Both services are related, in that they both backup and restore data. But they have different purposes. Azure Backup is used to protect and restore data at a more granular level. So, if some files become corrupted, you can use Azure Backup to restore them. But, if you wanted to replicate the configuration and data of a system to another datacenter, then you would use Azure Site Recovery.

Azure Backup protects data on-premises and in the cloud. Azure Site Recovery coordinates replication, failover, and failback. Both services are important because your disaster recovery solution needs to keep your data safe and recoverable (Backup) and keep your workloads available (Site Recovery) when outages occur.

[table id=1]

 

Azure Recovery Services

In the Classic Azure Portal, Site Recovery and Backup were separate services. In the modern Azure Portal, these 2 services are contained within the Recovery Services Vault service.

 

Azure Site Recovery (ASR)

Azure Site Recovery (ASR) supports multiple scenarios, including Disaster Recovery, Migration, and Dev/Test.

• Automated VM level Replication and Failover
  • Continuous log based replication (Not snapshots!)
• RPO of seconds and RTO of minutes
  • 30 seconds, 5 mins, 15 minutes
• Planned and unplanned failover
• Orchestrated Recovery Plans for Disaster Recovery
• Failback support
• No impact DR Drills with Test Failover
• Migrate to Azure from anywhere
• Azure to Azure DR (NEW)
• Create on-demand test copies in Azure

 

ASR Use Cases

• Disaster Recovery
  • Single click Application recovery
  • Compliance Assurance without impacting Production
  • Disaster Avoidance – Hurricane warning
  • Failover during real disasters – Fire, Earthquake, etc.
• Migration
  • No impact on production during replication – Just minutes of downtime during cutover
  • Test before migration using Test failover
  • Free Migration
• Dev/Test environment
  • Using Test Failover get an identical copy of your production environment
  • Use this for testing different scenarios like Patch Tuesday, New release testing
  • Deploy new patches/bits to production with confidence.

 

Azure Backup (AB)

Azure Backup (AB) supports backups across your heterogeneous environment, including VMware virtual machines.

• Azure Backup is part of OMS add-on as well as sold standalone
• First party SaaS service running in public Azure
• Backs up Windows Server, SCDPM, IaaS VM to Azure
• Supports file/folder, VMs (Linux & Windows), SQL Server, Exchange Server, SharePoint Server
• Flexible long term retention policy
• Granular recovery

 

AB Use Cases

• Accidental deletion
  • In the case that someone accidentally deletes business critical data, or that data become corrupted
• Patch Testing
  • Take a backup (and restore) if patch testing fails or causes application issues
• Alternative location recovery
  • Create a copy of your workload in an alternative location, whether for disaster recovery purposes or testing
• Security
  • Secure your data from the growing threat of ransomware attacks

 

A Note About Encryption

Both Azure Backup and Azure Site Recovery include encryption.

With Azure Backup, the data is encrypted using AES256 and is transmitted over HTTPS. The data is also encrypted at rest, and you store and keep the encryption passphrase locally, it is never sent or stored in Azure.

With Azure Site Recovery, the data being transmitted is never inspected or intercepted. Only the metadata is used for orchestrating replication and failover. Both encryption-in-transit and encryption-at-rest (in Azure) are supported.

Site Recovery is ISO 27001:2013, 27018, HIPAA, DPA certified and is SOC and FedRAMP compliant.

 

In Conclusion

So in conclusion, both backup and site recovery is needed to ensure you are fully covered and have a complete BCDR solution. If you forego one piece, then you only have half the puzzle, and could be in a compromising situation.

By leveraging Azure Backup (AB) for long-term data retention (including protection against ransomware), and Azure Site Recovery (ASR) for quick failover with low RPO/RTO and on-demand copies for testing; your BCDR strategy is complete, and you will be ready for anything.