Determining Third-Party Risk Management Metrics for Cloud Service Providers

Cloud service providers (CSPs), also known as cloud vendors, are quickly becoming the norm in today’s business world. Many organizations are using CSPs to gain a competitive advantage and further their goals around innovation, while others are growing more reliant on this technology for their daily operations. In fact, Gartner predicted that cloud platforms will be considered a business necessity for most enterprises by 2028.

With this in mind, it’s important to consider how to use different metrics to measure and evaluate a cloud service provider’s risk and performance, particularly key risk indicators (KRIs) and key performance indicators (KPIs).

Let’s cover some tips to get started on determining which metrics to use and where to find the data you need. We’ll also provide some next steps to consider if you discover any significant changes in a cloud service provider’s KPIs or KRIs. 

How to Determine Cloud Service Provider Metrics 

Cloud service providers offer a wide range of scalable solutions and generally fall into one of three categories – infrastructure as a service (IaaS), platform as a service (PaaS), or software as a service (SaaS). Your organization should select KRIs and KPIs that are most relevant to the type of cloud service provider you’re using. 

Consider the following questions: 

• What type of data does the CSP have access to?
• Does the CSP support a critical product or service?
• Can the CSP’s actions (poor service, outages, cyber incidents, etc.) harm your organization’s reputation? 
• Is the CSP used to meet regulatory or legal compliance?
• How is the CSP being used to support your strategic goals?

In general, it helps to think about a cloud service provider’s risks posed to your organization when determining the types of KRIs and KPIs to measure. In other words, an increase or decrease in certain metrics would indicate that the cloud service provider is exposing your organization to elevated risk.

10 Common Cloud Service Provider KRI and KPI Metrics 

The following KRIs and KPIs are general enough to be used in most organizations. You’ll likely need to turn to a few different sources to gather the data. The first 6 metrics are most effective as KRIs and the data that comes from the cloud service provider itself. 

The remaining four metrics are better used as KPIs and can be collected from internal data. We’ve also included general examples of how these metrics might be reported for a single month but remember that the data will be unique to your organization and the cloud service provider. 

It’s important to collect data and compare it from month to month to identify any trends, emerging risks, or declining performance. Your organization’s interpretation of the data and benchmarks will largely depend on your organization’s risk appetite. 

The following KRIs come from the cloud service provider’s reporting:

1. Uptime percentage measures the amount of time that the cloud service is available to its users. This KRI should be tracked to ensure the service remains reliable and accessible.
   
   Example: Uptime percentage was 99.95% in May 2024
2. Average response time measures how long it takes for the cloud service provider to respond to a user request. A KRI that reflects high average response times implies that the service isn’t performing well or meeting the needs of the user.
   
   Example: Average server response time was 400 milliseconds for all web services during peak usage hours in May 2024
3. Security incidents should be tracked to validate whether the cloud service is secure, and whether it’s protecting user data. This KRI evaluates the potential risks that may arise from the cloud service provider’s poor security practices. 
   
   Example: Two security incidents were identified in May 2024
4. Downtime duration measures how long the cloud service is unavailable to users. This KRI should be tracked to monitor whether the cloud service remains stable and reliable.
   
   Example: Unplanned downtime duration totaled 0.2 hours in May 2024
5. Backup and recovery time measures the amount of time it takes to backup and recover data when a disaster or system failure occurs. This KRI evaluates whether the cloud service provider has effective backup and recovery procedures.
   
   Example: Cloud service was recovered within 2 hours after a system failure in May 2024
6. Capacity utilization measures how much of the cloud service’s resources are being used by your organization. This KRI helps you determine where there might be inefficiencies in the cloud service and where the capacity can be increased or reduced.
   
   Example: CPU and memory capacity utilization maintained 70-85% in May 2024

The following KPIs come from your organization’s internal sources, such as financial data, individuals who use the CSP, and contract management reports:

1. Cost per user is the amount of dollars your organization is spending to provide the cloud service to each user. This KPI should be tracked so you can determine whether the cloud service is cost efficient and whether there are opportunities to reduce costs. 
   
   Example: Cloud service cost $15 per user in May 2024
2. User adoption rate is the percentage of users who are actively using the cloud service. This KPI helps determine the cloud service’s success and identifies where user engagement can improve.
   
   Example: User adoption rate reached 85% in May 2024
3. Service level agreement (SLA) compliance measures the rate at which the cloud service is meeting contractual performance expectations. This KPI tells you how well the cloud service provider is performing and helps identify areas of improvement. 
   
   Example: SLA compliance reached 94% in May 2024
4. Customer satisfaction measures the number of users who are satisfied with the cloud service. This KPI helps track whether the cloud service is successful and helps identify areas that can be improved for customer satisfaction.  
   
   Example: Cloud service achieved a customer satisfaction rate of 89% in May 2024

4 Next Steps to Address Potential Cloud Service Providers Risks and Declining Performance  

If the KRIs or KPIs reveal any concerning data, it’s essential to consider your next steps and mitigate the risk of potential consequences, such as operational disruptions, cybersecurity incidents, and reputational harm. 

Consider these steps: 

1. Implement a formal remediation plan – Metrics that fall outside of an acceptable range should be addressed immediately to prevent any adverse consequences. The remediation plan should be timebound and include clear details on stakeholder roles and responsibilities. Remediation activities should also be tracked for progress and completion.
2. Increase performance reviews – Cloud service providers should undergo scheduled performance reviews at a frequency that aligns with their inherent risk and criticality. Here’s the recommended intervals for performance reviews:
   
   • Critical and high-risk vendors should be reviewed at least once per quarter
   • Moderate-risk vendors should be reviewed every six months to a year
   • Low-risk vendors should be reviewed as needed or before contract renewal 
   If the cloud service provider is performing poorly, these frequencies should increase until the issue is resolved.
3. Review your contract – There’s a good chance that your cloud service provider contract or agreement includes details on performance standards and expectations. It’s important to review your contract during any performance issues to verify whether your organization can receive any financial compensation, such as discounts, refunds, or service credits. 
4. Exit strategy – Depending on the severity of the declining performance, your organization may decide it’s best to terminate the cloud service provider’s contract and proceed with your exit strategy. Ideally, your exit strategy should have already been determined before you signed the contract. An exit strategy can refer to a few different options, such as switching to a new cloud service provider, bringing the outsourced activity in house, or terminating the outsourced activity altogether.

Cloud service providers can offer many exciting possibilities, but it’s important to regularly monitor their performance. Tracking and measuring KPIs and KRIs will continue to give your organization the ability to make more informed decisions about these third-party relationships.