Demystifying Analytics Audit

We suggested in our previous post that it makes sense to conduct an objective third part audit of Analytics projects, aka an Analytics Audit, even if it costs some extra money. We also enumerated the outcomes of such an Analytics Audit.

In many situations, the organizations have data and so they develop statistical models, without any regard to the choice of models or appropriateness of those models or the robustness of process adopted to develop these models. An Analytics Audit provides an objective third party assessment of these data-based Analytics projects, with respect to their choice of statistical models, the appropriateness of those models for the business context, and also the robustness of process adopted to develop these models.

Also, some organizations hire external consultants to build statistical/Analytical models for them as inputs to specific applications. Later, when they need to make the transition to in-house team(s), they may not be in a position to validate these models in terms of relevance and may not know how to keep these models current through periodic evaluation. Here again, the services of an experienced Analytics Audit professional would help in internalizing the Analytics models and also to set up a rhythm for periodic evaluation of these models’ relevance and currency.

But, what exactly is Analytics Audit and how it should be done? In our inaugural post on Why Analytics Audit, we had delineated the steps involved in a Typical Analytics Projects Approach. In this blog post, we would link Analytics Audit steps with the Analytics projects approach to highlight how an objective audit of Analytics projects would typically be carried out. The steps involved in Analytics Audit are listed in Table 2. This table also lists some of the questions that need to be asked and answered at each of these steps to ensure that Analytics projects achieve the organizational objectives, for which these projects were undertaken in the first place!

Table 2 – Analytics Audit Steps Involved in a Typical Analytics Projects Approach
S. No. Typical Analytics Projects Approach Analytics Audit Steps Questions to be Asked
1. Harvesting in-house data Checking quality, completeness and comprehensiveness of data Have we looked into all available in-house sources of data?
2. Data driven decisions (3D) identification Validating business requirement for those data driven decisions (3D) What are the decisions we want to make based on available data?
3. Organizational metrics delineation Validating appropriateness of organizational metrics Do the metrics measure what they are supposed to measure?
4. Metrics of interest prioritization Validating business logic for metrics prioritization Which are the pressing issues we want to address using data?
5. Base-lining of priority metrics Validating descriptive statistics of prioritized metrics What is the current state of prioritized metrics?
6. Analytical / statistical modeling Validating the statistical methods chosen for modeling

Validating the appropriateness of those chosen models

Validating the robustness of process of statistical modeling

Which are the most appropriate statistical modeling methods for the available data?

Are the models developed following standard modeling process?

7. Modeling results evaluation through 3D Validating modeling outcomes in the context of decisions of interest Does the modeling output helps in better decision-making?
8. Operationalization of modeling outcomes Validating incorporation of modeling output in actual decision-making Are the decisions truly data driven?

Have we incorporated Analytics projects’ output into the decision-making process?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s