We recognized in our previous post that just like in Academic world where we do double-blind peer-review of journal articles before they are published, we should have an independent review of Analytics projects as well, aka an Analytics Audit!
It also follows from the CMM (Capability Maturity Model) for the software development process in which we have audits of organizational capabilities. It follows necessarily that even though those capabilities are at the organizational level, we need to do an audit at the most basic level to ensure that all relevant processes are at the minimum requisite maturity level.
Similarly, we have PCMM (People Capability Maturity Model) for organizations, which go beyond any specific product or process, but deep into the realm of how we develop our people assets and how mature are those processes to develop and grow our most valuable resources, our human resources.
In the international banking domain, we have BASEL norms. Actually, we have more than one set of norms, with progressively deeper level auditing of Banks’ risk practices. And we do need to do audit of their processes, IT systems as well as the quantitative risk models so as to adequately provide for the risk capital!
In the Analytics domain, we have multiple maturity models measuring the capabilities of organizations across various dimensions. Most of these Analytics Capability Maturity Models (ACMMs) are propagated by the vendors of Analytics products and services. Though there are other Analytics Maturity Models (AMMs), which are offered either by non-profit organizations, like INFORMS or is postulated by researchers and practitioners.
Then, there is International Institute for Analytics, which provides Analytics research and advisory services. They have their own DELTA model for assessing Analytics maturity of organizations, led by Thomas H. Davenport.
In all these AMMs, there is an underlying assumption that an organization’s Analytics model building capabilities are commensurate with its level of maturity as per the assessment. It is implicit in these AMMs that if we provide for the elements identified by them, then we would have good Analytics model building capabilities. This may be true for some of the organizations, but may not hold for some of the others. To elaborate, the most common statistical inference anyone would draw is about correlation between two sets of data and that could be quite misleading! If someone doesn’t agree, she may look at the spurious and absurd correlations listed here!
So, what is the way out? As has been suggested in our previous posts, it makes sense to conduct an objective third part audit of Analytics projects. If outcome of the Analytics Audit confirms that the Analytics models are good, that gives us a confidence in the Analytics model building capabilities of the organization. If outcome of the Analytics Audit suggests that the Analytics models are not good enough and need improvement, that gives us a handle to do next iteration to make them good! Well, it may cost an extra bit for third party objective Analytics audits, but it saves a lot of heartburn and expenses downstream!