Originally posted as 3 separate articles. Now combined, with additional detail.
In adopting analytics for audits, one of the main challenges is the volume of exceptions generated.
How can we overcome this?
Traditional audit sampling typically involves evaluation of between 5 and 50 items; consequently, the number of exceptions doesn't fall out of that range.
However, when we use analytics across full populations, the number of exceptions produced can be high.
The sheer volume can mean:
- Difficulty in eliminating false positives - noise!
- Unnecessary pain for business folk (who must deal with the mountain)
- Internal audit losing faith in analytics
There are various approaches to overcome this, e.g. progressively categorising results into specific buckets based on key characteristics, reviewing a sample of those, and then extrapolating the results of the reviewed sample to the remainder of the population of exceptions.
However, when the key characteristics can't be easily categorised e.g., when the exceptions are based on and include both structured and unstructured data - the traditional approach doesn't quite fit.
An alternate method that generally works well involves the use of machine learning techniques. A similar approach - but using different techniques - that can significantly reduce the number of false positives:
- providing business stakeholders with relevant information to consider
- providing audit stakeholders with the comfort that we did not simply discard results but used a well-structured and defensible approach to create a manageable set of exceptions for follow-up.
An example of how this is applied
An assurance review for a medium sized organisation.
The organisation's Internal Audit function is relatively small - fewer than ten FTE - but progressive, punching significantly above its weight, and respected by stakeholders.
The team decided to take a data driven approach, opting to cover ALL accounts and transactions for just over one year.
We had about 800m records.
KNIME- an open source analytics platform - was used to analyse the data, across various data sets, including:
- customer and account master files
- account links
- account transactions
- customer relationship management (CRM)
Because the CRM data was primarily free text, we used a set of natural language processing (NLP) techniques, providing a level of structure, and then blended the processed data with the other structured data sets.
With the data in a format that could be used easily, we performed several analyses.
- Recalculations - full population recalculation of transactions
- Master data quality checks (e.g. duplicate records)
- Profiling and understanding manual adjustments
- Checking the application of product rules.
- Confirming adherence to customer instructions, including account links
Most of these don’t need much explanation. But why offset account links?
Let’s explain why we decided to do this and the challenges that we faced.
Some customers have multiple accounts. In some cases - depending on the combination of products - linking those accounts can save customers money.
But the linking can easily fail because many systems were not originally built to deal with this type of account relationship.
The systems generally work well with standalone accounts. But combining them often means a patchy workaround with some scenarios that have not been envisaged or properly tested.
There has been a fair level of regulatory (and media) attention to such failures over the past few years, with hefty infringement penalties and costs.
So for our project, we decided to check whether the linkages were established properly. This means identifying expected links and then comparing those to the actual links that were in operation.
If we expected to see a link (identified for example in a customer interaction or complaint), and the link hasn’t been established, we would then need to investigate this as an exception.
We found thousands of potential exceptions.
We expected that many of the exceptions would turn out to be false positives, but we couldn’t possibly investigate them all to find the real exceptions.
How can we find the needles in that haystack?
The traditional approaches would typically look like one of these:
- select and test a random sample
- profile a representative sample (e.g., in a spreadsheet) to find common characteristics, to manually identify and eliminate false positives.
Why not opt for the traditional approaches?
- random sampling may have merit, but with techniques to better target anomalies now available, this is not defensible and doesn't add real value
- representative profiling can work with structured data, for a smaller set of features (columns) - but the free text data translated to over 5,000 columns, so this would not be feasible.
The analytics software that we were using has strong predictive modelling capability. So, we decided to make use of it.
This is the process; it sounds complicated, but is not difficult to implement:
More than 90% of the exceptions were eliminated (false positives).
We were left with a manageable set of a few hundred to investigate.
Model accuracy was approximately 70%.
Such a process is rarely going to be 100% accurate but this is certainly better than random sampling alone.
It can be defended.
Remember that this was achieved by a relatively small Internal Audit team.
Tools, approaches and techniques to improve IA's use of analytics are now readily available.
Are you using them?
This article is part of the assurance analytics series.