Ai Editorial: Machine learning and fraud – “scores” not important; results matter

First Published on 16th October, 2018

Ai Editorial: There are key pointers – denial rates, false positives and fraudulent transactions – that underline the performance of any machine learning technique in fraud prevention. As for what is the utility of scores, they are not important; results matter, writes Ai’s Ritesh Gupta  


It is intriguing to understand how machine learning works – working on data, variables etc., and how is precise model worked out and refined to control fraud, be it for related to a payment, data breach or account takeover.

The machine learning system starts with a basic model which is trained and improved with datasets over time. It is important to pre-process the data. To improve the efficiency and accuracy of the system, the data can be pre-processed with data slicing and augmentation and be cleaned sufficiently before it is used to train the model.

Making it work to control fraud

In the case of a fraud solution, the system will be given training sets consisting of a given set of known fraudulent transactions and known non-fraudulent transactions, so that the system will learn to differentiate and filter away fraudulent transactions, says Justin Lie, CashShield’s CEO. 

Considering that various industries have differing levels of risk and exposure to fraud, the data collected from different industries may be customized. For instance, some data sets that may be collected from an airline merchant (and no other industry) would include: flight boarding times, whether the customer chooses to add a meal, whether the customer has an existing loyalty membership or whether seat preferences have been added.

A few algorithms modelled from the training sets will be put to the test with real life data, and thereafter, the algorithm with the least error will be chosen as the best algorithm. The amount of data and how relevant the data was used in deriving the algorithm will affect its accuracy. Over time, the algorithm must be constantly trained with data, especially with new data so that the margin of error can be minimized and inaccurately classified transactions (fraudulent as non-fraudulent and non-fraudulent as fraudulent) will be corrected.

Significance of “score” associated with machine learning 

There are key pointers – denial rates, false positives and fraudulent transactions – that underline the performance of any machine learning technique. As for what is the utility of scores, Lie says scores are not important; results matter.

“Most merchants would aim to increase their transactions and reduce their fraud, and the performance of any machine learning technique should be evaluated based on whether this goal can be achieved. Nevertheless, it is important to note that each merchant would have differing goals with respect to fraud; for some, raising acceptance rates and growing aggressively is most important, while for some others, minimizing fraud rates down to zero is the most important KPI,” says Lie.  

“With minimal risk, it is likely that overly strict filters are put in place and many genuine users have been blocked at the expense of lowering fraud rates. Therefore, the performance of the fraud solution would depend on the goals of the merchant. For example, taking in more risk may increase fraud rates slightly, but also lower false positives and rejection rates.”

Commenting on the significance of scores in terms of performance in controlling fraud and letting legitimate transactions go through, Lie said most fraud solutions on the market would be able to automate a good bulk of the transactions based on the score; extremely low scores will be rejected automatically and extremely good scores will be accepted. However, for the borderline transactions, a team of manual reviewers is required to make sense of the score. Generally, some guidelines will be given to the manual review team to look for further clues based on the data collected, and some working experience will be used, but most of the time the manual review team is relying on their gut feeling, which is affected by a risk-averse outlook to reject potentially genuine transactions to prevent fraud rather than to risk having passed a fraudulent transactions.

Therefore, fraud scores can only help a merchant this much, but ultimately, the fraud score is not the be all and end all in identifying fraud.

Human intelligence counts

Machine learning models would only still provide merchants with only a fraud score; to make sense of the score, fraud solutions or merchants would still need to rely on humans to make a decision.

“The problem here, is that humans are often risk-averse and would reject borderline risky transactions for fear that it could be fraudulent, and end up blocking more genuine customers than expected,” said Lie.

As such, a multi-disciplinary approach combining machine learning and other techniques is important to improve the efficiency and quality of the fraud detection process.


Follow Ai on Twitter: @Ai_Connects_Us