In this series of articles, I’m going to describe a low-latency risk assessment & mitigation platform built on top of a relational SQL database. The purpose of our setup is to provide a live query subsystem capable of handling large quantities of requests whilst keeping load bursts isolated from the main operational cluster.

Abstract

The information we’re going to work with is going to be a common fintech dataset, revolving around financial transactions and risk ratings of individuals. It could be utilized in a variety of applications, including:

• Line of credit products
• Insurance platforms
• Credit card issuers
• Installment loan lenders
• Lead processors / verification centers

Scenario

Before we dive deep into the technical aspects, let’s briefly discuss the nature of our queries, which could be described by the following characteristics:

1. Our system needs to deliver answers rapidly in order to be used as a part of synchronous data processing. The fintech product we offer requires an unambiguous response from our platform in order to deliver its core functionality. Increased latency will result in degraded performance of our product, while lack of response will cause processing errors.

Example: if the Risk Mitigation queries are used to approve or deny credit card purchases, excessive processing times might lead to timed out transactions.

2. We want to rely on our existing data rather than on 3^rd party decision-making. If a 3^rd party service is used, it should only be considered an extra input for our own analysis. Our system needs to be capable of self-assessing the importance (weight) of each parameter being measured.

Example: if a Credit Rating agency data is used to evaluate a potential borrower, their scoring should only be considered one of the many factors being analyzed. By retrospective analysis of our existing loan book performance and historical Credit Ratings of its borrower, we should be able to benchmark this particular agency’s credibility.

3. Our solution needs to self-improve by acquiring more live operations data on a regular basis. It also must be able to identify insufficient information and auto-tune itself by supplementing alternative data sources.

Example: When evaluating new customers, a health insurance company looks at their age as a main contributing factor to assess their risk. As they decide to introduce new policies targeting pensioners for the first time, their existing portfolio will not have enough data to provide meaningful population samples. Another set of parameters has to be processed in order to provide risk mitigation intelligence for this new group of customers.

Action Plan

I am going to describe a software set used to achieve the goals outlined above in a series of short notes, each one focused on an individual aspect of the setup:

1. Relational Database
2. OLAP vs Deep Learning
3. Dynamic Mondrian Cubes
4. Implementation Constraints (ORM design, conflict monitoring)