Data management in banking: 34% reduction in data processing time

Innowise has developed a centralized repository to store, process, and secure large amounts of data related to corporate clients, banking accounts, and payment transactions.

Customer

Industry

Banking

Region

Client since

2021

Our customer is a prominent financial institution that offers retail banking, corporate banking, wealth management, insurance, brokerage services, and more. Founded in the early 20th century, they have evolved significantly over the decades, embracing new technologies and practices to improve customer experience and operational efficiency.

Detailed information about the client cannot be disclosed under the provisions of the NDA.

Challenge

Disparate banking-related data across varios sources that hindered decision-making

Our client faced challenges managing and extracting value from the vast and growing volumes of data across corporate clients, banking accounts, and payment transactions. With data dispersed across multiple legacy systems, the bank experienced significant data processing and analysis challenges, struggling to gain timely insights for decision-making. Besides, the existing infrastructure lacked scalability, while maintaining various obsolete systems became unsustainable.

Another issue stemming from this central challenge was the difficulty of complying with stringent regulatory banking-related requirements. Data storage and management systems were scattered, making it difficult to track, report, and audit data effectively. This increased the risk of involuntary violations as banking reps had to spend much time collecting and verifying data.

As a result, the client commissioned Innowise to build a robust data lake architecture that could consolidate their diverse data sets into a single, scalable, and secure environment for data management in banking. They sought a convenient solution to keep track of information about clients, accounts, and transactions and comply with regulatory standards, accelerating the cycle from raw data inputs to actionable business insights.

Solution

Data lake development to store cleansed and structured banking data

We developed a centralized database to store and integrate data streams gathered from a variety of sources, including electronic banking, mobile applications, and social media. Our experienced specialists implemented a medallion lakehouse architecture, focusing on an ACID-driven, multi-layered approach to build a single source of truth for banking data storage.

Data layering

The foundation of the data lake, the bronze layer, contains raw data ingested from various sources like JSON files, RDBMS, and more, securely stored in its original form. Based on the bronze layer, the silver layer refines this data, cleansing and normalizing it for advanced analytics. Ultimately, the pinnacle of the data structure, the gold layer, contains business-level aggregates for high-level reporting and dashboarding, enabling the bank to generate actionable insights.

Our project team identified all potential data sources, including transactional systems, customer databases, online portals, and more. Our specialists mapped each data source, understanding its format, update frequency, and relevance. As the core step, we developed automated data ingestion pipelines using ETL (extract, transform, load) tools to handle different data formats like CSV, JSON, XML, and RDBMS. Depending on the nature of the data source, we established real-time or batch-processing pipelines.

Real-time pipelines were used for data streams that required immediate processing (like transactional data), while batch processing was reserved for less time-sensitive data. As data entered the bronze layer, it underwent an initial validation process, checking for integrity, format consistency, and any corrupt or incomplete records.

Silver layer: data processing and refinement

At this stage, our project team focused on enriching the raw data from the bronze layer and transforming it into a more structured and usable format. Our vetted developers identified and corrected typographical errors, data format inconsistencies, and discrepancies, as well as removed duplicate records to avoid misleading insights. We implemented data imputation and flagging strategies for datasets with missing values, sending these records for further review depending on the nature and significance of lost data.

Then, our project team enhanced the data by adding relevant context or additional information. For example, we augmented transaction data with customer demographic information, enabling a more comprehensive analysis. Once data was refined and aggregated, we applied indexing techniques for faster querying and retrieval. Finally, data from different sources are cross-referenced and linked, while similar information from various sources is consolidated into unified datasets, making it easier to perform holistic analysis. By ensuring the data is clean, consistent, and well-structured, we paved the way for advanced analytics and business intelligence in the gold layer.

Golden layer: analytics-ready data

The gold layer is the apex of our data lake architecture, where data is transformed into analytics-ready information specifically tailored for high-level analysis, reporting, and decision-making. Data from the silver layer is further aggregated to create comprehensive, high-level summaries. We focused on summarizing data in ways that align with key business metrics and objectives, such as credit risk assessments, market trends, or customer segmentation.

Our developers designed and implemented interactive dashboards and reports, giving the bank’s decision-makers real-time insights and visualizations. With a focus on security, we established a robust data governance framework to manage the data’s quality, usability, and security. Our engineers ensured scalable architecture, accommodating growing data volumes and complexity without degradation in performance and maintaining the integrity and reliability of the analytics output.

In the gold layer, we turned data into a strategic asset, allowing the bank to make informed decisions, better understand customer needs, and stay ahead in the competitive banking industry.

Thus, based on refined banking-related data, Innowise enabled the client to implement the Next Best Action (NBA) and Next Best Offer (NBO) methodologies. NBA prioritizes a customer-centric approach, analyzing recent interactions to suggest the most suitable actions, including sending birthday messages, improving service quality, gathering feedback, providing onboarding instructions, and more. By leveraging predictive analytics, NBA selects actions tailored to the customer’s current situation, aiming for positive outcomes. In turn, NBO optimizes the selection of personalized offers from a client’s extensive product range. NBO automatically evaluates and suggests products likely to resonate with customers by delivering offers at the right time, price, and through the most effective channels.

Additionally, our developers consolidated data from different tables and models stored in the data warehouse to create comprehensive, cohesive, and practical profiles for each customer, enabling better-informed decisions and actions. The comprehensive and thoughtful approach to managing analytics-ready data ensures that the bank can leverage its data to its fullest potential, increasing conversion rates and driving growth.

Technologies

Data engineering

Cloudera Data Platform, Hadoop, Spark, Airflow

Back-end

Python, Fast API, Scala, Akka

Database

MS SQL Server, Oracle

BI tools

Power BI, SSRS, QlickView

Process

Innowise won a competitive tender before diving into the project. Following our success in the tender, we embarked on the software development process, demonstrating our skills and alignment with the client’s vision.

We created a PoC, aiming to utilize Kubernetes and moving away from the existing Cloudera-based systems. However, due to the limitations of the customer’s current data center, they showed signs of hesitation regarding Kubernetes implementation and support.

During the discovery phase, our project team conducted thorough research to understand the current data landscape and identify key data sources and requirements. Later, we created a detailed design that incorporated the bronze, silver, and gold layers for data processing and refinement, ensuring the data flows seamlessly in accordance with ACID principles. Then, we performed extensive testing to guarantee the integrity and performance of the data lake, implementing a feedback mechanism for continuous improvement. Finally, our project team successfully deployed the data lake, integrating it with the bank’s existing systems and providing training and support to the bank’s employees.

Team

Business Analysts

Project Manager

BI Developer

Data Engineers

DevOps Engineer

Data Quality Engineer

Results

Streamlined data processes and regulatory compliance after data lake developement

The implementation of the data lake for our banking client has yielded transformative results across various dimensions of their operations. Previously struggling with fragmented and unstructured information spread across multiple sources, they now effortlessly access corporate client data, banking accounts, and payment transaction information with predictability and ease. The bank’s teams now operate reliable and consistent data, paving the way for more accurate analytics and reporting. The consolidation of data into a single, scalable lakehouse architecture has led to significant cost savings in data storage and management as the result of eliminating redundant systems and streamlining data processes.

The integration of automated data pipelines and streamlined data layers has significantly reduced data processing time, enabling faster decision-making and more responsive customer service.

Also, our client has enhanced customer relationship management and business performance by delivering personalized, timely, and relevant actions and offers based on refined and standardized data. This approach has increased conversions and revenue and optimized marketing budgets by targeting tailored offers only to interested prospects.

Moreover, with the new data lake, compliance reporting has become more efficient in an industry where legal violations can have significant consequences.

Project duration

March 2021 - Ongoing

34%

reduction in data processing time

26%

improvement in compliance reporting

Related cases

Contact us!

Book a call or fill out the form below and we’ll get back to you once we’ve processed your request.

What happens next?

Having received and processed your request, we will get back to you shortly to detail your project needs and sign an NDA to ensure the confidentiality of information.

After examining requirements, our analysts and developers devise a project proposal with the scope of works, team size, time, and cost estimates.

We arrange a meeting with you to discuss the offer and come to an agreement.

We sign a contract and start working on your project as quickly as possible.

Data management in banking: 34% reduction in data processing time

Customer

Challenge

Disparate banking-related data across varios sources that hindered decision-making

Solution