DevOps automation for laboratory management system: over 70% reduction in compute costs

Innowise team optimized the cloud infrastructure and automated CI/CD processes for a market leader in DNA testing, increasing the scalability and efficiency of their data processing operations.

Customer

Industry

Healthcare

Region

USA

Client since

2023

Our client is a leader in the biotechnology field, specializing in DNA testing and microbiome analysis. They are known for providing rapid and reliable analyses using advanced molecular techniques, catering to a broad spectrum of applications in human health, agriculture, and environmental studies. Their services range from study design to comprehensive data analysis, contributing to research and development in the healthcare industry.

Detailed information about the client cannot be disclosed under the terms of the NDA.

Challenge

Issues with Infrastructure updates, migration to GitHub, and CI/CD optimization

Our collaboration with the client, initiated in 2023, was primarily centered around addressing their complex DevOps challenges. The client faced with a multi-faceted problem set, each aspect essential for their operational efficiency in microbiome research:

Infrastructure expansion and migration: The client was in the process of expanding their AWS environments with upgraded architectures. This expansion entailed the migration of several legacy environments into new systems.
Legacy infrastructure management: Another aspect of their challenge was the management of legacy Terraform code and Kubernetes cluster resources. The existing setup was cumbersome and outdated, requiring a substantial overhaul to meet current and future demands.
IP address pool depletion: A critical concern was the depletion of IP addresses caused by inefficient subnetting in their network. This hindered their current network management and posed a significant threat to their future expansion plans.
Migration from GitLab to GitHub: The client was also looking to transition from GitLab to GitHub, which included the integration of GitHub Codespaces. This move aimed to facilitate remote development and avoid the need for storing code on local machines, increasing security and collaboration.

Optimization of CI/CD Processes: There was a pressing need to optimize their continuous integration and continuous deployment (CI/CD) processes. Improving these processes was essential to enhance the deployment speed and reliability.
Implementation of Blue/Green Deployment Strategy: Lastly, the client sought to implement a Blue/Green deployment strategy. This approach was aimed at minimizing deployment risks and ensuring seamless rollouts of new features and updates.

The client expected our team to address these challenges and enhance their operational efficiency, reduce costs, and accelerate their development process. As they specialized in microbiome research with vast data processing needs, improvements in their DevOps practices were essential to bolster their capability to deliver services swiftly.

Solution

DevOps automation for LMS

Our solution for the client focused on three core areas: infrastructure optimization, DevOps automation, and CI/CD process enhancement.

Infrastructure optimization

We started our collaboration by overhauling the client’s infrastructure. Our team opted for a serverless architecture using AWS services. This approach was critical for handling the vast amounts of data processed in microbiome analysis.

AWS environment & legacy migration: We expanded the AWS environments, incorporating new architectures that supported the client’s growth and diversified needs. The legacy systems were smoothly migrated to these new environments with no disruption to ongoing operations.
Network management and IP address allocation: The challenge of depleted IP addresses was tackled by implementing a strategic subnetting approach. This resolved the immediate issue of address shortage and streamlined future network expansion.
IaC refactoring with Terraform and Terraspace: Our team restructured the Infrastructure as Code (IaC) setup, migrating from legacy Terraform to Terraspace. This move enhanced the manageability and scalability of the infrastructure code, allowing for multi-account management.

DevOps automation

Development operations practices were at the heart of our solution, aimed at enhancing DevOps infrastructure automation and simplifying processes.

Implementation of Karpenter: Introducing Karpenter marked a significant milestone in resource optimization. This tool reduced the number of necessary machines and slashed compute costs by 70%.
Argo Project Tools integration: We utilized ArgoCD, Argo Rollouts, and ArgoCD Image Updater to automate deployment processes. This setup freed the developers from manual pipeline management, allowing them to focus on core development tasks.
Blue-Green-Canary deployment model: Our deployment strategy combined the strengths of the Blue/Green and Canary models. This hybrid approach expedited development cycles and minimized deployment risks, marking a significant advancement in the client’s operational capabilities.

CI/CD process enhancement

The final pillar of our DevOps automation services focused on refining the CI/CD processes. This involved a series of strategic implementations aimed at boosting performance.

Migration to GitHub and Codespaces integration: We facilitated the client’s transition from GitLab to GitHub, setting up GitHub Codespaces for more secure and efficient code management. This shift enhanced code security and streamlined development workflows.
Pipeline refactoring and automation: We restructured CI/CD pipelines to separate the continuous deployment from integration. This segmentation improved the deployment process’ efficiency and reduced potential errors.
Monitoring and cost optimization: Implementing Grafana Stack and Prometheus allowed us to establish a comprehensive monitoring system. This system assisted in tracking AWS costs and optimizing resource usage for cost-effective operations.

Technologies

Back end

Python; FastAPI

Front end

React; Node.js; Material UI

Databases

PostgreSQL

Platforms

Web

Cloud Services & Platforms

AWS; Lambda Functions; EKS; ECR

Containerization and Management

Docker; Kubernetes

Infrastructure as Code (IaC)

Helm; Terraform; Terraspace

Continuous Integration & Deployment

ArgoCD; Argo Rollouts; ArgoCD Image Updater; GitLab; GitHub; Karpenter; Kubecost

Monitoring and Logging

Grafana; Grafana Loki; Promtail; Prometheus

Network Management

Kubernetes CNI plugin

Server Monitoring

Prometheus

Quality Assurance

Playwright

Operating Systems

Linux; amd64; arm64

Version Control Systems

GitHub; ArgoCD

Integrations

Microsoft AD SSO

Process

Our process was based on the Agile methodology with a Scrum framework that emphasized flexibility, continuous improvement, and rapid delivery. We organized our work into one-week sprints, with daily stand-up meetings at 18:00 CET to ensure alignment and address any immediate concerns. We held sprint planning sessions every Friday, where the team collectively reviewed and planned tasks for the upcoming week. These sessions were coupled with retrospectives and presentations.

Task management and communication

Our developers had full ownership of the backlog, giving them the autonomy to manage and plan tasks. Direct communication with the Product Owner (PO) was infrequent, as the primary goals and requirements for tasks were set and often detailed by the project’s lead designer.

Development and documentation

Our approach to task implementation involved developing Proof of Concept (POC) models for most tasks, followed by creating detailed diagrams reviewed by the team. This practice allowed us to validate ideas and ensure feasibility before full-scale development. We thoroughly maintained the project documentation to ensure the smooth execution of our tasks.

Cloud security and DevOps automation challenges

A significant aspect of our project was managing and optimizing the client’s extensive laboratory data reports, which amounted to petabytes of data. This massive data size resulted in substantial AWS costs.

Sprint reviews and adaptation

We conducted sprint reviews at the end of each week, which included demonstrations of new features and discussions about the upcoming sprint. These meetings were crucial for adapting our strategies to any changing requirements.

Team

DevOps Engineers

Front-End Developer

Full-Stack Developer

Results

70% reduction in compute costs

Our collaboration with the client has led to significant improvements in their DevOps infrastructure and processes. The key results achieved are as follows:

Cost optimization: The implementation of Karpenter successfully reduced the number of machines used in the cluster from 15 to 6, leading to a 70% reduction in compute costs. This substantial cost saving has greatly enhanced the cost-effectiveness of their operations.
Development process acceleration: The introduction of the Blue-Green-Canary release model greatly accelerated the development process. This strategic implementation expedited development and added a layer of security to the release process.
Enhancements through DevOps automation: Utilizing Argo Project tools has notably enhanced the development pipeline. The client’s development team can now update the Kubernetes cluster with new application versions by simply pushing code, with the entire build and deployment process being automated. This has greatly reduced manual intervention in routine tasks, allowing engineers to focus more on strategic development activities.
Infrastructure management simplification: The migration of Terraform code to the Terraspace framework, along with the refactoring of Kubernetes cluster management, has simplified infrastructure management. These changes have made managing the infrastructure more straightforward and efficient.
Improved observability: The integration of Grafana Stack and Prometheus for additional metrics has significantly enhanced system observability. This improvement has resulted in better data monitoring, facilitating more informed decision-making and system management.

Currently, we are in the final stages of setting up the AWS environments, ensuring they are tailored to meet the client’s requirements. Looking ahead, there are plans to further improve the release model, building upon the successful implementation of the Blue-Green-Canary strategy.

Project duration

July 2022 - Ongoing

70%

reduction in compute costs

development speed increase

50%

decrease in manual intervention

Related cases

Contact us!

Book a call or fill out the form below and we’ll get back to you once we’ve processed your request.

What happens next?

Having received and processed your request, we will get back to you shortly to detail your project needs and sign an NDA to ensure the confidentiality of information.

After examining requirements, our analysts and developers devise a project proposal with the scope of works, team size, time, and cost estimates.

We arrange a meeting with you to discuss the offer and come to an agreement.

We sign a contract and start working on your project as quickly as possible.