Web platform for reconstructing 3D body parts

Innowise has developed a revolutionary tool for automatic 3D reconstruction of bones, skin, and other organs from X-rays and CT scans based on ML algorithms.

Customer

Industry

Healthcare, IoT

Region

Client since

2021

Our client is a medical equipment company manufacturing high-tech devices and software that assist clinicians in their daily work.

Detailed information about the client cannot be disclosed under the provisions of the NDA.

Challenge

As the healthcare industry evolves, new disruptive technologies are constantly emerging. Since surgery requires competence and accuracy, doctors need precise medical equipment that mitigates human errors and prevents unforeseen circumstances.

Our client required a 3D body modelling software capable of recreating bones, skin, and other organs from X-rays and computed tomography. By converting flat scans into three-dimensional volumetric models, medical practitioners would be able to improve visibility in treating patients and gain more insight into diseases and abnormalities. Students and interns would also use these 3D medical models to practice diagnostics and surgical procedures before performing them.

Solution

Our main task was to organically integrate the 3D construction platform into the customer's ecosystem and make it compatible with X-rays and CT scans exported from radiology, cardiology, and other laboratories so that they could be accessed across hospital workstations and personal laptops.

DICOM compatibility

Beforehand, we ensured that our web platform seamlessly works with DICOM files. Digital imaging and communications in medicine (DICOM) format is a common standard for exchanging medical imaging information and related data. Following this step, we emphasized additional security protections since DICOM files contain confidential health information.

As a result, our dedicated developers have created a space where all the imported DICOM files with data about patients, their diagnoses, treatment, dates, and the results of testing are stored.

From X-rays and CT scans to 3D visualizations

Though the non-contrast technique is accessible for 3D reconstruction, intravenous (IV) contrast scans (colourless liquids based on iodine) are recommended for more accurate 3D visualizations.

As soon as the X-ray or CT scan is downloaded into the system, it takes only a couple of clicks to turn black-and-white images into three-dimensional reconstructions. To determine the level of 3D detail, clinicians manually set threshold attenuation values. While the platform scans each CT slice line by line, it records the exact coordinates of each pixel that shows an attenuation value greater than the threshold. Then, these selected pixels represent voxels that contain body fragments denser than the selected threshold. As a result, after these manipulations, volumetric 3D reconstructions appear.
Once 3D rendering is completed, clinicians can manage objects through a convenient toolbar with a magnifying glass to zoom in/out, a gradient shade bar to add/remove the skin, tissue, muscles, and bony structures, and scissors to cut away excess parts. Nevertheless, the main tool is a cube that allows practitioners to rotate an image on its axis and give a more accurate picture of the pathology.

Smart ROI manager

To highlight pathology, our team developed an advanced ROI (region of interest – the boundaries of a tumour) manager. Here, doctors highlight pathologies so that they are immediately recognizable in the 3D reconstructions after rendering. By placing dots on the tumours, clinicians measure the extent of lesions to make informative decisions on surgical operations. Furthermore, clinicians can rename and highlight pathological zones in different colours so they stand out from healthy areas. To make segmentation even more precise, our team set thresholds, pixel values, and preliminary previews to allow for more detailed 3D customization. This includes generating detailed reports with anatomical annotations and labels, as well as measuring distances between organs for more accurate surgical planning.

Once all processing stages have been completed, practitioners can export and share the 3D image, setting asses according to users’ roles.

Technologies & tools

Back-end

Python, FastAPI, PyQt

Front-end

JavaScript, React

Databases

MS SQL Server

ML, MLOps

Weights and Biases, MLFlow, PyTorch, OpenCV, TensorFlow, Keras, ONNXRuntime, PyDICOM, Albumentations

Cloud

AWS (S3, EC2, Lambda), AWS SageMaker (Studio, Model Monitoring, Inference endpoint)

Qase, Postman, Swagger, TestFlight, Arduino, Thonny

Process

Even though the project was ambitious and challenging, our specialists were able to complete it successfully. First, our specialists estimated the scope of work and evaluated major milestones. To meet technical and business requirements, we chose the best-suited tech stack based on our extensive expertise.

Our dedicated team used Python to create the 3D medical modelling software and ensure smooth third-party integrations. Since costly hardware on the client side was not financially rational, we took full advantage of AWS capabilities to unravel cloud software architecture. Through API gateways, we also developed a desktop version that performs the same as the web platform.

To make 3D reconstruction accurate and reliable, we used different ML tools and approaches for solving detection, classification, and segmentation tasks, as well as data labelling. Additionally, our project team employed ML capabilities and computer vision to increase the level of training models. To meet the customer’s requirements, Innowise took several concurrent approaches regarding native 3D and image slice processing. As a result, we presented an innovative 3D rendering tool with an ML-based automatic pipeline for retraining and putting models into production tailored to medical needs.

Our team worked based on Scrum agile development methodology with regular team meetups and communication via Google Meet. Currently, the project is in progress with Innowise continuously working on further developing the platform and ensuring integrations with third-party medical apps and services.

Team

Project Manager

ML Engineers

Back-End Developers

Front-End Developers

UI/UX Designer

QA Engineers

Results

3D modelling in the medical industry provides amazing opportunities for reconstructing bones from computed tomography (CT) X-ray scans in a non-invasive manner. As a result, our top-tier 3D rendering platform enables professionals to precisely measure tumour areas and other pathologies, monitor organs over time, evaluate tissue composition, and accurately assess fractures without actually having to touch a patient. From now on, doctors accurately view the anatomy and diagnose various diseases that are invisible through traditional methods. Moreover, our solution allows for generating detailed reports with anatomical annotations and labels, as well as measuring distances between organs for more accurate surgical planning. With the help of our platform, surgeons can now plan their operations in a more precise and efficient manner.

Project duration

January 2021 - in progress

Related cases

Contact us!

Book a call or fill out the form below and we’ll get back to you once we’ve processed your request.

What happens next?

Having received and processed your request, we will get back to you shortly to detail your project needs and sign an NDA to ensure the confidentiality of information.

After examining requirements, our analysts and developers devise a project proposal with the scope of works, team size, time, and cost estimates.

We arrange a meeting with you to discuss the offer and come to an agreement.

We sign a contract and start working on your project as quickly as possible.