Data pipelines with Apache Kafka: 36% increase in decision-making accuracy

Our initial task was to create an architecture to offload information from data sources and transmit it to Apache Kafka. First, we built a connector for Codebeamer, a comprehensive project management platform the client used for software development and collaboration. We chose Apache Kafka due to its exceptional ability to handle large-scale, high-throughput, and real-time data streams in a fault-tolerant, scalable, and distributed manner. 

Initially, Innowise’s specialists analyzed Codebeamer’s API documentation comprehensively to identify the most efficient methods for extracting project data, including work items, changesets, and user activities. We also examined the authentication mechanism, data request limits, and the API’s return formats.

Based on the API analysis, we designed the connector architecture with a focus on modularity, scalability, and fault tolerance. Our software engineers utilized Java to code the connector, which was responsible for linking to Codebeamer’s API, fetching data, and writing it to a Kafka topic. We implemented a converter to transform the data from Codebeamer’s format to a Kafka-compatible format. This included mapping various data fields to Kafka’s key-value pairs and handling schema variations. Finally, our project team managed robust configuration, enabling users to dynamically specify API credentials, polling intervals, and target Kafka topics.

In the first stage, the connector polled Codebeamer’s API to fetch new and updated data at configurable intervals. Then, it transformed the data into a Kafka-compatible format, ensuring each piece of information is represented as a discrete event. We utilized batch processing capabilities to efficiently handle large volumes of data without overwhelming Codebeamer’s API or the Kafka cluster.