IMPACT FACTOR : 5.212

ENHANCING DATA RELIABILITY AND INTEGRITY IN DISTRIBUTED SYSTEMS USING APACHE KAFKA AND SPARK (KEY IJW**********535)

• Suraj Dharmapuram, Rahul Arulkumaran, Ravi Kiran Pagidi, Dr S P Singh, Prof. (dr) Sandeep Kumar, Shalu Jain

Abstract

In distributed systems, data reliability and integrity are paramount for ensuring accurate and consistent data flow across various applications. Apache Kafka and Apache Spark are powerful tools that can be leveraged together to create robust data pipelines, effectively enhancing data reliability and integrity. Kafka is a distributed messaging platform known for its fault tolerance and ability to handle high-throughput data streams, making it ideal for real-time data streaming applications. Spark, a unified analytics engine, is highly compatible with Kafka, offering capabilities for batch and stream processing, which allows developers to process large datasets with low latency. Integrating Kafka and Spark provides a comprehensive solution to tackle challenges associated with data loss, duplication, and processing errors, which are common in distributed systems.