IoT Data Quality Assessment Framework using Adaptive Weighted Estimation Fusion

Abstract

Timely data quality assessment has been shown to be crucial for the development of IoT-based applications. Different IoT applications' varying data quality requirements pose a challenge, as each application requires a unique data quality process. This creates scalability issues as the number of applications increases, and it also has financial implications, as it would require a separate data pipeline for each application. To address this challenge, this paper proposes a novel approach integrating fusion methods into end-to-end data quality assessment to cater to different applications within a single data pipeline. By using real-time and historical analytics, the study investigates the effects of each fusion method on the resulting data quality score and how this can be used to support different applications. The study results, based on two real-world datasets, indicate that Kalman fusion had a higher overall mean quality score than Adaptive weighted fusion and Naïve fusion. However, Kalman fusion also had a higher computational burden on the system. The proposed solution offers a flexible and efficient approach to addressing IoT applications' diverse data quality needs within a single data pipeline.