Extracting useful signals from flawed sensor data: Developing hybrid data-driven approaches with physical factors

Abstract

Increased availability and affordability of sensors, especially water quality sensors, is poised to improve process control and modelling in water and wastewater systems. Sensor measurements are often flawed by unavoidable influent complexity and sensor instability, making extraction of useful signals problematic. Although a natural reaction is to put extra effort into sensor maintenance to achieve more reliable measurements, useful signals can be extracted from those unqualified signals by appropriate usage of available data-driven tools instructed by physical factors (e.g. prior process knowledge, physical constraints, phenomenal observations). Such methodology is herein defined as hybrid approaches. While the concept of coupling physical factors into data-driven tools is not new in downstream applications such as process modelling and control, little literature has explicitly applied it in the first and equally important step - signal processing. With flawed influent five-day biochemical oxygen demand (BOD₅) sensor measurements as an example, this paper provides a comprehensive case study demonstrating how physical factors were incorporated throughout the procedures of processing a flawed signal for its maximum value. Results showed that useful signals were extracted and validated via an assembly of well-established machine learning tools, whose performance was improved with physical factors. An Improved Standard Signal Processing Architecture (ISSPA) is also proposed based on the results of this research.

Keywords: Automation; Data quality; Machine Learning; Pattern separation; Sensors; Similarity.