Data Pre-processing to Identify Environmental Risk Factors Associated with Diabetes

This study explores the connection between environmental conditions and diabetes by employing a data-driven methodology that utilises admitted patient data from hospitals in New South Wales along with weather, pollution, and demographic data. The study acknowledges that sensor-derived environmental data often have missing values, necessitating improved data cleaning and imputation techniques. Hence, the objectives of this study are twofold: first, to develop a framework to analyse missing data and to create improved algorithms for cleaning and imputing them; and second, to identify environmental conditions associated with type II diabetes.

In line with this, the research evaluates existing techniques and introduces a novel data-cleaning framework enabling comprehensive data analysis and visualization for environmental data. It employs enhanced algorithms to fill large gaps in environmental data. The algorithms were evaluated against a set of existing methods, considering various scenarios with missing percentages ranging from 10% to 50% and gap sizes spanning from 50 to 500. To ensure reliable results, each scenario was repeated multiple times, and the positions of missing values were randomly changed in each repetition to avoid any misleading results due to chance. The algorithm consistently produces lower errors across all considered scenarios. The research then conducts spatial exploratory analysis to identify areas with higher rates of type II diabetes-related hospital admissions. Subsequently, statistical and machine learning models are employed to investigate the relationship between type II diabetes and environmental factors. The findings of this research hold significant value for policymakers, providing valuable insights into managing and mitigating diabetes.