Big data quality prediction informed by banking regulation

Big data has been transformed into knowledge by information systems to add value in businesses. Enterprises relying on it benefit from risk management to a certain extent. The value, however, depends on the quality of data. The quality needs to be verified before any use of the data. Specifically, measuring the quality by simulating the real life situation and even forecast it accurately turns into a hot topic. In recent years, there have been numerous researches on the measurement and assessment of data quality. These are yet to utilize a scientific computational method for the measurement and prediction. Current methods either fail to make an accurate prediction or do not consider the correlation and time sequence factors of the data. To address this, we design a model to extend machine learning technique to business applications predicting this. Firstly, we implement the model to detect data noises from a risk dataset according to an international data quality standard from banking industry and then estimate their impacts with Gaussian and Bayesian methods. Secondly, we direct sequential learning in multiple deep neural networks for the prediction with an attention mechanism. The model is experimented with various network methodologies to show the predictive power of machine learning technique and is evaluated by validation data to confirm the model effectiveness. The model is scalable to apply to any industries utilizing big data other than the banking industry.