An Effective Integrated Method for Learning Big Imbalanced Data
2014
The imbalance of data has great effects on the performance of learning algorithms due to the presence of under-represented data and severe class distribution skews. This is one of the new challenges of machine learning data mining. Choosing a suitable metric that addresses the properties and domain characteristics of learning real-world data is critical for achieving a good result in most of machine learning and data mining algorithms. When the dataset is big and imbalanced, even with an accurate metric, it is extremely difficult to achieve good learning performance. This paper proposes an integrated method for learning large imbalanced datasets. In particular, a combination of metric learning algorithms and balancing techniques are experimented. Their performances are compared based on a set of evaluation metrics running on bootstrap datasets of different sizes. The best combination is then selected for learning the full imbalanced datasets. Experiments using the water pipeline datasets collected from various Australia regions in the past two decades show that our proposed method is both practical and effective.
Keywords:
- Data mining
- Active learning (machine learning)
- Online machine learning
- Computer science
- Generalization error
- Semi-supervised learning
- Instance-based learning
- Unsupervised learning
- Computational learning theory
- Machine learning
- Stability (learning theory)
- Artificial intelligence
- Pattern recognition
- Bootstrapping (electronics)
- Correction
- Source
- Cite
- Save
- Machine Reading By IdeaReader
24
References
11
Citations
NaN
KQI