MICCF: A Mutual Information Constrained Clustering Framework for Learning Clustering-Oriented Feature Representations
0
Citation
48
Reference
10
Related Paper
Abstract:
Deep clustering is a crucial task in machine learning and data mining that focuses on acquiring feature representations conducive to clustering. Previous research relies on self-supervised representation learning for general feature representations, such features may not be optimally suited for downstream clustering tasks. In this article, we introduce MICCF, a framework designed to bridge this gap and enhance clustering performance. MICCF enhances feature representations by combining mutual information constraints at different levels and employs an auxiliary alignment mutual information module for learning clustering-oriented features. To be specific, we propose a dual mutual information constraints module, incorporating minimal mutual information constraints at the feature level and maximal mutual information constraints at the instance level. This reduction in feature redundancy encourages the neural network to extract more discriminative features, while maximization ensures more unbiased and robust representations. To obtain clustering-oriented representations, the auxiliary alignment mutual information module utilizes pseudo-labels to maximize mutual information through a multi-classifier network, aligning features with the clustering task. The main network and the auxiliary module work in synergy to jointly optimize feature representations that are well-suited for the clustering task. We validate the effectiveness of our method through extensive experiments on six benchmark datasets. The results indicate that our method performs well in most scenarios, particularly on fine-grained datasets, where our approach effectively distinguishes subtle differences between closely related categories. Notably, our approach achieved a remarkable accuracy of 96.4% on the ImageNet-10 dataset, surpassing other comparison methods. The code is available at https://github.com/Li-Hyn/MICCF.git .Keywords:
Feature (linguistics)
Feature Learning
Cite
Person re-identification (PReID) has received increasing attention due to it is an important part in intelligent surveillance. Recently, many state-of-the-art methods on PReID are part-based deep models. Most of them focus on learning the part feature representation of person body in horizontal direction. However, the feature representation of body in vertical direction is usually ignored. Besides, the spatial information between these part features and the different feature channels is not considered. In this study, we introduce a multi-branches deep model for PReID. Specifically, the model consists of five branches. Among the five branches, two of them learn the local feature with spatial information from horizontal or vertical orientations, respectively. The other one aims to learn interdependencies knowledge between different feature channels generated by the last convolution layer. The remains of two other branches are identification and triplet sub-networks, in which the discriminative global feature and a corresponding measurement can be learned simultaneously. All the five branches can improve the representation learning. We conduct extensive comparative experiments on three PReID benchmarks including CUHK03, Market-1501 and DukeMTMC-reID. The proposed deep framework outperforms many state-of-the-art in most cases.
Feature Learning
Feature (linguistics)
Discriminative model
Representation
Identification
Convolution (computer science)
Cite
Citations (2)
Fine-grained visual classification (FGVC) is challenging task due to discriminative feature representations. The attention-based methods show great potential for FGVC, which neglect that the deeply digging inter-layer feature relations have an impact on refining feature learning. Similarly, the associating cross-layer features methods achieve significant feature enhancement, which lost the long-distance dependencies between elements. However, most of the previous researches neglect that these two methods are mutually correlated to reinforce feature learning, which are independent of each other in related models. Thus, we adopt the respective advantages of the two methods to promote fine-gained feature representations. In this paper, we propose a novel CLNET network, which effectively applies attention mechanism and cross-layer features to obtain feature representations. Specifically, CL-NET consists of 1) adopting self-attention to capture long-rang dependencies for each element, 2) associating cross-layer features to reinforce feature learning,and 3) to cover more feature regions,we integrate attention-based operations between output and input. Experiments verify that CLNET yields new state-of-the-art performance in three widely used fine-grained benchmark datasets, including CUB-200-2011, Stanford Cars and FGVC-Aircraft. The url of our code is https://github.com/dlearing/CLNET.git.
Feature (linguistics)
Feature Learning
Discriminative model
Benchmark (surveying)
Representation
Code (set theory)
Cite
Citations (3)
Image registration
Interpolation
Cite
Citations (3)
Click-Through Rate (CTR) prediction is a core task in nowadays commercial recommender systems. Feature crossing, as the mainline of research on CTR prediction, has shown a promising way to enhance predictive performance. Even though various models are able to learn feature interactions without manual feature engineering, they rarely attempt to individually learn representations for different feature structures. In particular, they mainly focus on the modeling of cross sparse features but neglect to specifically represent cross dense features. Motivated by this, we propose a novel Extreme Cross Network, abbreviated XCrossNet, which aims at learning dense and sparse feature interactions in an explicit manner. XCrossNet as a feature structure-oriented model leads to a more expressive representation and a more precise CTR prediction, which is not only explicit and interpretable, but also time-efficient and easy to implement. Experimental studies on Criteo Kaggle dataset show significant improvement of XCrossNet over state-of-the-art models on both effectiveness and efficiency.
Feature (linguistics)
Feature Engineering
Feature Learning
Click-through rate
Representation
Cite
Citations (0)
Feature learning is an initial step applied to computer vision tasks and is broadly categorized as: 1) deep feature learning; 2) shallow feature learning. In this paper we focus on shallow feature learning as these algorithms require less computational resources than deep feature learning algorithms. In this paper we propose a shallow feature learning algorithm referred to as Extreme Learning Machine Network (ELMNet). ELMNet is module based neural network consist of feature learning module and a post-processing module. Each feature learning module in ELMNet performs the following operations: 1) patch-based mean removal; 2) ELM auto-encoder (ELM-AE) to learn features. Post-processing module is inserted after the feature learning module and simplifies the features learn by the feature learning modules by hashing and block-wise histogram. Proposed ELMNet outperforms shallow feature learning algorithm PCANet on the MNIST handwritten dataset.
Feature (linguistics)
Feature Learning
MNIST database
Extreme Learning Machine
Autoencoder
Instance-based learning
Cite
Citations (13)
Feature Learning aims to extract relevant information contained in data sets in an automated fashion. It is driving force behind the current deep learning trend, a set of methods that have had widespread empirical success. What is lacking is a theoretical understanding of different feature learning schemes. This work provides a theoretical framework for feature learning and then characterizes when features can be learnt in an unsupervised fashion. We also provide means to judge the quality of features via rate-distortion theory and its generalizations.
Feature (linguistics)
Feature Learning
Distortion (music)
Empirical Research
Cite
Citations (0)
The presence of mutual information in the research of deep learning has grown significantly. It has been proven that mutual information can be a good objective function to build a robust deep learning model. Most of the researches utilize estimation methods to approximate the true mutual information. This technical report delivers an extensive study about definitions as well as properties of mutual information. This article then delivers some reviews and current drawbacks of mutual information estimation methods afterward.
Pointwise mutual information
Conditional mutual information
Cite
Citations (0)
Image registration
Interpolation
Cite
Citations (50)
Information Theory
Interaction information
Total correlation
Pointwise mutual information
Conditional mutual information
Feature (linguistics)
Value (mathematics)
Cite
Citations (0)
Feature (linguistics)
Feature Engineering
Feature Learning
Click-through rate
Representation
Cite
Citations (13)