Distributed Training for Speech Recognition using Local Knowledge Aggregation and Knowledge Distillation in Heterogeneous Systems
2
Citation
6
Reference
10
Related Paper
Citation Trend
Abstract:
Data privacy and data protection are crucial issues for automatic speech recognition (ASR) system when relying on client generated data for training. The best protection is achieved when training is distributed fashion, close to the client local data, rather than centralising the training. However, distributed training suffers from system heterogeneity, due to clients having unequal computation resources, and data heterogeneity, due to training data being non-independent and identically distributed (non-IID). To tackle these challenges, we introduce FedKAD, a Federated Learning (FL) framework that uses local Knowledge Aggregation over top level feature maps and Knowledge Distillation. We show that our FedKAD achieves better communication efficiency than standard FL methods that use uniform models, due to transferring parameters of smaller size client models, and overall better accuracy than FedMD, an alternative KD-based approach designed for heterogeneous data. Our work enables faster, cheaper and more inclusive participation of clients in heterogeneous distributed training.Keywords:
Federated Learning
Feature (linguistics)
Training set
Training set
Nothing
Simulation training
Cite
Citations (1)
Fairness and Effectiveness in Federated Learning on Non-independent and Identically Distributed Data
Federated learning is a distributed machine learning method that protects privacy by allowing participants to train models locally rather than uploading data. However, federated learning has a significant barrier because of the non-independent and identically distributed (Non-IID) nature of each participant's local data. FedFE, a novel fair and effective federated optimization algorithm, is presented in this paper. FedFE introduces momentum gradient descent in the federated training process and proposes a fair weighting strategy based on participant performance in training to eliminate the unfairness caused by the preference for some participants in the federated aggregation process. Experiments on a large number of Non-IID datasets have demonstrated that the proposed algorithm improves on existing baseline algorithms in terms of fairness, effectiveness, and convergence speed.
Distributed learning
Federated Learning
Cite
Citations (1)
Throughout this paper we shall be concerned with a sequence of mutually independent and identically distributed random variables ξ 1 ξ 2 , · · ·, ξ n , · · · taking on real values. We shall use the notation ζ n = ξ 1 + · · · + ξ n for n = 1, 2, · · · and ζ 0 = 0.
Sequence (biology)
Pairwise independence
Cite
Citations (3)
Federated learning is a recent advance in privacy protection. In this context, a trusted curator aggregates parameters optimized in decentralized fashion by multiple clients. The resulting model is then distributed back to all clients, ultimately converging to a joint representative model without explicitly having to share the data. However, the protocol is vulnerable to differential attacks, which could originate from any party contributing during federated optimization. In such an attack, a client's contribution during training and information about their data set is revealed through analyzing the distributed model. We tackle this problem and propose an algorithm for client sided differential privacy preserving federated optimization. The aim is to hide clients' contributions during training, balancing the trade-off between privacy loss and model performance. Empirical studies suggest that given a sufficiently large number of participating clients, our proposed procedure can maintain client-level differential privacy at only a minor cost in model performance.
Federated Learning
Differential Privacy
Private information retrieval
Cite
Citations (616)
Sequence (biology)
Cite
Citations (2)
Abstract : The Armored Family of Vehicles (AFV) is a new major acquisition program to build the next generation of armored vehicles. The goals of the program are to build the vehicles with the greatest commonality of parts feasible, for cost reasons, and to take advantage of technology advances as needed to meet the mid-1990s threat. This report presents the methods used, the analyses performed, and the resulting conclusions that formulate a hands-on training concept for the AFV in the institution and in the unit. The training media considered were alternative forms of embedded training and stand-alone training devices. Keywords: Embedded training,
Cite
Citations (1)
Data set containing the ShiftCrypt values of the proteins used for training ConforMine. This set is not to be confused the the Molecular Dynamics (MD) data set, also used for training of this model. The data set also contains a python script which recreates the filtering of sequences performed in the training steps of ConforMine, which discards all proteins for which no valid ShiftCrypt predictions were obtained.
Training set
Data set
Cite
Citations (0)
Generation algorithms of record times and values obtained from sequences of independent and non-identically distributed random variables which distribution functions are defined on a common support are proposed in the present paper. Known algorithms of generation of record times and values are given in introduction for the case when the initial random variables are independent and identically distributed. The brief review of scientific literature associated with this topic is also given in Introduction. It is also pointed out there that all efficient algorithms of record generation are based on the Markov property of records. In Section 2 the distribution functions of record times and values are derived for the case when the initial random variables are independent and non-identically distributed. The corresponding record generation algorithms are for the first time proposed. These algorithms are based on the derived distributions and the Markov property of records that also holds in the case when the initial observations are independent but non-identically distributed. In the end of this work in Section 3 the proposed algorithms are tested by simulation experiments. In these experiments the records are generated for the case when the initial random variables have the Gumbel distribution functions
Gumbel distribution
Cite
Citations (1)
Federated learning is a recent advance in privacy protection. In this context, a trusted curator aggregates parameters optimized in decentralized fashion by multiple clients. The resulting model is then distributed back to all clients, ultimately converging to a joint representative model without explicitly having to share the data. However, the protocol is vulnerable to differential attacks, which could originate from any party contributing during federated optimization. In such an attack, a client's contribution during training and information about their data set is revealed through analyzing the distributed model. We tackle this problem and propose an algorithm for client sided differential privacy preserving federated optimization. The aim is to hide clients' contributions during training, balancing the trade-off between privacy loss and model performance. Empirical studies suggest that given a sufficiently large number of participating clients, our proposed procedure can maintain client-level differential privacy at only a minor cost in model performance.
Federated Learning
Differential Privacy
Private information retrieval
Cite
Citations (451)