Home Speakers List Program Day 1 Program Day 2

Day 1 Program (Dec 9, Sat)

9:00am Registration and Opening!
9:30am Asynchronous Federated Learning meets Jackson networksson networks
Eric Moulines (Ecole Polytechnique)
Our research deals with the asynchronous federated learning, where the nodes have different computational speeds. In this dynamic environment, each node works on models with potential delays and contributes to updates on the central server at its own pace. Existing analyzes of such algorithms often focus on rough bounds on average and maximum node delay, often overlooking the fundamental queuing dynamics inherent in the system. In this work, we present a novel non-uniform sampling scheme tailored to FL with Central Server but asynchronous updates. Our approach analysis is based on the underlying closed Jackson network structure inherent in the computational graph. Our carefully designed experiments conclusively demonstrate the efficiency of our method compared to current asynchronous algorithms, especially evident in the context of image classification. Our work sheds light on the importance of queueing dynamics and customized sampling strategies to improve system performance. This research provides a promising avenue for optimizing distributed machine learning in scenarios characterized by unequal computational capacities.
9:50am Statistical Inference, Asymmetry of Information, and Statistical Contract Theory
Michael I. Jordan (UC Berkeley)
Contract theory is the study of economic incentives when parties transact in the presence of private information. We augment classical contract theory to incorporate a role for learning from data, where the overall goal of the adaptive mechanism is to obtain desired statistical behavior. We consider applications of this framework to problems in federated learning, the delegation of data collection, and principal-agent regulatory mechanisms.
10:10am Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity
Samuel Horvath (MBZUAI)
We present a partially personalized formulation of Federated Learning (FL) that strikes a balance between the flexibility of personalization and cooperativeness of global training. In our framework, we split the variables into global parameters, which are shared across all clients, and individual local parameters, which are kept private. We prove that under the right split of parameters, it is possible to find global parameters that allow each client to fit their data perfectly, and refer to the obtained problem as overpersonalized. For instance, the shared global parameters can be used to learn good data representations, whereas the personalized layers are fine-tuned for a specific client. Moreover, we present a simple algorithm for the partially personalized formulation that offers significant benefits to all clients. In particular, it breaks the curse of data heterogeneity in several settings, such as training with local steps, asynchronous training, and Byzantine-robust training.
10:30pm Coffee Break
11:00am (Keynote) Using artificial intelligence to help restore the natural world
David Dao (ETH Zurich)
"Nature has been deteriorating at rates unparalleled in human history and the implications are global. Climate change and biodiversity loss are two bullets in the same gun. Perils we face in parallel, both driven by deforestation and land use change. If global tropical deforestation were a country, its resulting emissions would be larger than the whole of the European Union. Time is running out and we need urgent climate and environmental action. Unfortunately, we cannot value what we cannot measure. And we are failing to capture nature’s full contributions to society. In this talk, we argue that machine learning (ML) can play a significant role in responding to this critical call for action – but only when we develop ML algorithms in co-design with local and Indigenous communities – by empowering digital monitoring, reporting and verification and conservation projects. We will present our work at Gainforest, a global science-based non-profit and currently a Finalist of the $10M XPRIZE Rainforest, and how Gainforest is deploying affordable top-down and bottom-up monitoring solutions on the ground in partnership with governments and conservation partners in the Global South."
11:30am FedPop: A Bayesian Approach for Personalised Federated Learning
Alain Durmus (Ecole Polytechnique)
Personalised federated learning (FL) aims at collaboratively learning a machine learning model tailored for each client. Albeit promising advances have been made in this direction, most of the existing approaches do not allow for uncertainty quantification which is crucial in many applications. In addition, personalisation in the cross-silo and cross-device setting still involves important issues, especially for new clients or those having a small number of observations. This paper aims at filling these gaps. To this end, we propose a novel methodology coined FedPop by recasting personalised FL into the population modeling paradigm where clients’ models involve fixed common population parameters and random effects, aiming at explaining data heterogeneity. To derive convergence guarantees for our scheme, we introduce a new class of federated stochastic optimisation algorithms that relies on Markov chain Monte Carlo methods. Compared to existing personalised FL methods, the proposed methodology has important benefits: it is robust to client drift, practical for inference on new clients, and above all, enables uncertainty quantification under mild computational and memory overheads. We provide nonasymptotic convergence guarantees for the proposed algorithms and illustrate their performances on various personalised federated learning tasks.
11:50am Power Learning and Posthoc Privacy for the Private Embedding Release Problem in Collaborative Learning
Praneeth Vepakomma (MIT and MBZUAI)
"Cloud-based machine learning inference is an emerging paradigm where users query by sending their data through a service provider who runs an ML model on that data and returns the answer. This talk shares a new framework of ""Posthoc Privacy"" that provides formal privacy guarantees for an arbitrarily trained neural network by linking its local Lipschitz constant with its local sensitivity for releasing the learnt representations (embeddings). To guarantee privacy using local sensitivity, we extend the Propose-Test-Release (PTR) framework to make it tractable for neural network queries to enable a private collaborative inference. We then share a ""Power Mechanism"" that guarantees privacy for 'embedding' release and instead enables distributed training that requires only one round of communication. "
12:15pm Lunch
1:30pm Federated Conformal Predictors for Distributed Uncertainty Quantification
Yaodong Yu (UC Berkeley)
Conformal prediction is emerging as a popular paradigm for providing rigorous uncertainty quantification in machine learning since it can be easily applied as a post-processing step to already trained models. In this paper, we extend conformal prediction to the federated learning setting. The main challenge we face is data heterogeneity across the clients — this violates the fundamental tenet of exchangeability required for conformal prediction. We propose a weaker notion of partial exchangeability, better suited to the FL setting, and use it to develop the Federated Conformal Prediction (FCP) framework. We show FCP enjoys rigorous theoretical guarantees and excellent empirical performance on several computer vision and medical imaging datasets. Our results demonstrate a practical approach to incorporating meaningful uncertainty quantification in distributed and heterogeneous environments. We provide code used in our experiments https://github.com/clu5/federated-conformal.
1:50pm Overcoming Data Heterogeneity Challenges in Federated Learning
Xiaoxiao Li (University of British Columbia)
To overcome barriers of data sharing, federated learning (FL) is a trending framework to enable multi-institutional collaboration in machine learning. This presentation will discuss our ongoing progress in designing FL algorithms that embrace the heterogeneity properties of data from theory to practice. First, I will present our work on theoretically understanding FL training convergence and generalization using neural tangent kernel, called FL-NTK. Second, I will present our algorithms for tackling feature heterogeneity, label heterogeneity, and device heterogeneity. Third, I will present our up-to-date progress and promising future directions in FL for heterogenous data analysis.
2:10pm The Art of Learning Together: Robustness, Privacy and Fairness in Collaborative Learning
Karthik Nandakumar (MBZUAI)
Most existing federated learning (FL) frameworks assume a mutualistic relationship between participants, i.e., all the participating clients behave honestly and benefit from one another. However, just like in nature, real-world applications of FL often have to encounter other forms of relationships such as commensalism (some participants benefit while others are not harmed), parasitism (a few participants benefit at the expense of others), and competition (all the participants compete to maximize their individual benefits). This is especially true in the cross-silo setting, which typically requires collaboration between competitors (e.g., financial and healthcare institutions) driven by ulterior business interests. To operate in non-mutualistic scenarios, FL algorithms must satisfy additional properties such as collaboration fairness and Byzantine robustness. In this talk, we first introduce a distance-based outlier suppression mechanism to improve the robustness of FL algorithms against poisoning attacks by malicious clients. Next, we propose a framework called Confidential and Private Decentralized (CaPriDe) learning, which optimally leverages the power of fully homomorphic encryption (FHE) to enable collaborative learning without compromising on the confidentiality and privacy of data. Finally, we highlight the need for new definitions of fairness in a collaborative setting and novel methods for assessing them in a transparent and privacy-preserving manner.
2:30pm Coffee Break
3:00pm How to Collaborate with your Friends and Foes
Sai Praneeth Karimireddy (UC Berkeley)
We study the objective of a participant in FL in two settings. First, in the friendly setting, all participants want to improve their own model quality but do not mind improving each other's as well. In this setting, we show how clustering naturally emerges as the optimal algorithm to simultaneously improve everyone's model quality. In the second setting, the participants could be competing firms - in such a setting they care about doing better than others i.e. the relative quality. Even in this competitive setting, we show defection-proof iterative federated algorithms which is guaranteed to simultaneously improve the utilities of all participants.
3:20pm Data Valuation & Incentives in Collaborative Machine Learning
Bryan Kian Hsiang Low (National University of Singapore)
In this talk, Dr. Bryan Low will present his preliminary research efforts in valuing data and incentivizing multiple parties with data to collaborate in building higher-quality models, such as through the fairness, individual rationality, and privacy incentives.
3:40pm Towards Auction-based Federated Learning
Han Yu (Nanyang Technological University)
Federated Learning (FL) is an emerging area of AI focusing on training machine learning models in a privacy-preserving manner. The success of FL, especially in open collaboration settings, rests on being able to continuously attract high quality data owners to participate. This, at the same time, also opens the FL to adversaries trying to exploit other parties’ sensitive privacy information. It is important to adopt an ecosystem management approach to building trust and controlling risk in FL. In this talk, I will share with you some attempts we made at the Trustworthy Federated Ubiquitous Learning (TrustFUL) Research Lab in the direction of incorporating auction mechanisms into FL to support dynamic formation of collaborative machine learning groups and provide the necessary governance to support the long-term sustainable operation of such an ecosystem.
4pm Panel Discussion