This document outlines a PhD proposal on active communication for distributed learning. The thesis aims to propose active sampling strategies that reduce communication overhead in distributed machine learning setups. The problem is defined as minimizing the number of communications between nodes while achieving similar or better accuracy compared to a single node learning on all data. Several proposed approaches and extensions are outlined, and prior related work by the author on domain adaptation and multi-task learning is summarized.

1. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Active Communication for Distributed Learning
Avishek Saha
School Of Computing, University of Utah
PhD Proposal
4th May, 2011
1 / 29

2. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Outline
1 Introduction
2 Thesis
3 Problem
4 Proposed Approaches
5 Prior Work
6 Timeline
2 / 29

3. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Outline
1 Introduction
2 Thesis
3 Problem
4 Proposed Approaches
5 Prior Work
6 Timeline
3 / 29

4. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Large scale learning
1 Machine learning on large scale data
2 Data size surpasses the limits of a single core
3 Distributed learning on multiple cores
4 / 29

5. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Challenges
1 Parallelization of serial algorithms/new parallel algorithms
2 Scheduling issues in multiple cores
3 Combining results from multiple cores
4 Communication bottlenecks in distributed learning
5 / 29

6. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Challenges
1 Parallelization of serial algorithms/new parallel algorithms
2 Scheduling issues in multiple cores
3 Combining results from multiple cores
§ ¤
4 Communication bottlenecks in distributed learning
¦ ¥
6 / 29

7. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Observations
1 We need to reduce inter-node communication
7 / 29

8. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Observations
1 We need to reduce inter-node communication
2 Each node is learning on a data subset
8 / 29

9. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Observations
1 We need to reduce inter-node communication
2 Each node is learning on a data subset
3 Can we leverage the information already learned by
individual nodes?
9 / 29

10. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Observations
1 We need to reduce inter-node communication
2 Each node is learning on a data subset
3 Can we leverage the information already learned by
individual nodes?
Our Idea
Use Active Learning to exchange only informative data points
10 / 29

11. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Outline
1 Introduction
2 Thesis
3 Problem
4 Proposed Approaches
5 Prior Work
6 Timeline
11 / 29

12. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Thesis
Thesis Statement
This thesis considers a distributed learning setting to learn at
scale and aims to propose active sampling strategies that
reduce communication overhead in such distributed setups.
12 / 29

13. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Outline
1 Introduction
2 Thesis
3 Problem
4 Proposed Approaches
5 Prior Work
6 Timeline
13 / 29

14. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Assumptions
1 Noiseless
2 Two-party Communication
3 Distribution free setting
4 Adversarial Partition
5 {Instance,Label}-based communication
14 / 29

15. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Problem
Setup
A K B
DA = labeled datasets for node A DB = labeled datasets for node B
hAB
hA = hypothesis learned on DA hB = hypothesis learned on DB
C
D = DA ∪ DB = labeled datasets for node C
hC = hypothesis learned on D
Goal
minimize K, such that, er (hAB ) ≤ er (hC ) +
where K = number of communications and is constant (0 < < 1).
15 / 29

16. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Outline
1 Introduction
2 Thesis
3 Problem
4 Proposed Approaches
5 Prior Work
6 Timeline
16 / 29

17. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Proposed Approaches
17 / 29

18. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Extensions
1 Agnostic
2 Privacy
3 Multiparty
4 Alternate forms of communication
18 / 29

19. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
My Focus
1 Agnostic
2 Privacy
3 Multiparty
4 Alternate forms of communication
19 / 29

20. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Outline
1 Introduction
2 Thesis
3 Problem
4 Proposed Approaches
5 Prior Work
6 Timeline
20 / 29

21. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Prior Work
Past Projects
Databases
1 Metric Functional Dependencies (MFD)
2 Sequential Dependencies (SD)
Machine Learning
1 Semi-supervised Domain Adaptation (EA++)
2 Online MultiTask Learning (OMTL)
3 Active Learning for Domain Adaptation (ALDA)
21 / 29

22. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Prior Work
Past Projects
Databases
1 Metric Functional Dependencies (MFD)
2 Sequential Dependencies (SD)
Machine Learning
1 Semi-supervised Domain Adaptation (EA++)
2 Online MultiTask Learning (OMTL)
3 Active Learning for Domain Adaptation (ALDA)
22 / 29

23. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
EA++
EA: φs
EA++: Ls Ls 0
1 EA++ φt Lt 0 Lt
2 OMTL Ls Lt 0 U -U
R3d
3 ALDA φu
U Ls: labeled source data
Lt: labeled target data
Rd U : unlabeled data
1 EA++
2 OMTL
3 ALDA
23 / 29

24. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
OMTL
w Task 1
w1
A
1 EA++ Task
Relationship
w2
Matrix
2 OMTL Data
wK
3 ALDA Omtl Task K
Omtl = Online MultiTask Learning
wi = weight vector of task i, wi ∈ Rd
wT = (wT , wT , . . . , wT ) ∈ RKd
1 2 K
1 EA++ A = Task relationship (interaction) matrix
2 OMTL
3 ALDA
24 / 29

25. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
ALDA
Target
Active Learning
1 EA++ Source
φ: Domain Adaptation
uφ
2 OMTL
3 ALDA Learn classifier
on source
labeled data
wds
Free Oracle Of
wsrc
Hybrid Costly Oracle Learner
1 EA++ Oracle Oe (in target)
2 OMTL
3 ALDA
25 / 29

26. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Connecting the Past and the Future
26 / 29

27. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Outline
1 Introduction
2 Thesis
3 Problem
4 Proposed Approaches
5 Prior Work
6 Timeline
27 / 29

28. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Timeline
Ph.D. TIMELINE
2011 2012
May June July Aug Sep Oct Nov Dec Jan Feb Mar April May
Submit
work on
noiseless
setting
1. Submit work on agnostic setting
2. Look into privacy related issues
1. Identify interesting directions in privacy
2. Work on one/two privacy problems
3. Submit privacy related works
1. Write thesis
2. Interview
3. Defend and graduate!
28 / 29

29. Introduction Thesis Problem Proposed Approaches Prior Work Timeline
Questions?
THANK YOU !!
29 / 29