Upcoming SlideShare
×

# pptx - Psuedo Random Generator for Halfspaces

361 views

Published on

0 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

Views
Total views
361
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
3
0
Likes
0
Embeds 0
No embeds

No notes for slide
• Design randomized algorithm, how to generate randomness?The first question we can ask our self is This is pretty hard…The second eaisier
• Draw a dnf
• ### pptx - Psuedo Random Generator for Halfspaces

1. 1. Yi Wu (CMU)<br />Joint work with <br />ParikshitGopalan (MSR SVC) <br />Ryan O’Donnell (CMU)<br />David Zuckerman (UT Austin)<br />Pseudorandom Generators for Halfspaces<br />TexPoint fonts used in EMF. <br />Read the TexPoint manual before you delete this box.: AAAAA<br />
2. 2. Outline<br />Introduction<br />Pseudorandom Generators<br />Halfspaces<br />Pseudorandom Generators for Halfspaces<br />Our Result<br />Proof<br />Conclusion<br />2<br />
3. 3. Deterministic Algorithm<br />Program<br />Input<br />Output<br />The algorithm deterministically outputs the correct result.<br />3<br />
4. 4. Randomized Algorithm<br />Program<br />Input<br />Output<br />Random Bits.<br />The algorithm outputs the correct result with high probability.<br />4<br />
5. 5. Primality testing <br />ST-connectivity<br />Order statistics <br />Searching<br />Polynomial and matrix identity verification<br />Interactive proof systems<br />Faster algorithms for linear programming<br />Rounding linear program solutions to integer<br />Minimum spanning trees <br />shortest paths minimum cuts<br />Counting and enumeration<br />Matrix permanent <br />Counting combinatorial structures<br />Primality testing <br />ST-connectivity<br />Order statistics <br />Searching<br />Polynomial and matrix identity verification<br />Interactive proof systems<br />Faster algorithms for linear programming<br />Rounding linear program solutions to integer<br />Minimum spanning trees <br />shortest paths minimum cuts<br />Counting and enumeration<br />Matrix permanent <br />Counting combinatorial structures<br />Primality testing <br />ST-connectivity<br />Order statistics <br />Searching<br />Polynomial and matrix identity verification<br />Interactive proof systems<br />Faster algorithms for linear programming<br />Rounding linear program solutions to integer<br />Minimum spanning trees <br />shortest paths minimum cuts<br />Counting and enumeration<br />Matrix permanent <br />Counting combinatorial structures<br />Primality testing <br />ST-connectivity<br />Order statistics <br />Searching<br />Polynomial and matrix identity verification<br />Interactive proof systems<br />Faster algorithms for linear programming<br />Rounding linear program solutions to integer<br />Minimum spanning trees <br />shortest paths minimum cuts<br />Counting and enumeration<br />Matrix permanent <br />Counting combinatorial structures<br />Randomized Algorithms<br />5<br />
6. 6. Is Randomness Necessary?<br />Open Problem: <br />Can we simulate every randomized polynomial time algorithm by a deterministic polynomial time algorithm (the “BPP P” cojecture)? <br />Derandomization of randomized algorithms.<br />Primality testing [AKS]<br />ST-connectivity [Reingold]<br />Quadratic residues [?]<br />6<br />
7. 7. How to generate randomness?<br />Question: How togenerate randomness for every randomized algorithm?<br />Simpler Question: How to generate “pseudorandomness” for some class of programs?<br />7<br />
8. 8. Pseudorandom Generator (PRG)<br />Both program Answer Yes/No with almost the same probability<br />Yes /No<br />Yes/ No<br />Input<br />Program <br />Input<br />Program<br />n “pseudorandom” bit <br />PRG <br />Quality of the PRG: number of seed<br />n random bit <br />Seed<br />k<<n random bit <br />8<br />
9. 9. Why study PRGs?<br />Algorithmic Applications<br />When k = log (n), we can derandomize the algorithm in polynomial time.<br />Streaming Algorithm.<br />Complexity Theoretic Implications<br />Lower Bound of Circuit Class.<br />Learning Theory.<br />9<br />
10. 10. PRG for Classes of Program<br />Space Bounded Program [Nis92]<br />Constant-depth circuits [Nis91, Baz07, Bra09] <br />Halfspaces[DGJSV09, MZ09]<br />10<br />
11. 11. Outline<br />Introduction<br />Pseudorandom Generators<br />Halfspaces<br />Pseudorandom Generators for Halfspaces<br />Our Result<br />Proof<br />Conclusion<br />11<br />
12. 12. Halfspaces<br />-<br />-<br />-<br />-<br />-<br />-<br />+<br />+<br />+<br />-<br />-<br />+<br />+<br />+<br />-<br />+<br />+<br />-<br />-<br />+<br />+<br />+<br />+<br />Halfspaces: Boolean functions h:Rn -> {-1,1} of the form <br />h(x) = sgn(w1x1+…+wnxn- θ)<br /> where w1,…, wn,θ R. <br /><ul><li>Well-studied in complexity theory
13. 13. Widely used in Machine Learning: Perceptron, Winnow, boosting, Support Vector Machines, Lasso, Liner Regression.</li></ul>12<br />
14. 14. Product Distribution<br />For halfspace h(x), x is sampled from some product distribution; i.e., each xi is independently sampled from distribution Di .<br /> For example, each Dican be <br />Uniform distribution on {-1,1}<br />Uniform distribution on [-1,1]<br />Gaussian Distribution<br />13<br />
15. 15. Index <br />Introduction<br />Pseudorandom Generators<br />Halfspaces<br />Pseudorandom Generators for Halfspaces<br />Main Result<br />Proof<br />Conclusion<br />14<br />
16. 16. PRG for halfspaces<br />Both program Answer Yes/No with almost the same probability<br />Yes/No<br />Yes/No<br />h(x) = sign(w1x1+…+wnxn-θ)<br />h(x) = sign(w1x1+…+wnxn-θ)<br />Pseudorandom Variable<br /> x1, x2 …xn<br />PRG <br />x1, x2 …xnfrom some product distribution<br />k<<n random bit <br />15<br />
17. 17. Geometric Interpretation, PRG for uniform distribution over [-1,1]2<br />16<br />
18. 18. Geometric Interpretation, PRG for uniform distribution over [-1,1]2<br />Total Number of points = poly(dim)<br />Number of points in the halfspace is proportional to area.<br />17<br />
19. 19. Application to Machine Learning<br />-<br />-<br />-<br />-<br />-<br />-<br />+<br />+<br />+<br />-<br />-<br />+<br />+<br />+<br />-<br />+<br />+<br />-<br />-<br />+<br />+<br />+<br />+<br />How many testing points is it enough to estimate the accuracy of the N dimensional linear classifier?<br />Good PRG implies we only need deterministically check the accuracy on a set of poly(N) points!<br />18<br />
20. 20. Other Theoretical Applications<br />Discrepancy Set for Convex Polytopes<br />Circuit Lower bound on functions of halfspaces<br />Counting the Solution of Knapsacks<br />19<br />
21. 21. Outline <br />Introduction<br />Pseudorandom Generator<br />Halfspace<br />Pseudorandom Generators for Halfspaces<br />Our Results<br />Proof<br />Conclusion<br />20<br />
22. 22. Previous Result<br /> [DiGoJaSeVi,MeZu] PRG For Halfspace over uniform distribution on boolean cube ({-1,1}n) with seed length O(log n).<br />21<br />
23. 23. Our Results:Arbitrary Product Distributions<br />PRG for halfspaces under arbitrary product distribution over Rnwith the same seed length.<br />Only requirement: E[xi4] is a constant.<br />Gaussian Distribution<br />Uniform distribution on the solid cube.<br />Uniform distribution on the hypercube.<br />Biased distribution on the hypercube.<br />Almost any “natural distribution”<br />22<br />
24. 24. Our Results Functions of k-Halfspaces<br />PRG for the intersections of k-halfspaces with seed length k log (n). <br />PRG for arbitrary functions of k-halfspaces with seed length k2 log (n). <br />23<br />
25. 25. Outline<br />Introduction<br />Pseudorandom Generator<br />Halfspace<br />Pseudorandom Generators for Halfspaces<br />Our Result<br />Proof<br />Conclusion<br />24<br />
26. 26. Key Observation: Dichotomy of Halfspaces<br />Under product distributions , every halfspace is close to one of the following: <br />“Dictator” (halfspaces depending on very few variables, e.g. f(x) = sgn(x1)) <br />“majority”(no variables has too much weight, e.g. f(x) = sgn(x1+x2+x3+…+xn).<br />25<br />
27. 27. Dichotomy of weight distribution<br />Weights decreasing fast (Geometrically)<br />Weights are stable after certain index.<br />26<br />
28. 28. Weights Decrease fast (Geometrically)<br />Intuition: for sign(2n x1 + 2n-1 x2+ 2n-2 x3 +…xn)<br />If each xi is from {-1,1} , it is just sign(x1). <br />27<br />
29. 29. Weights are stable<br />Intuition: for sign(100 x1 + x2 + x3+…xn) <br />Then by for every fixing of x1, it is a majority on the rest of the variables.<br />28<br />
30. 30. Our PRG for Halfspace (Rough)<br />Randomly hashing all the coordinate into groups. <br />Use 4-wise independent distribution within each group. <br />If it is “Dictator-like”: All the important variables are in different groups.<br />If it is “Majority-like”<br />(x1 + x2 +.. xn ) is close to Gaussian. 4-wise independent Distribution (somehow) can handle Gaussian.<br />29<br />
31. 31. Outline<br />Introduction<br />Pseudorandom Generator<br />Halfspace<br />Pseudorandom Generators for Halfspaces<br />Our Result<br />Proof<br />Conclusion<br />30<br />
32. 32. Conclusion<br />We construct PRG for halfspaces under arbitrary product distribution and functions of k halfspaces with small seed length.<br />Future Work<br />Building PRG for larger classes of program; e.g., Polynomial Threshold function (SVM with polynomial kernel).<br />31<br />
33. 33. 32<br />