pptx - Psuedo Random Generator for Halfspaces
Upcoming SlideShare
Loading in...5
×
 

pptx - Psuedo Random Generator for Halfspaces

on

  • 312 views

 

Statistics

Views

Total Views
312
Views on SlideShare
312
Embed Views
0

Actions

Likes
0
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Design randomized algorithm, how to generate randomness?The first question we can ask our self is This is pretty hard…The second eaisier
  • Draw a dnf

pptx - Psuedo Random Generator for Halfspaces pptx - Psuedo Random Generator for Halfspaces Presentation Transcript

  • Yi Wu (CMU)
    Joint work with
    ParikshitGopalan (MSR SVC)
    Ryan O’Donnell (CMU)
    David Zuckerman (UT Austin)
    Pseudorandom Generators for Halfspaces
    TexPoint fonts used in EMF.
    Read the TexPoint manual before you delete this box.: AAAAA
  • Outline
    Introduction
    Pseudorandom Generators
    Halfspaces
    Pseudorandom Generators for Halfspaces
    Our Result
    Proof
    Conclusion
    2
  • Deterministic Algorithm
    Program
    Input
    Output
    The algorithm deterministically outputs the correct result.
    3
  • Randomized Algorithm
    Program
    Input
    Output
    Random Bits.
    The algorithm outputs the correct result with high probability.
    4
  • Primality testing
    ST-connectivity
    Order statistics
    Searching
    Polynomial and matrix identity verification
    Interactive proof systems
    Faster algorithms for linear programming
    Rounding linear program solutions to integer
    Minimum spanning trees
    shortest paths minimum cuts
    Counting and enumeration
    Matrix permanent
    Counting combinatorial structures
    Primality testing
    ST-connectivity
    Order statistics
    Searching
    Polynomial and matrix identity verification
    Interactive proof systems
    Faster algorithms for linear programming
    Rounding linear program solutions to integer
    Minimum spanning trees
    shortest paths minimum cuts
    Counting and enumeration
    Matrix permanent
    Counting combinatorial structures
    Primality testing
    ST-connectivity
    Order statistics
    Searching
    Polynomial and matrix identity verification
    Interactive proof systems
    Faster algorithms for linear programming
    Rounding linear program solutions to integer
    Minimum spanning trees
    shortest paths minimum cuts
    Counting and enumeration
    Matrix permanent
    Counting combinatorial structures
    Primality testing
    ST-connectivity
    Order statistics
    Searching
    Polynomial and matrix identity verification
    Interactive proof systems
    Faster algorithms for linear programming
    Rounding linear program solutions to integer
    Minimum spanning trees
    shortest paths minimum cuts
    Counting and enumeration
    Matrix permanent
    Counting combinatorial structures
    Randomized Algorithms
    5
  • Is Randomness Necessary?
    Open Problem:
    Can we simulate every randomized polynomial time algorithm by a deterministic polynomial time algorithm (the “BPP P” cojecture)?
    Derandomization of randomized algorithms.
    Primality testing [AKS]
    ST-connectivity [Reingold]
    Quadratic residues [?]
    6
  • How to generate randomness?
    Question: How togenerate randomness for every randomized algorithm?
    Simpler Question: How to generate “pseudorandomness” for some class of programs?
    7
  • Pseudorandom Generator (PRG)
    Both program Answer Yes/No with almost the same probability
    Yes /No
    Yes/ No
    Input
    Program
    Input
    Program
    n “pseudorandom” bit
    PRG
    Quality of the PRG: number of seed
    n random bit
    Seed
    k<<n random bit
    8
  • Why study PRGs?
    Algorithmic Applications
    When k = log (n), we can derandomize the algorithm in polynomial time.
    Streaming Algorithm.
    Complexity Theoretic Implications
    Lower Bound of Circuit Class.
    Learning Theory.
    9
  • PRG for Classes of Program
    Space Bounded Program [Nis92]
    Constant-depth circuits [Nis91, Baz07, Bra09]
    Halfspaces[DGJSV09, MZ09]
    10
  • Outline
    Introduction
    Pseudorandom Generators
    Halfspaces
    Pseudorandom Generators for Halfspaces
    Our Result
    Proof
    Conclusion
    11
  • Halfspaces
    -
    -
    -
    -
    -
    -
    +
    +
    +
    -
    -
    +
    +
    +
    -
    +
    +
    -
    -
    +
    +
    +
    +
    Halfspaces: Boolean functions h:Rn -> {-1,1} of the form
    h(x) = sgn(w1x1+…+wnxn- θ)
    where w1,…, wn,θ R.
    • Well-studied in complexity theory
    • Widely used in Machine Learning: Perceptron, Winnow, boosting, Support Vector Machines, Lasso, Liner Regression.
    12
  • Product Distribution
    For halfspace h(x), x is sampled from some product distribution; i.e., each xi is independently sampled from distribution Di .
    For example, each Dican be
    Uniform distribution on {-1,1}
    Uniform distribution on [-1,1]
    Gaussian Distribution
    13
  • Index
    Introduction
    Pseudorandom Generators
    Halfspaces
    Pseudorandom Generators for Halfspaces
    Main Result
    Proof
    Conclusion
    14
  • PRG for halfspaces
    Both program Answer Yes/No with almost the same probability
    Yes/No
    Yes/No
    h(x) = sign(w1x1+…+wnxn-θ)
    h(x) = sign(w1x1+…+wnxn-θ)
    Pseudorandom Variable
    x1, x2 …xn
    PRG
    x1, x2 …xnfrom some product distribution
    k<<n random bit
    15
  • Geometric Interpretation, PRG for uniform distribution over [-1,1]2
    16
  • Geometric Interpretation, PRG for uniform distribution over [-1,1]2
    Total Number of points = poly(dim)
    Number of points in the halfspace is proportional to area.
    17
  • Application to Machine Learning
    -
    -
    -
    -
    -
    -
    +
    +
    +
    -
    -
    +
    +
    +
    -
    +
    +
    -
    -
    +
    +
    +
    +
    How many testing points is it enough to estimate the accuracy of the N dimensional linear classifier?
    Good PRG implies we only need deterministically check the accuracy on a set of poly(N) points!
    18
  • Other Theoretical Applications
    Discrepancy Set for Convex Polytopes
    Circuit Lower bound on functions of halfspaces
    Counting the Solution of Knapsacks
    19
  • Outline
    Introduction
    Pseudorandom Generator
    Halfspace
    Pseudorandom Generators for Halfspaces
    Our Results
    Proof
    Conclusion
    20
  • Previous Result
    [DiGoJaSeVi,MeZu] PRG For Halfspace over uniform distribution on boolean cube ({-1,1}n) with seed length O(log n).
    21
  • Our Results:Arbitrary Product Distributions
    PRG for halfspaces under arbitrary product distribution over Rnwith the same seed length.
    Only requirement: E[xi4] is a constant.
    Gaussian Distribution
    Uniform distribution on the solid cube.
    Uniform distribution on the hypercube.
    Biased distribution on the hypercube.
    Almost any “natural distribution”
    22
  • Our Results Functions of k-Halfspaces
    PRG for the intersections of k-halfspaces with seed length k log (n).
    PRG for arbitrary functions of k-halfspaces with seed length k2 log (n).
    23
  • Outline
    Introduction
    Pseudorandom Generator
    Halfspace
    Pseudorandom Generators for Halfspaces
    Our Result
    Proof
    Conclusion
    24
  • Key Observation: Dichotomy of Halfspaces
    Under product distributions , every halfspace is close to one of the following:
    “Dictator” (halfspaces depending on very few variables, e.g. f(x) = sgn(x1))
    “majority”(no variables has too much weight, e.g. f(x) = sgn(x1+x2+x3+…+xn).
    25
  • Dichotomy of weight distribution
    Weights decreasing fast (Geometrically)
    Weights are stable after certain index.
    26
  • Weights Decrease fast (Geometrically)
    Intuition: for sign(2n x1 + 2n-1 x2+ 2n-2 x3 +…xn)
    If each xi is from {-1,1} , it is just sign(x1).
    27
  • Weights are stable
    Intuition: for sign(100 x1 + x2 + x3+…xn)
    Then by for every fixing of x1, it is a majority on the rest of the variables.
    28
  • Our PRG for Halfspace (Rough)
    Randomly hashing all the coordinate into groups.
    Use 4-wise independent distribution within each group.
    If it is “Dictator-like”: All the important variables are in different groups.
    If it is “Majority-like”
    (x1 + x2 +.. xn ) is close to Gaussian. 4-wise independent Distribution (somehow) can handle Gaussian.
    29
  • Outline
    Introduction
    Pseudorandom Generator
    Halfspace
    Pseudorandom Generators for Halfspaces
    Our Result
    Proof
    Conclusion
    30
  • Conclusion
    We construct PRG for halfspaces under arbitrary product distribution and functions of k halfspaces with small seed length.
    Future Work
    Building PRG for larger classes of program; e.g., Polynomial Threshold function (SVM with polynomial kernel).
    31
  • 32