4.1   4.2
                       .



                   .


2010/10/12




             4.1   4.2
AGENDA




    1

    2
.
    3



.
    4

.




.                4.1   4.2
AGENDA




    1

    2
.
    3



.
    4

.




.                4.1   4.2
4
1

2




                .

                .

    4.1   4.2
AGENDA




    1

    2
.
    3



.
    4

.




.                4.1   4.2
(classification categorization)




                             4.1   4.2
2




SVM




          4.1   4.2
AGENDA




    1

    2
.
    3



.
    4
.




.                4.1   4.2
d               P(c|d)                        c∈C
            P(c|d)
        1


                                         ...
P(d|c)

    d
            d
          d
        P(d|c)




                     d
-
-


                         4.1   4.2
AGENDA




    1

    2
.
    3



.
    4
.




.                4.1   4.2
-

     P(d|c)


d
                         ∏      δω,d
              P(d|c) =         pω,c (1 − pω,c )1−δω,d
            ...
-




                        ∏      δω,d
      P(c)P(d|c) = pc         pω,c (1 − pω,c )1−δω,d
                        ω∈V...
D

                 D          = {(d(1) , c(1) ), (d(2) , c(2) ), ..., (d|D| , c|D| )}



                  ∑
log P(D)    ...
pc



               max .     log P(D)
                         ∑
                 s.t.        pc = 1.
                  ...
∂L(θ, λ)
           = 0
 ∂ pω,c
∂L(θ, λ)
           = 0
  ∂ pc
∂L(θ, λ)
           = 0
  ∂λ




                 4.1   4.2

∂L(θ, λ)         ∂      ∑
                        
                        
                                        ∑...
pω,c


            Nω,c       (N c − Nω,c )
                   −                 = 0
              pω,c      1 − pω,c
(1 −...
pc

     Nc
        +λ =          0
     pc
                          Nc
            pc    =   −
                         ...
c       ω
pω,c =
             c
         c
 pc =




                     4.1   4.2
4.1

P                      3


    d(1)       =           ”good bad good good”
    d(2)       =           ”exciting excit...
4.1


                       V    =    {bad, boring, exciting, good}



N P = 3,     N N = 3,     N bad,P = 1,   N bad,N =...
4.2

4.1                                                      d


                  d = ”good good bad boring”

      pP p...
4.2

4.1                                                      d


                  d = ”good good bad boring”

      pP p...
4.3




4.1       d(1)

      d(1) = ”good bad good good fine”


                                d


      d = ”bad bad bor...
4.3


                “fine”                      fine


              N f ine,P                                   N f ine,N...
4.3


   d               “bad”     ”boring”                          ”good”
”exciting”                                    ...
MAP

                                        0.00



MAP

                          ∏        ∏           
            ...
MAP




                                   ∑       
                                   
                               ...
MAP



    ∑
0       c   pc = 1


                     Nω,c + (α − 1)
    pω,c =
                          Nc + 2
        ...
4.4

4.3
                              MAP
                             α=1
      P                  3


          d(1)   ...
4.4



Table:
                        MAP                                         MAP
       pP        0.50     0.50      ...
AGENDA




    1

    2
.
    3



.
    4
.




.                4.1   4.2
V            1                      |d|

P(d|c)
d             ω              nω,d

                    ∑       (∑ n )! ∏...
c

                            ∑        (∑ n )! ∏
                            
                                    
 ...
2




    4.1   4.2
∑
log P( D) =              log P(d, c)
              (d,c)∈ D
                                                   
      ...
                       
                          ∑        ∑
                                             
        ...

∂L(θ, β, γ)         ∂   ∑
                       
                                     P(|d|)|d|!    ∑             ∑∑...
βc
       ∑
             qω,c = 1
       ω∈V
     1 ∑
             nω,c = 1
     β c ω∈V
                               1
...
c            ω
qω,c =
         c



         c           ω
pω,c =
                 c




                         4.1   4.2
MAP




                                                               0.00
                                              ...
MAP



L(θ, β, γ) = log P(θ) + log P(D)
                                               
               ∑ ∑          ...
AGENDA




    1

    2
.
    3



.
    4
.




.                4.1   4.2
d


MAP




          4.1   4.2
(               )




Ml for nlp chapter 4




                           4.1   4.2
4.1   4.2
Upcoming SlideShare
Loading in …5
×

Ml4nlp04 1

2,384 views

Published on

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,384
On SlideShare
0
From Embeds
0
Number of Embeds
1,180
Actions
Shares
0
Downloads
13
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Ml4nlp04 1

  1. 1. 4.1 4.2 . . 2010/10/12 4.1 4.2
  2. 2. AGENDA 1 2 . 3 . 4 . . 4.1 4.2
  3. 3. AGENDA 1 2 . 3 . 4 . . 4.1 4.2
  4. 4. 4 1 2 . . 4.1 4.2
  5. 5. AGENDA 1 2 . 3 . 4 . . 4.1 4.2
  6. 6. (classification categorization) 4.1 4.2
  7. 7. 2 SVM 4.1 4.2
  8. 8. AGENDA 1 2 . 3 . 4 . . 4.1 4.2
  9. 9. d P(c|d) c∈C P(c|d) 1 P(c)P(d|c) P(c|d) = P(d) . 2 P(d) P(c)P(d|c) c max . P(c)P(d|c) c max = arg max c P(d) = arg max P(c)P(d|c) c . 4.1 4.2
  10. 10. P(d|c) d d d P(d|c) d - - 4.1 4.2
  11. 11. AGENDA 1 2 . 3 . 4 . . 4.1 4.2
  12. 12. - P(d|c) d ∏ δω,d P(d|c) = pω,c (1 − pω,c )1−δω,d ω∈V V ω pc P(c) pω,c pc 2 δω,d pω,c (1 − pω,c )1−δω,d c ω d 4.1 4.2
  13. 13. - ∏ δω,d P(c)P(d|c) = pc pω,c (1 − pω,c )1−δω,d ω∈V c pω,c P(d|c) P(d|c) pω,c 1 4.1 4.2
  14. 14. D D = {(d(1) , c(1) ), (d(2) , c(2) ), ..., (d|D| , c|D| )} ∑ log P(D) = log P(d, c) (d,c)∈D   ∑  ∏ δω,d      = log  pc    pω,c (1 − pω,c )1−δω,d     (d,c)∈D ω∈V   ∑   ∑    log pc +  =    (δω,d log pω,c + (1 − ωω,d ) log(1 − pω,c ))    (d,c)∈D ω∈V ∑ ∑∑ ∑∑ = N c log pc + Nω,c log pω,c + (N c − Nω,c ) log(1 − pω,c ) c c ω∈V c ω∈V Nc : c Nω,c : c ω 4.1 4.2
  15. 15. pc max . log P(D) ∑ s.t. pc = 1. c L(θ, λ) ∑        L(θ, λ) = log P(D) + λ   pc − 1    c θ: { pomega,c }ωinV,c∈C , {pc } c∈C 4.1 4.2
  16. 16. ∂L(θ, λ) = 0 ∂ pω,c ∂L(θ, λ) = 0 ∂ pc ∂L(θ, λ) = 0 ∂λ 4.1 4.2
  17. 17.  ∂L(θ, λ) ∂ ∑   ∑∑ =    N c log pc + Nω,c log pω,c ∂ pω,c ∂ pω,c  c c ω∈V ∑∑ ∑      + (N c − Nω,c ) log(1 − pω,c ) + λ      pc − 1    c ω∈V c ∂(1−pω,c ) Nω,c ∂pω,c = + (N c − Nω,c ) pω,c (1 − pω,c ) Nω,c (N c − Nω,c ) = − pω,c 1 − pω,c  ∂L(θ, λ) ∂ ∑  ∑∑ =    N c log pc + Nω,c log pω,c ∂pc  ∂ pc c c ω∈V ∑∑ ∑      +   (N c − Nω,c ) log(1 − pω,c ) + λ    pc − 1    c ω∈V c Nc = +λ pc 4.1 4.2
  18. 18. pω,c Nω,c (N c − Nω,c ) − = 0 pω,c 1 − pω,c (1 − pω,c )Nω,c − pω,c (N c − Nω,c ) = 0 pω,c (N c − Nω,c + Nω,c ) = Nω,c Nω,c pω,c = Nc 4.1 4.2
  19. 19. pc Nc +λ = 0 pc Nc pc = − λ ∑ pc = 1 c 1∑ − Nc = 1 λ c ∑ λ = − Nc c Nc Nc pc = − = ∑ λ c Nc 4.1 4.2
  20. 20. c ω pω,c = c c pc = 4.1 4.2
  21. 21. 4.1 P 3 d(1) = ”good bad good good” d(2) = ”exciting exciting” d(3) = ”good good exciting boring” N 3 d(4) = ”bad boring boring boring” d(5) = ”bad good bad” d(6) = ”bad bad boring exciting” P N 4.1 4.2
  22. 22. 4.1 V = {bad, boring, exciting, good} N P = 3, N N = 3, N bad,P = 1, N bad,N = 3, N boring,P = 1, N boring,N = 2, Nexciting,P = 2, Nexciting,N = 1, N good,P = 2, N good,N = 1, NP NN pP = N P +N N = 3+3 = 0.50 3 pN = N p+NN = 3+3 = 0.50 3 N bad,P N bad,N pbad,P = N P = 1 = 0.33 3 pbad,N = NN = 3 = 1.00 3 N boring,P N bof ing,N pboring,P = N P = 3 = 0.33 1 pbof ing,N = NN = 2 = 3 0.67 Nexciting,P pexciting,P = N P = 2 = 0.67 3 Nexciting,N 1 pexciting,N = = 3 = 0.33 N good,P N good,N pgood,P = N P = 2 = 0.67 3 pgood,N = NN = 1 = 0.33 3 4.1 4.2
  23. 23. 4.2 4.1 d d = ”good good bad boring” pP pd|P pN pd|N pP pd|P = pP × pbad,P × pboring,P × (1 − pexciting,P ) × pgood,P = 0.5 × 0.33 × 0.33 × (1 − 0.67) × 0.67 = 0.012 pN pd|N = pN × pbad,N × pboring,N × (1 − pexciting,N ) × pgood,N = 0.5 × 1.00 × 0.67times(1 − 0.33) × 0.33 = 0.074 4.1 d N 4.1 4.2
  24. 24. 4.2 4.1 d d = ”good good bad boring” pP pd|P pN pd|N pP pd|P = pP × pbad,P × pboring,P × (1 − pexciting,P ) × pgood,P = 0.5 × 0.33 × 0.33 × (1 − 0.67) × 0.67 = 0.012 pN pd|N = pN × pbad,N × pboring,N × (1 − pexciting,N ) × pgood,N = 0.5 × 1.00 × 0.67times(1 − 0.33) × 0.33 = 0.074 4.1 d N 4.1 4.2
  25. 25. 4.3 4.1 d(1) d(1) = ”good bad good good fine” d d = ”bad bad boring boring fine” 4.1 4.2
  26. 26. 4.3 “fine” fine N f ine,P N f ine,N p f ine,P = NP = 1 3 = 0.33 p f ine,N = NN = 0 3 = 0.00 pP pd|P = pP × pbad,P × pboring,P × (1 − pexciting,P ) × p f ine,P × (1 − pgood,P ) = 0.5 × 0.33 × 0.33 × (1 − 0.67) × 0.33 × (1 − 0.67) = 0.002 pN pd|N = pN × pbad,N × pboring,N × (1 − pexciting,N ) × p f ine,N × (1 − pgood,N ) = 0.5 × 1.00 × 0.67 × (1 − 0.33) × 0.00 × 0.67 = 0.00 P 4.1 4.2
  27. 27. 4.3 d “bad” ”boring” ”good” ”exciting” P p f ine,N = 0.00 N pN pd|N = 0.00 0 MAP 4.1 4.2
  28. 28. MAP 0.00 MAP ∏  ∏  ∑      × α−1    α−1    log P(θ) + log P(D) ∝ log  pc      pω,c  +       log P(d, c) c ω,c (d,c)∈ D ∑ ∑ = (α − 1) log pc + (α − 1) log pω,c c ω,c   ∑  ∏ δ     1−δω,d   + log  pc   ω,d ( pω,c (1 − pω,c ) )    (d,c)∈ D ω∈V ∑ c p(c) = 1 4.1 4.2
  29. 29. MAP ∑        L(θ, λ) = log P(θ) + log P( D) + λ   pc − 1    c ∂L(θ, λ) (α − 1) Nω,c N c − Nω,c = + − ∂ pω,c pω,c pω,c 1 − pω,c ∂L(θ, λ) (α − 1) N c = + +λ ∂ pc pc pc 4.1 4.2
  30. 30. MAP ∑ 0 c pc = 1 Nω,c + (α − 1) pω,c = Nc + 2 Nc + 1 pc = ∑ c N c + |C| α 4.1 4.2
  31. 31. 4.4 4.3 MAP α=1 P 3 d(1) = ”good bad good good fine” d(2) = ”exciting exciting” d(3) = ”good good exciting boring” N 3 d(4) = ”bad boring boring boring” d(5) = ”bad good bad” d(6) = ”bad bad boring exciting” 4.1 4.2
  32. 32. 4.4 Table: MAP MAP pP 0.50 0.50 pN 0.50 0.50 pbad,P 0.33 0.40 pbad,N 1.00 0.80 pboring,P 0.33 0.40 pboring,N 0.67 0.60 pexciting,P 0.67 0.60 pexciting,N 0.33 0.40 p f ine,P 0.33 0.40 p f ine,N 0.00 0.20 pgood,P 0.67 0.60 pgood,N 0.33 0.40 MAP smoothing MAP 4.1 4.2
  33. 33. AGENDA 1 2 . 3 . 4 . . 4.1 4.2
  34. 34. V 1 |d| P(d|c) d ω nω,d  ∑  (∑ n )! ∏     ω ω,d P(d|c) = P  K =  nω,d  ∏  nω,d     qω,c ω ω∈V nω,d ! ω∈V K: ( ∑ ) ∑ P K = ω nω,d : ω nω,d 4.1 4.2
  35. 35. c ∑  (∑ n )! ∏     ω ω,d pc P   nω,d  ∏  nω,d P(c)P(d|c) =     qω,c ω ω∈V nω,d ! ω∈V ∑  (∑ n )! ∏     ω ω,d arg max P(c)P(d|c) = arg max pc P   nω,d  ∏  n     q ω,d c c ω ω∈V nω,d ! ω∈V ω,c ∏ nω = arg max pc qω,c c ω∈V ∏ nω c pc ω∈V qω,c 4.1 4.2
  36. 36. 2 4.1 4.2
  37. 37. ∑ log P( D) = log P(d, c) (d,c)∈ D   ∑  p(|d|)|d|!  ∏ n     ω,d  = log  ∏   pc qω,c    (d,c)∈ D ω∈Vn ! ω,d ω∈V ∑ P(|d|)|d|! ∑ ∑ ∑ = log ∏ + log pc + nω,d log qω,c (d,c)∈ D ω∈V nω,d ! (d,c)∈ D (d,c)∈ D ω∈V ∑ P(|d|)|d|! ∑ ∑∑ = log ∏ + log nc pc + nω,c log qω,c (d,c)∈ D ω∈V nω,d ! c c ω∈V max. log P(D) ∑ s.t. pc = 1. c∈C ∑ qω,c = 1; ∀c ∈ C ω∈V 4.1 4.2
  38. 38.     ∑ ∑    ∑         L(θ, β, γ) = log P(D) + βc    qω,c − 1 + γ        pc − 1    c∈C ω∈V c∈C ∂L(θ, β, γ) = 0 ∂qω,c ∂L(θ, β, γ) = 0 ∂ pc ∂L(θ, β, γ) = 0 ∂β ∂L(θ, β, γ) = 0 ∂γ 4.1 4.2
  39. 39.  ∂L(θ, β, γ) ∂  ∑   P(|d|)|d|! ∑ ∑∑ =    log ∏ + nc log pc +   nω,c log qω,c ∂qω,c ∂qω,c (d,c)∈D ω∈V nω,d ! c c ω∈V  ∑ ∑ ∑     βc ( −1) + γ( pc − 1)  c∈C ω∈V c∈C nω,c = + βc = 0 qω,c nω,c qω,c = βc 4.1 4.2
  40. 40. βc ∑ qω,c = 1 ω∈V 1 ∑ nω,c = 1 β c ω∈V 1 βc = ∑ ω∈V nω,c nω,c qω,c = ∑ ω nω,c pc 4.1 4.2
  41. 41. c ω qω,c = c c ω pω,c = c 4.1 4.2
  42. 42. MAP 0.00 MAP MAP ∏  ∏  ∑        α−1  log P(θ) + log P(D) ∝ log       pα−1  ×      qω,c  +   c    log P(d, c) c ω,c (d,c)∈D     ∑   ∑    ∑  P(|d|)|d|!   ∏ n  ω,d   =  (α − 1)   log pc + log qω,c  +   log  ∏   pc qω,c      n !  c ω,c (d,c)∈D ω∈V ω,d ω∈V ∑ ∑ c p(c) = 1 ω qω,c = 1 4.1 4.2
  43. 43. MAP L(θ, β, γ) = log P(θ) + log P(D)     ∑ ∑    ∑        + βc     pω,c − 1 + γ        pc − 1    c∈C ω∈V c∈C ∂L(θ, β, γ) (α − 1) nω,c = + + βc ∂qω,c qω,c qω,c ∑ 0 ω∈V qω,c = 1 nω,c + (α − 1) qω,c = ∑ ω nω,c + |W|(α − 1) 4.1 4.2
  44. 44. AGENDA 1 2 . 3 . 4 . . 4.1 4.2
  45. 45. d MAP 4.1 4.2
  46. 46. ( ) Ml for nlp chapter 4 4.1 4.2
  47. 47. 4.1 4.2

×