Tomáš Souček, Jean-Baptiste Alayrac, Antoine Miech, Ivan Laptev, Josef Sivic, "Look for the Change: Learning Object States and State-Modifying Actions From Untrimmed Web Videos" CVPR2022 https://openaccess.thecvf.com/content/CVPR2022/html/Soucek_Look_for_the_Change_Learning_Object_States_and_State-Modifying_Actions_CVPR_2022_paper.html

1. Look for the Change:
Learning Object States
and State-Modifying Actions
from Untrimmed Web Videos
Tomas Soucek, Jean-Baptiste Alayrac, Antoine Miech,
Ivan Laptev, Josef Sivic, CVPR2022
2023/06/08

2. n
•
•
•
•
•
n
• 3
•
•
•

3. n [Hazel+, CVPR2020]
n
[Antoine+, CVPR2020]
•

4. n
•
n
•
•
n
•
•
•

5. n
• 𝑔 ℎ
•
•

16. nChangelt Dataset
• 667 48
• 2,600 video 44
•
•
•
•
•
•
•
•
•

17. n
• 𝑔 ℎ
• 512 2 MLP
• ReLU
n
• 2 ResNeXT
• ImageNet-21K [Deng+, CVPR2009]
• 3 TSM ResNet50
• HowTo100M AudioSet [Jean+,
CVPR2020]
n
• ChangeIt
• untrimmed video
•
n
• St. prec.
• grand truth
• 0 1 0.5
• Ac. prec.
• grand truth
• 0 1

18. nChangeIt
• untrimmed video
•
• video
n
• 2
•
•

20. n
• 3
•
•
•
n
• 44,000
• 2,600
• 44

22. n
• 𝑥
•
n
• 𝑥
• 2
<latexit sha1_base64="hhQxHmd87xV8RtBlToI+ZPp/gsg=">AAACd3ichVHLSsNAFD2N7/qquhFcGCxKBSk34gtXohuXvqpCLSWJ0zqYJiFJi1r8AX/AhSAoiIqf4cYfcOEniEsFEVx4mwZERb3DzJw5c8+dMzOGa0k/IHqIKQ2NTc0trW3x9o7Oru5ET++675Q9U2RMx3K8TUP3hSVtkQlkYIlN1xN6ybDEhrG7UNvfqAjPl469Fuy7IlfSi7YsSFMPmMoneov5ampv9FDdkraapTFVy+UTSUpTGOpPoEUgiSiWnMQltrANBybKKEHARsDYgg6fWxYaCC5zOVSZ8xjJcF/gEHHWljlLcIbO7C6PRV5lI9bmda2mH6pNPsXi7rFSxTDd0zU90x3d0CO9/1qrGtaoednn2ahrhZvvPupfff1XVeI5wM6n6k/PAQqYCb1K9u6GTO0WZl1fOTh+Xp1dGa6O0Dk9sf8zeqBbvoFdeTEvlsXKCeL8Adr35/4J1sfT2lR6cnkiOTcffUUrBjCEFL/3NOawiCVk+Nw9nOISV7E3ZVAZUVL1VCUWafrwJRTtA/5mj8Q=</latexit>
g(x) 2 [0, 1]
<latexit sha1_base64="59ecp/CmnfkTJhIM37orr0hYQmo=">AAACgnicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIZsRXG2pUaNbqKABZRiCWQkxmnkK0gY6CYWy8gLKBngEYKGAyDKEMZQYoCMgXWM4Qw5DCkM+QzFDKkMuQypDHUAJk5zAkMhQDYTSDIYMBQwFQLJahGihWBGRlguVTGWoZuIB6S4GqUoEqEoGi2UAyHciLhormAfkgM4vBupOBtuQAcRFQpwKDqsFVg5UGnw1OGKw2eGnwB6dZ1WAzQG6pBNJJEL2pBfH8XRLB3wnqygXSJQwZCF143VzCkMZgAXZrJtDtBWARkC+SIfrLqqZ/DrYKUq1WM1hk8Bro/oUGNw0OA32QV/YleWlgatBsBi5gBBiiBzcmI8xIz9BMzzTQRNnBCRoVHAzSDEoMGsDwNmdwYPBgCGAIBdrbzLCSYRPDZiYWJi0mQyZjiFImRqgeYQYUwGQDADN6kms=</latexit>
h1(x), h2(x) 2 [0, 1]

23. n
n
n
n
<latexit sha1_base64="qpgPHaZ2xhtfO6uVX+DJeryn2o4=">AAACanichVHLSsNAFD2Nr1ofrXWjuBGr4qpMxBeuRDcua2ut4KMkcVpD0yQk00AN/oA7V4JdKYiIn+HGH3DRTxDdKbhx4W0aEBX1DjNz5sw9d87MqLahu4KxZkTq6Ozq7on2xvr6BwbjiaHklmvVHI3nNcuwnG1VcbmhmzwvdGHwbdvhSlU1eEGtrLX2Cx53XN0yN0Xd5ntVpWzqJV1TBFGF3L4vHxe9YiLF0iyI8Z9ADkEKYWSsxDV2cQALGmqogsOEIGxAgUttBzIYbOL24BPnENKDfY5jxEhboyxOGQqxFRrLtNoJWZPWrZpuoNboFIO6Q8pxTLEHdsNe2D27ZY/s/ddaflCj5aVOs9rWcrsYPxnJvf2rqtIscPip+tOzQAlLgVedvNsB07qF1tZ7R2cvueXslD/NLtkT+b9gTXZHNzC9V+1qg2cbiNEHyN+f+yfYmk3LC+n5jbnUymr4FVGMYQIz9N6LWME6MsgH7k5xjkbkWUpKo9JYO1WKhJphfAlp8gNP5Ix1</latexit>
S1
v
<latexit sha1_base64="I8/A59F386fbQS/L+/X4ZY9o5aA=">AAACanichVHLSsNAFD2Nr1pftW6UbsRacVWmxReuim5c9mEfULUkcaqhaRKSNFBLf8CdK8GuFETEz3DjD7jwE0R3Fdy48DYNiBb1DjNz5sw9d87MSIaqWDZjTz5hYHBoeMQ/Ghgbn5icCk6H8pZeN2Wek3VVN4uSaHFV0XjOVmyVFw2TizVJ5QWput3dLzjctBRd27UbBt+viUeaUlFk0SaqkD1oJlplpxyMsBhzY74fxD0QgRcpPXiDPRxCh4w6auDQYBNWIcKiVkIcDAZx+2gSZxJS3H2OFgKkrVMWpwyR2CqNR7QqeaxG625Ny1XLdIpK3STlPKLskd2yDntgd+yZffxaq+nW6Hpp0Cz1tNwoT53OZt//VdVotnH8pfrTs40KNlyvCnk3XKZ7C7mnd07OO9nNTLS5xK7YC/m/ZE/snm6gOW/ydZpn2gjQB8R/Pnc/yCdi8bXYanolktzyvsKPMBawTO+9jiR2kELOdXeGC7R9r0JImBPCvVTB52lm8C2ExU9R54x2</latexit>
S2
v
<latexit sha1_base64="IpHdmTYdCeeEqIbP0+oPuMb/5vw=">AAACanichVG7SgNBFD1Z3/GRqI2SRoyKVZgVX1hFbSzjI0bwEXbXUYfsi91NIC75ATsrwVQKIuJn2PgDFvkE0U7BxsK7mwVRUe8wM2fO3HPnzIxq68L1GGvEpJbWtvaOzq54d09vXyLZP7DpWmVH43nN0i1nS1VcrguT5z3h6XzLdrhiqDovqKXlYL9Q4Y4rLHPDq9p811AOTXEgNMUjqrC45+dqxUoxmWYZFsbITyBHII0oclbyGjvYhwUNZRjgMOER1qHApbYNGQw2cbvwiXMIiXCfo4Y4acuUxSlDIbZE4yGttiPWpHVQ0w3VGp2iU3dIOYJx9sBu2Au7Z7fskb3/WssPawReqjSrTS23i4mTofW3f1UGzR6OPlV/evZwgPnQqyDvdsgEt9Ca+srx2cv6wtq4P8Eu2RP5v2ANdkc3MCuv2tUqX6sjTh8gf3/un2BzKiPPZmZWp9PZpegrOpHCKCbpveeQxQpyyIfuTnGOeuxZGpCGpVQzVYpFmkF8CWnsA2nVjII=</latexit>
AP
v
<latexit sha1_base64="lG8ZL+qPfH1mbH7ilmJeI0qZ2Ys=">AAACanichVHLSsNAFD2Nr1pftW6Uboq14qpMxBeufGxcSavWClVLEsc6NE1Ckga0+APuXAl2pSAifoYbf8BFP0F0p+DGhbdpQFTUO8zMmTP33Dkzo1q6cFzGGiGprb2jsyvcHenp7esfiA7GNh2zams8p5m6aW+pisN1YfCcK1ydb1k2VyqqzvNqebm5n/e47QjT2HAPLb5TUUqG2Bea4hKVX9ytrR4XvWI0ydLMj8RPIAcgiSAyZvQa29iDCQ1VVMBhwCWsQ4FDrQAZDBZxO6gRZxMS/j7HMSKkrVIWpwyF2DKNJVoVAtagdbOm46s1OkWnbpMygRR7YDfshd2zW/bI3n+tVfNrNL0c0qy2tNwqDpwMr7/9q6rQ7OLgU/WnZxf7mPO9CvJu+UzzFlpL7x2dvazPr6Vq4+ySPZH/C9Zgd3QDw3vVrrJ8rY4IfYD8/bl/gs3JtDyTns5OJReWgq8II45RTNB7z2IBK8gg57s7xTnqoWcpJo1I8VaqFAo0Q/gS0tgHZc+MgA==</latexit>
AN
v
<latexit sha1_base64="hhQxHmd87xV8RtBlToI+ZPp/gsg=">AAACd3ichVHLSsNAFD2N7/qquhFcGCxKBSk34gtXohuXvqpCLSWJ0zqYJiFJi1r8AX/AhSAoiIqf4cYfcOEniEsFEVx4mwZERb3DzJw5c8+dMzOGa0k/IHqIKQ2NTc0trW3x9o7Oru5ET++675Q9U2RMx3K8TUP3hSVtkQlkYIlN1xN6ybDEhrG7UNvfqAjPl469Fuy7IlfSi7YsSFMPmMoneov5ampv9FDdkraapTFVy+UTSUpTGOpPoEUgiSiWnMQltrANBybKKEHARsDYgg6fWxYaCC5zOVSZ8xjJcF/gEHHWljlLcIbO7C6PRV5lI9bmda2mH6pNPsXi7rFSxTDd0zU90x3d0CO9/1qrGtaoednn2ahrhZvvPupfff1XVeI5wM6n6k/PAQqYCb1K9u6GTO0WZl1fOTh+Xp1dGa6O0Dk9sf8zeqBbvoFdeTEvlsXKCeL8Adr35/4J1sfT2lR6cnkiOTcffUUrBjCEFL/3NOawiCVk+Nw9nOISV7E3ZVAZUVL1VCUWafrwJRTtA/5mj8Q=</latexit>
g(x) 2 [0, 1]
<latexit sha1_base64="59ecp/CmnfkTJhIM37orr0hYQmo=">AAACgnicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIZsRXG2pUaNbqKABZRiCWQkxmnkK0gY6CYWy8gLKBngEYKGAyDKEMZQYoCMgXWM4Qw5DCkM+QzFDKkMuQypDHUAJk5zAkMhQDYTSDIYMBQwFQLJahGihWBGRlguVTGWoZuIB6S4GqUoEqEoGi2UAyHciLhormAfkgM4vBupOBtuQAcRFQpwKDqsFVg5UGnw1OGKw2eGnwB6dZ1WAzQG6pBNJJEL2pBfH8XRLB3wnqygXSJQwZCF143VzCkMZgAXZrJtDtBWARkC+SIfrLqqZ/DrYKUq1WM1hk8Bro/oUGNw0OA32QV/YleWlgatBsBi5gBBiiBzcmI8xIz9BMzzTQRNnBCRoVHAzSDEoMGsDwNmdwYPBgCGAIBdrbzLCSYRPDZiYWJi0mQyZjiFImRqgeYQYUwGQDADN6kms=</latexit>
h1(x), h2(x) 2 [0, 1]