Recommended
Recommended
More Related Content
Similar to Introduction to Matsuo Laboratory (ENG).pptx
Similar to Introduction to Matsuo Laboratory (ENG).pptx (20)
Recently uploaded
Recently uploaded (20)
Introduction to Matsuo Laboratory (ENG).pptx
- 1. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO Introduction to Matsuo Lab March 2024 Photography, video recording and disclosure to third parties without permissions are strictly prohibited.
- 2. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 2 Matsuo Lab startups And more Matsuo Lab, Graduate School of Engineering, The University of Tokyo • Matsuo Lab belongs to the Graduate School of Engineering at The University of Tokyo and specializes in Artificial Intelligence (AI) and Web Engineering research • Comprised of 200+ members including staff and students • + startups in various application area (including robotics, NLP, vision, …) Researchers (about 10 people) Staff (About 50 people) Assigned students (About 50 people) TAs (About 20 people) Prof. Matsuo Masters and doctoral courses: Technology Management for Innovation, Graduate School of Engineering Department: Systems Innovation Program, Faculty of Engineering ＋ Areas of specialization Deep Learning (2011 ~ ) R&D on cutting-edge technology (Deep Learning) Application to image recognition and robotics Web engineering (2002 ~ ) Analysis of social media, big data, etc. Design and operation of web services
- 3. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 3 Representative Officer Prof. Yutaka Matsuo 1997 Graduated from the Faculty of Engineering at The University of Tokyo with a Bachelor’s degree in Information and Communication Engineering 2002 Completed a doctoral program and earned a doctorate in engineering at the Graduate School of Engineering at The University of Tokyo Became a researcher with the National Institute of Advanced Industrial Science and Technology (AIST) From Oct. 2005. Visiting Scholar, Stanford University From Oct. 2007 Associate Professor, Institute of Engineering Innovation / Center for Structuring of Knowledge / Department of Technology Management for Innovation, Graduate School of Engineering, The University of Tokyo From 2014. Joint Representative and Project Associate Professor, Chair for Global Consumer Intelligence, Department of Technology Management for Innovation, Graduate School of Engineering, The University of Tokyo 2012–14 Editor-in-Chief, Transactions of the Japanese Society for Artificial Intelligence, then Chair of Ethics Committee (present post) June 2017 Founder and Director, Japan Deep Learning Association (JDLA) From April 2019 Professor of Research Into Artifacts, Center for Engineering (RACE) / Department of Technology Management for Innovation, Graduate School of Engineering, The University of Tokyo From June 2019 Concurrent Outside Director, SoftBank Group Corp. From Oct. 2021 Member, Council of New Form of Capitalism Realization Japan Deep Learning Association
- 4. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 4 Building an Ecosystem • Matsuo lab is aiming to build an ecosystem where the results of research are not kept within academia, but widely shared in the form of startups and services, so that the benefits of those economic activities recirculate and promote further research, in The University of Tokyo and the Hongo area. Implementation Fundamental research Education Incubation (eventually becoming big companies) Expertise, resources, etc. for success to be returned to the academia. Return the know-how and resources for success to the university Nurturing of technological “seeds” and let them in society Provide advanced education base on fundamental research Provide practical learning opportunities through classroom lectures, OJT and participation in joint researches Create a spiral of innovation
- 5. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 5 Overview 3. Implementation 1. Fundamental Research 2. Education 4. Incubation R&D focused on Deep Learning and its application, aiming to make machines smarter and explain the principles of intelligence Promotion of DX in industry through joint researches on Deep Learning in collaboration with private companies Development and provision of human resource development programs for students and adults (not limited to students and faculty members of The University of Tokyo) Nurture and support launch of startups from universities and laboratories by providing entrepreneurship education
- 6. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 6 Fundamental Research | Team Mission ■Mission Create intelligence and discover the principle of human intelligence 1. How the human brain works is one of the big mysteries 2. Making machine smarter have a huge social impact ■Research Fields 1. Algorithm of deep learning (RL, Generative Models, Transfer Learning) 2. Application of deep learning (NLP, Vision, Robotics)
- 7. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 7 Fundamental Research | World Models • The key to future Deep Learning development is technology to simulate the real world, and the machine equivalent of human imagination. • The world’s leading AI companies are already conducting ongoing research on this topic. Sources: https://deepmind.com/blog/article/neural-scene-representation-and-rendering, https://worldmodels.github.io Technology Overview Examples Humans can use imagination to compensate for gaps in information and to posit future conditions. E.g. Imagining future from the current state Similarly, the key to the future development of AI is to efficiently learn “common sense” from experiences, and to be able to “imagine” the future. The core technology used is World Models. CRASH!! E.g. Looking at a part of an object and imagining the whole picture Leading companies and laboratories like Google focus on World Models are conducting research in this topic Example from DeepMind (Google): Reconstructing an entire object or group of objects from a series of limited views The object of this game is to avoid the bullets. The system’s ability to avoid being hit is improved by incorporating an efficient mechanism to imagine the future “Neural scene representation and rendering”, S. A. Eslami, et al., Science, 360(6394):1204–1210, 2018. Based on images from three different perspectives, AI reconstructs the 3D view. Example from Google Brain: Efficient prediction of future events “Recurrent world models facilitate policy evolution”. D. Ha, J. Schmidhuber, NeurIPS 2018, pp. 2455–2467, 2018.
- 8. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 8 Our fundamental research on the world model • Object-centric world models: • A framework for recognizing and predicting representations for each object in an image or video without explicit supervision. • We propose a model that separates the representations of objects that are related to interaction (dynamic representations, such as positions) and those that are not related (global representations, such as colors). • We successfully separated the representations so that we can change only the color of the object without changing its position. [Greff+ 20] Dynamic representation Global representation
- 9. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 9 Workshops on the world model • Organized session on the world model at JSAI2023 (the largest conference on AI in Japan) • Workshop on world models at IROS2023 (top-tier robotics conference)
- 10. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 10 Changes in automated driving technology using the world model Current Pipeline Processing 1. Recognizes the presence of a bicycle in front of the car 2. Predicts the path of a bicycle 3. Recognizes that there are an obstacle in the path of the car 4. Judges that the bicycle will be in the path of the car due to the presence of a telephone pole 5. Generate path to avoid the bicycle 6. Determines the amount of control to run along the path 1. Input current sensing data into the world model 2. Output predictions from the world model 3. Generate a path that does not conflict with the predicted bicycle path 4. Determines the amount of control to run along that path 5. Output control amount from the world model as well Example: A telephone pole and a bicycle running are in front of your car Given as a rule
- 11. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 11 Fundamental Research | Application (Robotics) • By applying deep learning, we aim to develop intelligent robotics systems • Combining recent progress in deep learning (including LLMs), our tidy up robot system won 1st prize at the RoboCup23 (Japan) and 3rd prize in RobotCup23 (world competition)。
- 12. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 12 Fundamental Research | Research on prompt engineering We are also researching on large language models. Our research member, Takeshi Kojima, found prompt, “Let’s think step by step”, which elicits the logical knowledges and improve logical reasoning. Standard Prompting Proposed Prompting (Zero-Shot CoT) • LLM are typically give poor performance on multi-step reasoning (e.g. math) • Internal working of the LLMs is also unclear • Simply add a magical phrase (known as prompt), “Let’s think step by step” elicit logical knowledge • Improve reasoning performance e.g., MultiArith (17.7% -> 78.7%) ”Large Language Models are Zero-Shot Reasoners”, NeurIPS2022, (900+ citations at 2023/11/21)
- 13. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 13 LLM Model"Weblab-10B" from Matsuo Lab. （2023/8/18） • Developed a Large Language Model (LLM) with 10 billion parameters for Japanese and English by pre- training and post-training (fine tuning), and released the model to the public. • The model was released to the public by pre-training and post-training (fine tuning). The model was designed to increase the amount of training data by using not only Japanese but also English datasets for training, and to improve the accuracy of Japanese by transferring knowledge between languages. • This is the highest level of publicly available model in Japan.
- 14. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 14 Fundamental Research | Accepted publications • Top-tier International Conferences ：ICLR, NeurIPS, ICML, AAAI, IJCAI etc… 10 top-tier papers accepted 2020 2021 2022 2023 3 8 10 4月〜3月までの採録数 Number of researchers doubled from 10 3 7 20 (20 FTE) 11 10 (8.1FTE) We are also building new technologies that are completely different from traditional deep learning toward an innovative theory that connects the brain and AI. 2020 2021 2022 2023
- 15. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 15 Fundamental Research | Accepted conferences 15 Deep Generative and World Models • “A System for Morphology-Task Generalization via Unified Representation and Behavior Distillation”, ICLR2023 • “End-to-End Training of DBMs by Unbiased Contrastive Divergence with Local Mode Initialization”, ICML2023 • “DreamSparse: Escaping from Plato’s Cave with 2D Frozen Diffusion Model given Sparse Views”, NeurIPS2023 Reinforcement Learning and Robotics • “Control Graph as Unified IO for Morphology-Task Generalization”, ICLR2023 (Spotlight) • ICLR2021, ICML2021, NeurIPS2021, ICLR2022 (Spotlight) etc. Transfer learning • “Collective Intelligence for 2D push Manipulations with Mobile Robots”, RA-L, 2023 • “Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization”, NeurIPS2021 (Spotlight) • IJCAI2022 etc. Natural language processing (NLP) • “Large-Language Models are Zero-Shot Reasoners”, NuerIPS2022 • EMNLP2021×2, ACL2021, NAACL2022×2, EMNLP2022×2，ICLR2023 Theory • “Regularization and Variance-Wighted Regression Achieves Minmax Optimality in Linear MDPs: Theory and Practice”, ICML2023 • "Group Equivariant Conditional Neural Processes“, ICLR2021
- 16. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 16 Overview 3. Implementation 1. Fundamental Research 2. Education 4. Incubation Promotion of DX in industry through joint researches on Deep Learning in collaboration with private companies Development and provision of human resource development programs for students and adults (not limited to students and faculty members of The University of Tokyo) Nurture and support launch of startups from universities and laboratories by providing entrepreneurship education R&D focused on Deep Learning and its application, aiming to make machines smarter and explain the principles of intelligence
- 17. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 17 Education | Overview of Lectures Web Engineering and Business Models Web工学 Fundamental Projects Web Engineering Chair for Global Consumer Intelligence (Cultivation of data scientists) Data Science Business Models Web工学 Deep Learning Basic courses Deep Learning Web工学 Entrepreneurship • In AY 2022-23, Matsuo lab provides 15 lectures covering the following topics: • “Web engineering” provides lectures on a fundamental technology for user experience design • “Data Science” focus on data-science technology and its applications in business and management • “Deep Learning” covering topics from the basis of Deep Learning to cutting-edge knowledge • “Entrepreneurship” offers ideas and knowledge on starting a technology-based company Intensive courses Spring seminar (Image recognition) Summer school （Financial trading and machine learning, Deep Generative Model, Natural Language Processing) Entrepreneurship Education Design Donation Course Invitation to Deep Tech Entrepreneurs Visionary startups Chair for AI Business Transformation Chair for World Models Seminar on data-driven entrepreneurship Seminar on data-driven business proposition
- 18. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 18 Education | Results (Attendance) • From April 2014 to February 2021, a total of over 7,500 students and working adults attended the courses. • In AY 2023-24 alone, attendance is expected to surpass 11,000, raising cumulative attendance to over 200,000 in the coming year. 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 2014 2015 2016 2017 2018 2019 2020 2021 To approx. 11,000 2022 2023 5,700
- 19. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 19 Overview 3. Implementation 1. Fundamental Research 2. Education 4. Incubation R&D focused on Deep Learning (World Models) aiming to achieve smarter machines and explain the principles of knowledge Promotion of DX in industry through joint researches on Deep Learning in collaboration with private companies Development and provision of human resource development programs for students and adults (not limited to students and faculty members of The University of Tokyo) Nurture and support launch of startups from universities and laboratories by providing entrepreneurship education
- 20. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 20 Social Implementation | Results and Case Studies • Matsuo Lab is pursuing the application of research in a wide variety of topics, industries or types of data, such as self-driving cars, object detection, image processing, behavior analysis or forecasting. Theme Industry Project Overview Image analysis Healthcare Development of image diagnosis algorithms to assist in the diagnosis diagnosis of major dementias such as Alzheimer’s disease, by detecting micro-hemorrhages in MRI scans Behavior analysis Manufacturing (Parts) Detection and visualization of people’s movement in the factory to factory to analyze the cause of defective product rates and devise the devise the transfer of skilled workers’ expertise Forecasting Manufacturing (Chemicals) Early detection of abnormalities in chemical plants and identification of identification of their causes (Photos are for illustrative purposes only.)
- 21. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 21 Overview 3. Implementation 1. Fundamental Research 2. Education 4. Incubation R&D focused on Deep Learning (World Models) aiming to achieve smarter machines and explain the principles of knowledge Promotion of DX in industry through joint researches on Deep Learning in collaboration with private companies Development and provision of human resource development programs for students and adults (not limited to students and faculty members of The University of Tokyo) Nurture and support launch of startups from universities and laboratories by providing entrepreneurship education
- 22. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 22 Incubation | Kigyo (起業) Quest • Kigyo Quest is a program that abstracts the success model of startups launched at Matsuo Lab and provide the knowledge to increase the probability of success • Launched in the summer of 2021 • By April 2022, there are 3 companies registered and currently preparing to launch 1st Stage Getting Your Weapons (Education) Attend certified Kigyo Quest classes to learn the basics of your technology (data science/deep learning). 2nd Stage Getting Stronger Through Practice (Application) Join a company as a certified AI engineering intern and gain experience in development, project management and pitching proposals to clients as on-the-job training. Developing the skills and business strengths needed for launching your own business. 3rd Stage Forming Your Party, Starting Your Journey Find partners and set foot into the real business world. Gain experience in making proposals to clients and acquire the knowledge and skills necessary to start up a company New entrants Senior engineer Entrepreneur in the making START
- 23. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO 23 Incubation | 23 Startups Launched by Matsuo Lab 23 listed • Since 2012, 23 start-ups (including 2 listed companies) have been launched by the graduates of Matsu Lab
- 24. ©︎MATSUO LAB, THE UNIVERSITY OF TOKYO
Editor's Notes
- Thank you for your interest in Matsuo laboratory. Through this presentation, I would like to give you a brief introduction of our lab
- Matsuo Lab belongs to the Graduate School of Engineering at The University of Tokyo and specializes in Artificial Intelligence (AI) and Web Engineering research In our lab, we have over 50 staff members including 10 researchers, who are engaged in fundamental research, and 40 engineering and business experts, who are planning AI lectures and managing entrepreneurship classes, and handling collaborative research with private companies. We also have 40 assigned students learning at the Lab.
- Our representative, Professor Matsuo also serves as an outside director and technical advisor to companies, He also serves as a government commissioner and association board member, contributing widely to industry, government, and academia. Recently, He is appointed as the Chairman of AI strategy roundtable for Japanese Government.
- To realize our vision, we are engaging in 4 activities: Fundamental research, Education, Implementation, and Incubation. We aim to nurture the technological seeds born from Fundamental Research and engage in joint research with companies. The knowledge gained from these activities are used to develop human resources, such as student entrepreneurs. University ventures and graduates who have learned about technology will become leaders in promoting digital transformation in the industrial world and contribute to Japanese society as a whole. If the results are returned to academia, the next generation of technology and human resources will be created. In this spiral of innovation, we hope that many pioneers will be born in this spiral of innovation. Next, I will describe each of our activities.
- In fundamental research, we conduct research and development focusing on deep learning and its application.
- Through fundamental research, we aim to discover human intelligence in engineering To achieve this, we are working from two perspectives: first, on algorithms for deep learning, and second, on applications of deep learning.
- One of the technologies we believe is important for achieving intelligence is the world model. A world model is a technology to make Deep Learning models train AI to compensate the gaps in information and to be able to predict the future with understanding of physical law and passage of time. Learning this model would allow AI to think like human; in other world, they will be able to predict the future from the current state and to imagine parts of an object that cannot be seen. For example, humans are able to predict what happens when a glass falls. Although we are aware that the glass would break, robots and AIs won’t be able to predict on site. In order to overcome this inability, world model provides AI and robots the ability to predict and respond to causal dynamics in the physical world. The world model has attracted attention from institutes and companies around the world, including GAFA, and various studies have recently conducted. As an example, the research by google can predict how the external world will look from a new perspective based on partial observations of the external world. Furthermore, in the research below, after the world model of the game environment is learned from the video, we can learn the agent's behavior only within that world model. The advantage of using world model as an approach is that, unlike in the real world, the learning can be repeated as many as the researcher want. In a real environment, things can break if you move them around too many times, but learning on the world model does not have this problem.These world model studies are expected to be applied to robotics and other fields.
- Here I present one of the our foundational studies of the world model. Existing world models did not take into account what objects exist in the environment and did not learn individual representations. However, humans understand what each object is and can predict what will happen when that object moves. In other words, we can learn object-centric representations based on environmental information. Object-centric world model is an area of research that attempts to achieve this in world models. In this approach, a representation corresponding to each object is acquired by learning. The goal is to infer the appropriate representation from the image and predict the original image from it, even without annotations about the objects in the image. We proposed to separate object representations into time-dependent dynamic representations and non-time-dependent global representations. Dynamic representations correspond to elements such as object positions, and global representations correspond to elements such as object colors. For example, in the figure, there are two sequences with two objects. and these positions change over time. Our proposed world model is capable of swapping only the colors of objects between two transitions by replacing only the global representations.. This result demonstrates that we can obtain global representations in object-centric world models. We also found that this separation of representations improves the performance of future predictions by properly capturing object-sense interactions.
- We are also involved in various activities to promote the world model in our research community. For example, we hold a session on the World Model every year at JSAI, the largest AI conference in Japan. This year, we held a workshop on the World Model at IROS, one of the top-tier robotics conferences in the world.
- The world model is also applied in automated driving technology to improve its capabilities. For example, there are two obstacles in front of a car, bicycle and pole. The current pipeline processing in automated driving can recognize the presence of bicycle but not the pole. By using world model, the automated driven car can predict the movement of bicycle driving in pathway that avoids the poles. We are aiming to apply this world model to automated driving technology and so on.
- The Matsuo Lab also has a team working on the application of deep learning algorithms to robotics. We have a robotics contest team called "TRAIL," which currently consists of first-year undergraduates, The team is aiming to improve the accuracy of its household robots by training them to clean up a room or pick up and deliver a specified item from a shelf. The team won first place in Japan Robocup and won third place in the World Robocup held in France 2023 summer. In this presentation, we will present you the video of robot working on tasks. Playback normally until 02:00 Explanation "This is actually how the robot actually opens the drawers based on the technology I just described." After opening the first drawer Explanation "The video is long, so I'm going to shortcut it." Shortcut the video to 02:59. 02:59 "The robot recognizes an object on the floor and grabs it with its arm, "and stores it in a designated place for each type of object. Robot grabs an apple. Explanation: "The robot recognizes and grabs the apple. Robot grabs detergent. Explanation: "The robot recognized and grabbed the detergent and put it in a different basket than the apples because it is detergent. Robot grabs banana. Explanation: "The banana is also fruit, so it recognized it and put it on the same tray as the apple. Robot grabbing other things Explanation: "There are other types of tasks in the robotics competition as well” Stop the video. Explanation "As you can see in the video, the robot is still moving slowly. Matsuo Lab is also aiming to realize more complex and smooth robot activities using the world model technology I just introduced."
- We are also researching on large language models, LLM. I know many of you use ChatGPT, so I assume you are familiar with prompt engineering. Our work presented here is a pioneering study of prompt engineering. Our research member, Takeshi Kojima, found a prompt “Let’s think step by step” and how it improves the answer generated by ChatGPT, especially on logical problems. For example, the chatGPT gave wrong answer for standard prompt which is on left side. On the other hand, with the proposed prompt, chatGPT was able to think thoroughly and gave correct answer.
- While we conduct research on how to utilize the existing service and technology, we also aim to develop our own. We released Large Language Models called WEBLAB-10B, trained with 10billion size parameters, 2023 summer. At the time of its release it was the most accurate open LLM in Japan.
- Our researches have been accepted by many top-tier international conferences, and the number is increasing every year. In parallel, the number of researchers in Matsuo Laboratory is also increasing every year. We are also building new technologies that are completely different from traditional deep learning
- These are a list of recently accepted papers. A wide range of research, from basic research to applied research, has been accepted at international conferences. We will continue our research toward the realization of intelligence.
- Another important function of Matsuo laboratory is providing lectures.
- Matsuo Lab offers more than 15 educational programs under four themes: Web Engineering, Data Science, Deep Learning, and Entrepreneurship Development. Most of the courses can be taken by not only the students of the University of Tokyo but also students of other universities and high schools.
- The number of students has been increasing since the lecture began. The current number of students taking our open DL classes is over 5700 annually, and the total number of attendance is expected to be over 10,000 at the end of this year.
- We also provide an internship program for those students who want to gain hands-on experience after taking lectures. As a junior engineer member, students will join in collaborative research program.
- The collaborative research projects are conducted with clients of various industries such as motor, chemical, construction, medical industry, and so on. Some of the collaborative project in 2023 includes Collaboration with medical institutions on early detection of Alzheimer's disease by detecting minute hemorrhages from MRI images of the brain, We are also working with a chemical plant on a project for early detection of abnormalities in the plant and investigation of their causes. we also have collaboration researches with companies that are not mentioned here.
- Students who participate in such internships will acquire business sense and team development skills, which are fundamental in starting their startup companies. We offer various projects to nurture and support such entrepreneurs.
- One of the example of entrepreneurship program we offer is Kigyo quest. Kigyo means start up in Japanese and it is an education program to nurture entrepreneurs through E-learning classes and on-the-job-training projects. This program is designed to help participants become AI startup entrepreneurs after the completion of this quest.
- So far, there are 19 start-ups launched by graduates of Matsuo lab and two of them got listed.
- Thank you for listening.