Detecting person s direction of interest (1)

•

0 likes•105 views

This project was completed during the Lviv Data Science Summer School 2018 (http://cs.ucu.edu.ua/en/summerschool/summer-school-2018/).

Education

Detecting person's direction
of interest
George Barvinok
Marta Didych
Denys Filippov
Yurii Ostapchuk
Andrii Palyha
UCU
Data Science
Summer School
Supervised by:
Oles Dobosevych
Ricker Lyman Robotics

Project
● Proposed by Ricker Lyman Robotic
● Customers behaviour analysis
● Retail domain

Given
● 3 days
● 5 people
● some data
● @dobosevych

Goals
● Try and analyse different CV techniques
● Build a PoC
● Learn something

Dataset
Top-view cameras
● Stanford dataset (https://www.albert.cm/projects/viewpoint_3d_pose/)
○ Depth cameras
○ Labeled joints
● Politecnica delle Marche (http://vrai.dii.univpm.it/re-id-dataset)
○ Depth & Colored
○ No labels

OpenCV Background subtraction + Hough Circles
Real-time solution!
● Background subtraction
● Mrophological operations
to remove noise
● Bounding box
● Histogram equalization
Blur
● Hough Circles
● CNN for head detection
●

Tensorflow Object Detection API
Pretrained SSD Mobilenet on COCO
Trained on ~250 images
Head detection

Head segmentation with U-net
Masks received from
depth maps and
trained using VGG U-net
Train on colored images
Still not enough data
Predicted masks from U-net

Putting things together
Prepare labeled dataset +
TF Object Detection API - detect people heads +
CNN Regression - detect head direction +
OpenCV - visualize gaze gradient

Python 3, OpenCV 3.4, Tensorflow 1.9, Keras 2.1
Google Cloud Platform for training on GPU
Hough Circles
U-net custom / VGG
Tensorflow Object Detection API
YOLO, SSD, Mobilenet
Technologies

Summary / Lessons Learnt
● Dataset - is the key, need more labeled data
● Classical CV is not enough but can improve quality
● Need much more optimizations for real-time solution
○ Right now - ~2 frames per second

What’s next?
● Prepare more data
● Different approach
○ whole body segmentation instead of head only
○ use more classical CV for preprocessing data
● Involve capturing from different angles
● Different models, hyperparameter tuning

Similar to Detecting person s direction of interest (1)

When Remote Sensing Meets Artificial Intelligence

WahyuRahmaniar2

See you @ GI_Forum

shj69

Panel session at the Society of American Archivists 2016 annual meeting in Atlanta, Georgia, with Ed Busch (Michigan State University), Chris Prom (University of Illinois at Urbana-Champaign), Molly Tighe (Chatham University), and Greg Wiedeman (SUNY Albany). It will happen to you, if it hasn't already: the campus photographer retires and leaves behind hundreds of photo CDs or a hard drive packed with JPEGs. What happens next? Digital photograph collections present serious challenges but offer opportunities to leverage automation (from deduplication to face recognition) and collaborative, cross-departmental workflows. Come hear this panel of experienced archivists discuss steps taken, lessons learned, and best practices developed for working (and teaching!) with born-digital photographs.

From 0 to 400 GB: Confronting the Challenges of Born-Digital Photographs

Kristen Yarmey

Thesispresentatie November

Robin De Croon

Andrii Boichuk: Video-based action recognition - past, present and future

Lviv Startup Club

Introduction to data science

Sampath Kumar

Computer vision - Applications and Trends

Kshitij Agrawal

What you did last summer?

DoThinger

Brodmann17 CVPR 2017 review - meetup slides

Brodmann17

Cvpr 2017 Summary Meetup

Amir Alush

Image search at facebook - making sense of one of the largest image databases...

MLconf

In this lecture we have Ohyoon Kwon, Giovanni Maggini and Piotr Steininger from the Code for Europe project, who will share their experiences in building apps for three challenges in the City of Amsterdam. How do you spread tourists in the city? How do you manage offer/request of spaces from people in a neighbourhood that want to join or start activities? How do you help a team of professionals working on the front line and on the edge of legality to prevent youngsters degrade? The Code for Europe project in Amsterdam offered these three challenges to the Fellows, and were given 9 months to solve them. After the turning point of their project, they will present discoveries, lessons learned, obstacles, apps that were built and future plans. Feedback from the Appsterdam community will help drive the development of the last phase of the project! "Code for Europe is a new organization looking to solve local civic challenges, by temporary agile teams of developers, in a way that the solutions are easily reusable for other European cities." The Fellows in Amsterdam present their progress at

Code for Europe Amsterdam Fellows presentation at Appsterdam Lunchtime lectur...

Giovanni Maggini

Introduction talk to Computer Vision

Chen Sagiv

Kaggle Competitions, New Friends, New Skills and New Opportunities

Jo-fai Chow

A-13 Iomp-1.pptx

Jayendranath3

Luiz eduardo. introduction to mobile snitch

Yury Chemerkin

Week1- Introduction.pptx

fahmi324663

Cloud Programming Models: eScience, Big Data, etc.

Alexandru Iosup

Object extraction from satellite imagery using deep learning

Aly Abdelkareem

PR-146: CornerNet detecting objects as paired keypoints

jaewon lee

Similar to Detecting person s direction of interest (1) (20)

When Remote Sensing Meets Artificial Intelligence

See you @ GI_Forum

From 0 to 400 GB: Confronting the Challenges of Born-Digital Photographs

Thesispresentatie November

Andrii Boichuk: Video-based action recognition - past, present and future

Introduction to data science

Computer vision - Applications and Trends

What you did last summer?

Brodmann17 CVPR 2017 review - meetup slides

Cvpr 2017 Summary Meetup

Image search at facebook - making sense of one of the largest image databases...

Code for Europe Amsterdam Fellows presentation at Appsterdam Lunchtime lectur...

Introduction talk to Computer Vision

Kaggle Competitions, New Friends, New Skills and New Opportunities

A-13 Iomp-1.pptx

Luiz eduardo. introduction to mobile snitch

Week1- Introduction.pptx

Cloud Programming Models: eScience, Big Data, etc.

Object extraction from satellite imagery using deep learning

PR-146: CornerNet detecting objects as paired keypoints

More from Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 24th Abstract. In digital marketing, memes have become an attractive tool for engaging an online audience. Memes have an impact on buyers’ and sellers’ online behavior and information spreading processes. Thus, the technology of generating memes is a significant tool for social media engagement. In this study, we collected a new memes dataset of ∼650K meme instances, applied state of the art Deep Learning technique – GPT-2 model [1] towards meme generation, and compared machine-generated memes with human-created. We justified that MTurk workers can be used for the approximate estimating of users’ behavior in a social network, more precisely to measure engagement. Generated memes cause the same engagement as human memes, which didn’t collect engagement in the social network (historically). Still, generated memes are less engaging then random memes created by humans.

Master defence 2020 - Andrew Kurochkin - Meme Generation for Social Media Aud...

Lviv Data Science Summer School

Master defence 2020 - Andrew Kurochkin - Meme Generation for Social Media Aud...

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 23d Abstract. In modern days synthesis of human images and videos is arguably one of the most popular topics in the Data Science community. The synthesis of human speech is less trendy but deeply bonded to the mentioned topic. Since the publication of WaveNet paper by Google researchers in 2016, the state-of-the-art approach transferred from parametric and concatenative systems to deep learning models. Most of the work on the area focuses on improving the intelligibility and naturalness of the speech. However, almost every significant study also mentions ways to generate speech with the voices of different speakers. Usually, such an enhancement requires the model’s re-training in case of generating audio with the voice of a speaker that was not present in the training set. Additionally, studies focused on highly modular speech generation are rare. Therefore there is a room left for research on ways to add new parameters for other aspects of the speech, like sentiment, prosody, and melody. In this work, we aimed to implement a competitive text-to-speech solution with the ability to specify the speaker without model re-training and explore possibilities for adding emotions to the generated speech. Our approach generates good quality speech with the mean opinion score of 3,78 (out of 5) points and the ability to mimic speaker voice in real-time, which is a big improvement over the baseline that merely obtains 2,08. On top of that, we researched sentiment representation possibilities. We built an emotion classifier that performs on the level of the current state of the art solutions by giving an accuracy of more than eighty percent.

Master defence 2020 - Nazariy Perepichka - Parameterizing of Human Speech Gen...

Lviv Data Science Summer School

Master defence 2020 - Anastasiia Khaburska - Statistical and Neural Language ...

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 24th Abstract. This work presents a context-based question answering model for the Ukrainian language based on Wikipedia articles using Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2018) model, which takes a context (Wikipedia article) and a question to the context. The result of the model is an answer to the question. The model consists of two parts. The first one is a pre-trained multilingual BERT model, which is trained on the top-100, the most popular languages on Wikipedia articles. The second part is the fine-tuned model, which is trained on the data set of questions and answers to the Wikipedia articles. The training and validation data is Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016). There are no question answering datasets for the Ukrainian language. The plan is to build an appropriate dataset with machine translation and use it for the fine-tuning training stage and compare the result with models which were fine-tuned on the other languages. The next experiment is to train a model on the Slavic language datasets before fine-tuning on the Ukrainian language and compare the results.

Master defence 2020 - Serhii Tiutiunnyk - Context-based Question-answering Sy...

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 24th Abstract. This work tackles the problem of matching Wikipedia red links with existing articles. Links in Wikipedia pages are considered red when leading to nonexistent articles. In other Wikipedia, editions could exist articles that correspond to such red links. In our work, we propose a way to match red links in one Wikipedia edition to existent pages in another edition. We solve this task in the context of Ukrainian red links and English existing pages. We created a dataset of 3 171 most frequent Ukrainian red links and a dataset of 2 957 927 pairs of red links and the most probable candidates for the corresponding pages in English Wikipedia. This dataset is publicly released. We defined the task as a Named Entity Linking problem. Red links are named entities and we link Ukrainian red links to English Wikipedia pages. In this work, we provide a thorough analysis of the data and define its conceptual characteristics to exploit in entity resolution. These characteristics are graph properties (connections with the pages where red links occur and connections with the pages which occur in the same pages with red links) and word properties (title names). BabelNet knowledge base was applied to this task. We evaluated its powers in terms of F1 score (29 %) and regarded it as a baseline for our approach. To improve the results we introduced several similarity metrics based on mentioned red links characteristics. Combined in a linear model they resulted in F1 score 85 % which is our best result. In our thesis, we also discuss the bottlenecks and limitations of the current approach and outline the ideas for future improvements. To the best of our knowledge, we are the first to state the problem and propose a solution for red links in the Ukrainian Wikipedia edition. All the code for this project is publicly released on github.

Master defence 2020 - Kateryna Liubonko - Matching Red Links to Wikidata Items

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 24th Abstract. Every day a lot of visitors leave countless reviews about hotels, restaurants, cafes, attractions or other services. In most cases, they set the rate about this service, sometimes they also set the rate about the specific topic if service provides this possibility. However, the main information about user opinion is hidden inside the body of review text. Thereby, in this work, we propose a solution to analyze one or several user reviews, determine sentiments and acquire important characteristics for these reviews. We determine which characteristics were influenced by such reviews. In this case, the proposed solution can detect sentiments from text and classify for pos-itive and negative. Then it acquires top positive and negative phrases, which can explain why the user left such review. Besides, we analyze all reviews about one hotel or just several reviews and summarize the most important positive and negative properties for a specific hotel.

Master defence 2020 - Dmytro Babenko - Determining Sentiment and Important Pr...

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 23rd Abstract. Advances in the demand response for energy imbalance management (EIM) ancillary services can change the future power systems. These changes are subject to research in academia and industry. Although an important/promising part of this research is the application of Machine Learning methods to shape future power systems domain, the domain has not fully benefited from this application yet. Thus, the main objective of the presented project is to investigate and assess opportunities for applying reinforcement learning (RL) to achieve such advances by developing an intelligent voltage control-based ancillary service that uses thermostatically controlled loads (TCLs). Two stages of the project are presented: a proof of concept (PoC) and extensions. The PoC includes modeling and training of a voltage controller utilizing Q-learning, chosen due to its efficiency that is achieved without unnecessary sophistication. Simplest relevant for demand response power system of 20 TCLs is considered in the experiments to provide ancillary service. The power system model is developed with Modelica tools. Extensions aim to exceed PoC performance by applying advanced RL methods: Q-learning modification that uses a window of environment states as an input (WIQL), smart discretization strategies for environment’s continuous state space and a deep Q-network (DQN) with experience replay. To investigate particularities of the developed controller, modifications in an experimental setup such as controller testing longer than training, different simulation start time is considered. The improvement of 4% in median performance is achieved compared to the competing analytical approach – optimal constant control chosen using whole time interval simulation for the same voltage controller design. The presented results and corresponding discussions can be useful for both further works on the RL-driven voltage controllers for EIM and other applications of RL in the power system domain using Modelica models.

Master defence 2020 - Oleh Lukianykhin - Reinforcement Learning for Voltage C...

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 23rd Abstract. Speaker classification is an essential task in the machine learning domain, with many practical applications in identification and natural language processing. This work concentrates on speaker classification as a subtask of general speaker diarization for real-world conversation scenarios. We research the domain of modern speech processing and present the original speaker classification approach based on the recent developments in convolutional neural networks. Our method uses a spectrogram as input to the CNN classifier model, allowing it to capture spatial information about voice frequencies distribution. Presented results show beyond human ability performance and give strong prospects for future development.

Master defence 2020 - Borys Olshanetskyi -Context Independent Speaker Classif...

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 23rd Abstract. Currently, the active development of image processing methods requires large amounts of correctly labeled data. The lack of quality data makes it impossible to use various machine learning methods. In case of limited possibilities for collecting real data, used methods for their synthetic generation. In practice, we can formulate the task of the high-quality generation of synthetic images as an efficient generation of complex data distributions, which is the object of study of this work. Generating high-quality synthetic data is an expensive and complicated process in terms of existing methods. We can distinguish two main approaches that are used to generate synthetic data: image generation based on rendered 3-D scenes and the use of GANs for simple images. These methods have some drawbacks, such as a narrow range of applicability and insufficient distribution complexity of the obtained data. When using GANs to generate complex distributions, in practice, we face a visible increase in the complexity of the model architecture and training procedure. A deep understanding of the real data complex distributions can be used to improve the quality of synthetic generation. Minimizing the differences in the real and synthetic data distributions can improve not only the generation process but also develop tools for solving the problem of data lack in the field of image processing.

Master defence 2020 - Philipp Kofman - Efficient Generation of Complex Data D...

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 23rd Abstract. Customer Lifetime Value (CLV) is a present value of the future cash flows attributed to a customer during their entire relationship with the company (Farris et al., 2010). CLV represents a 360-degree view of the client’s business situation (McKinsey, Customer Lifecycle Management), which takes into account the probability of customer churn and their future purchases. The modeling of CLV in retail is a complicated task due to the lack of access to historical data of purchases, the difficulty of customer identification, and building the historical reference with a particular customer. In this research, historical transactional data were taken from twelve North American brick-and-mortar grocery stores to compare different approaches to CLV modeling in terms of segmentation and forecast. The research has resulted in the suggestions on CLV estimation for the offline retail business case with given advantages and limitations of each approach.

Master defence 2020 - Anastasiia Kasprova - Customer Lifetime Value for Retai...

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 23rd Abstract. In this project (Glusco and Maksymenko, 2019), we treat the Reinforcement Learning problem of Exploration vs. Exploitation. The problem can be rephrased in terms of generalization and overfitting or efficient learning. To face the problem we decided to combine the techniques from different researches: we introduce noise as an environment’s characteristics (Packer et al., 2018); create multiple Reinforcement Learning agents and environments setup to train in parallel and interact within each other (Jaderberg et al., 2017); use parallel tempering approach to initialize environments with different temperatures (noises) and perform exchanges using Metropolis-Hastings criterion (Pushkarov et al., 2019). We implemented multi-agent architecture with a parallel tempering approach based on two different Reinforcement Learning agent algorithms – Deep Q Network and Advantage Actor-Critic – and environment wrapper of the OpenAI Gym (Gym: A toolkit for developing and comparing reinforcement learning algorithms) environment for noise addition. We used the CartPole environment to run multiple experiments with three different types of exchanges: no exchange, random exchange, smart exchange according to Metropolis-Hastings rule. We implemented aggregation functionality to gather the results of all the experiments and visualize them with charts for analysis. Experiments showed that a parallel tempering approach with multiple environments with different noise level can improve the performance of the agent under specific circumstances. At the same time, results raised new questions that should be addressed to fully understand the picture of the implemented approach.

Master defence 2020 - Dmitri Glusco - Replica Exchange For Multiple-Environme...

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 22nd Abstract. The thesis introduces the reader to the concepts of edge computing in terms of person re-identification and tracking problem. It describes the challenges, limitations, and current state-of-the-art solutions. The author proposed a pipeline for the task, launched several experiments on validating different parts of the system, and provided a theoretical explanation of the person re-identification process in the overlapping multi-camera environment.

Master defence 2020 - Ivan Prodaiko - Person Re-identification in a Top-view ...

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 22nd Abstract. Generative Adversarial Networks (GANs) in recent years has certainly become one of the biggest trends in the computer vision domain. GANs are used for generating face images and computer game scenes, transferring artwork style, visualizing designs, creating super-resolution images, translating text to images, etc. We want to present a model to solve an image problem: generate new outfits onto people’s images. This task seems to be extremely important for the offline/online trade and fashion industry.Changing clothing on people’s images isn’t a trivial task. The generated part of the image should have high quality without blurring. Another problem is generating long sleeves on the images with T-shirts, for example. As a result, well-known models are not suitable for this task. In the master project, we are going to reproduce the model for clothing hanging on people’s images based on the existing approaches and improve it in order to get better quality of the image.

Master defence 2020 - Yevhen Pozdniakov - Changing Clothing on People Images...

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 22nd Abstract. Generative adversarial networks (GANs) are one of the most popular models capable of producing high-quality images. However, most of the works generate images from the vector of random values, without explicit control of desired output properties. We study the ways of introducing such control for the user-selected region of interest (RoI). First, we overview and analyze the existing works in areas of image completion (inpainting) and controllable generation. Second, we propose our model based on GANs, which united approaches from the two mentioned areas, for the controllable local content generation. Third, we evaluate the controllability of our model on three accessible datasets – Celeba, Cats, and Cars – and give numerical and visual results of our method.

Master defence 2020 - Oleh Onyshchak - Image Recommendation for Wikipedia Ar...

Lviv Data Science Summer School

Master defence 2020 - Oleh Misko - Ensembling and Transfer Learning for Multi...

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 22nd Abstract. Today virtual and augmented reality applications become more and more popular. Such a trend creates a demand for 3D processing algorithms which may be applied to many areas. This work is focused on sigh language video sequences. There are a lot of prerecorded photos and video dictionaries that can be transformed into 3D and unified in one place. We research nuances of hand pose video sequence analysis as well as the influence of results refinement for 2D and 3D keypoint detection. Besides that, we designed a solution for the parametrization of hand shape and engineered system for 3D hand pose reconstruction. Model show good results on train data but lack generalization. Retraining on multiple datasets and usage of various data augmentation techniques will improve performance.

Master defence 2020 - Roman Riazantsev - 3D Reconstruction of Video Sign Lan...

Lviv Data Science Summer School

Master defence 2020 - Vadym Korshunov - Region-Selected Image Generation with...

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 21st Abstract. Novelty is an inherent part of innovations and discoveries. Such processes may be considered as the appearance of new ideas or as the emergence of atypical connections between existing ones. The importance of such connections hints for investigation of innovations through network or graph representation in the space of ideas. In such representation, a graph node corresponds to the relevant notion (idea), whereas an edge between two nodes means that the corresponding notions have been used in a common context. The question addressed in this research is the possibility to identify the edges between existing concepts where the innovations may emerge. To this end, a well-documented scientific knowledge landscape has been used. Namely, we downloaded 1.2M arXiv.org manuscripts dated starting from April 2007 and until September 2019; and extracted relevant concepts for them using ScienceWISE.info platform. Combining approaches developed in complex networks science and graph embedding the predictability of edges (links) on the scientific knowledge landscape where the innovations may appear is investigated. We argue that the conclusions drawn from this analysis may be used not only to the scientific knowledge analysis but are rather generic and may be applied to any domain that involves creativity within.

Master defence 2020 -Roman Moiseiev - Stock Market Prediction Utilizing Centr...

Lviv Data Science Summer School

Ukrainian Catholic University Faculty of Applied Sciences Data Science Master Program January 21st Abstract. Human navigation in information spaces has increasing importance in ever-growing data sources we possess. Therefore, an efficient navigation strategy would give a huge benefit to the satisfaction of human information needs. Often, the search space can be understood as a network and navigation can be seen as a walk on this network. Previous studies have shown that despite not knowing the global network structure people tend to be efficient at finding what they need. This is usually explained by the fact that people possess some background knowledge. In this work, we explore an adapted version of the network consisting of Wikipedia pages and links between them as well as human trails on it. The goal of our research is to find a procedure to label articles that are similar to a given one. Among others, this would lay a foundation for a recommender system for Wikipedia editors, which will suggest links from the given page to the related articles. Our work is, therefore, providing a basement for enhancing the Wikipedia navigation process making it more user-friendly.

Master defence 2020 - Maksym Opirskyi -Topological Approach to Wikipedia Arti...

Lviv Data Science Summer School

More from Lviv Data Science Summer School (20)