This document summarizes several papers on quantitative models of neural language representation. It discusses encoding models that use language representations like word embeddings to model brain activity measured by fMRI or MEG in response to linguistic stimuli like words, sentences and stories. The models are evaluated based on their ability to linearly encode and decode from brain activity and have high representational similarity with it. Several papers find that distributional word embeddings can accurately predict brain responses in language areas. Context is also found to improve modeling accuracy compared to individual words. The document analyzes the methods, results and implications of these quantitative models of neural language representation.
HIGH SCHOOL TIMETABLING USING TABU SEARCH AND PARTIAL FEASIBILITY PRESERVING ...P singh
The high school timetabling is a combinatorial optimization problem. It is proved to be NP-hard and has several hard and soft constraints. A given set of events, class-teacher meetings and resources are assigned to the limited space and time under hard constraints which are strictly followed and soft constraints which are satisfied as far as possible. The feasibility of timetable is determined by hard constraints and the soft constraints determine its quality. Difficult combinatorial optimization problems are frequently solved using Genetic Algorithm (GA). We propose Partial Feasibility Preserving Genetic Algorithm (PFP-GA) combined with tabu search to solve hdtt4, “hard timetabling” problem a test data set in OR-Library. The solution to this problem is zero clashes and maintaining teacher’s workload on each class in given venue. The modified GA procedures are written for intelligent operators and repair. The PFP-GA in association with Tabu Search (TS) converges faster and gives solution within a few seconds. The results are compared to that of using different methodologies on same data set.
Creative cognition in the city: underlying principles for creativity and inno...Andy Dong
This presentation was delivered at the 2nd Delft International Conference on Complexity, Cognition, Urban Planning and Design (http://www.bk.tudelft.nl/ccupd). The talk presents a theory for creative cities, by linking together the predictive brain hypothesis and design thinking.
HIGH SCHOOL TIMETABLING USING TABU SEARCH AND PARTIAL FEASIBILITY PRESERVING ...P singh
The high school timetabling is a combinatorial optimization problem. It is proved to be NP-hard and has several hard and soft constraints. A given set of events, class-teacher meetings and resources are assigned to the limited space and time under hard constraints which are strictly followed and soft constraints which are satisfied as far as possible. The feasibility of timetable is determined by hard constraints and the soft constraints determine its quality. Difficult combinatorial optimization problems are frequently solved using Genetic Algorithm (GA). We propose Partial Feasibility Preserving Genetic Algorithm (PFP-GA) combined with tabu search to solve hdtt4, “hard timetabling” problem a test data set in OR-Library. The solution to this problem is zero clashes and maintaining teacher’s workload on each class in given venue. The modified GA procedures are written for intelligent operators and repair. The PFP-GA in association with Tabu Search (TS) converges faster and gives solution within a few seconds. The results are compared to that of using different methodologies on same data set.
Creative cognition in the city: underlying principles for creativity and inno...Andy Dong
This presentation was delivered at the 2nd Delft International Conference on Complexity, Cognition, Urban Planning and Design (http://www.bk.tudelft.nl/ccupd). The talk presents a theory for creative cities, by linking together the predictive brain hypothesis and design thinking.
Multimodal Learning Analytics for Collaborative Learning Understanding and Su...Sambit Praharaj
This project has multiple focus points: using the help of Multimodal Learning Analytics to understand how co-located collaboration takes place, what are the indicators of collaboration (such as pointing at peer, looking at peer, making
constructive interruptions, etc.); then we try to form a Collaboration Framework (CF)
which defines the aspects of successful collaboration and forms a model. These
insights help us to build the support framework to enable efficient real-time feedback
during a group activity to facilitate collaboration.
A short introduction to raise awareness about the role of NLP in health informatics. The talk aims to briefly describe some of the linguistic and technical challenges in linking texts to knowledge bases/ontologies and present several techniques that we have explored (Limsopatham and Collier, 2016).
Talk given at Cologne AI Deep Learning Meetup 21.05.2019
Over past year, string of deep learning innovations around Transformers, ELMo, BERT and co. destroyed previous state-of-the art NLP benchmarks.
We‘ll look how we got there, what future might look like and what you can do with it.
A Brief history of NLP deep learning, showing common thread behind the recent hype with a small intermezzo on ethics.
Learning with me Mate: Analytics of Social Networks in Higher EducationDragan Gasevic
Effects of social interactions are reported in research on higher education to lead to positive outcomes such as higher levels of internalization, sense of community, academic achievement, metacognition, and student retention. The role of social networks has especially been emphasized in research due to the availability of theoretical foundations and analytic methods to investigate their effects in higher education. The increased use of technologies in education allows for the collection of large and rich datasets about social networks which call for the use of novel analytics methods. This talk will first give a brief overview of the existing work on and lessons learned from some well-known studies on social networks in higher education in diverse situations from face-to-face to massive open online courses. The talk will then identify critical challenges that require immediate attention in order for the study of social networks to make a sustainable impact on learning and teaching. The most important take away from the talk will be that
- computational aspects of the study of social networks need to be integrated deeply with theory, research and practice,
- novel methods for the study of critical dimensions (discourse, structure and dynamics) that shape network formation and network effects are necessary, and
- innovative instructional approaches are essential to address the changing conditions created by contemporary educational and technological contexts.
Trends in deep learning in 2020 - International Journal of Artificial Intelli...gerogepatton
The International Journal of Artificial Intelligence & Applications (IJAIA) is a bi monthly open access peer-reviewed journal that publishes articles which contribute new results in all areas of the Artificial Intelligence & Applications (IJAIA). It is an international journal intended for professionals and researchers in all fields of AI for researchers, programmers, and software and hardware manufacturers. The journal also aims to publish new attempts in the form of special issues on emerging areas in Artificial Intelligence and applications.
Thank You for referencing this work, if you find it useful!
Citation of a related scientific paper:
Manea, V., Wac, K., (2018). mQoL: Mobile Quality of Life Lab: From Behavior Change to QoL, Mobile Human Contributions: Opportunities and Challenges (MHC) Workshop in conjunction with ACM UBICOMP, Singapore, October 2018.
Katarzyna Wac, From Quantified Self to Quality of Life, Book Chapter in "Digital Health", Health Informatics, Springer Nature, p. 83-108, Dordrecht, The Netherlands, 2018.
The talk details:
Katarzyna Wac, “Quality of Life Technologies: From Cure to Care”, Société Suisse des Pharmaciens Hospitaliers (GSASA), November 2018, Switzerland
This presentation was on Empathic Mixed Reality, which we applied Mixed Reality technology to Empathic Computing in our studies. We shared an overview of our research and selected findings. This talk was given at ETRI and KAIST in Daejeon, South Korea, on the 24th of May 2017.
Speakers: Yuanyuan Lin, Dana Reijerkerk
Mastery of the Chinese characters could probably be considered as one of the most difficult and strenuous tasks for Chinese language learners. The present research is designed to address how Chinese characters are processed and organized in the cognitive approaches between memory and reasoning. In this session, presenters will share the findings, which divulge how fuzzy-trace theory benefits Chinese character learning and helps students to become more independent and effective language learners. The research also suggests that providing assistance to the students to form traces and visual-spatial analysis of the Chinese characters would significantly increase students’ performance. The research procedure, method, data, and results will be shared during the session. This session has implications for the daily classroom practices of using certain techniques to best acquire vocabulary in a second language.
This was part of the Doctoral Consortium presentation in the ICMI Conference 2019 at Suzhou, China on 14th October, 2019. Collaboration is an important skill of the 21st century. It can take place in an online (or remote) setting or in a colocated
(or face-to-face) setting. With the large scale adoption
of sensor use, studies on co-located collaboration (CC) has
gained momentum. CC takes place in physical spaces where
the group members share each other’s social and epistemic
space. This involves subtle multimodal interactions such
as gaze, gestures, speech, discourse which are complex in
nature. The aim of this PhD is to detect these interactions
and then use these insights to build an automated real-time
feedback system to facilitate co-located collaboration
Towards reproducibility and maximally-open dataPablo Bernabeu
Presented at the Open Scholarship Prize Competition 2021, organised by Open Scholarship Community Galway.
Video of the presentation: https://nuigalway.mediaspace.kaltura.com/media/OSW2021A+OSCG+Open+Scholarship+Prize+-+The+Final!/1_d7ekd3d3/121659351#t=56:08
Multimodal Learning Analytics for Collaborative Learning Understanding and Su...Sambit Praharaj
This project has multiple focus points: using the help of Multimodal Learning Analytics to understand how co-located collaboration takes place, what are the indicators of collaboration (such as pointing at peer, looking at peer, making
constructive interruptions, etc.); then we try to form a Collaboration Framework (CF)
which defines the aspects of successful collaboration and forms a model. These
insights help us to build the support framework to enable efficient real-time feedback
during a group activity to facilitate collaboration.
A short introduction to raise awareness about the role of NLP in health informatics. The talk aims to briefly describe some of the linguistic and technical challenges in linking texts to knowledge bases/ontologies and present several techniques that we have explored (Limsopatham and Collier, 2016).
Talk given at Cologne AI Deep Learning Meetup 21.05.2019
Over past year, string of deep learning innovations around Transformers, ELMo, BERT and co. destroyed previous state-of-the art NLP benchmarks.
We‘ll look how we got there, what future might look like and what you can do with it.
A Brief history of NLP deep learning, showing common thread behind the recent hype with a small intermezzo on ethics.
Learning with me Mate: Analytics of Social Networks in Higher EducationDragan Gasevic
Effects of social interactions are reported in research on higher education to lead to positive outcomes such as higher levels of internalization, sense of community, academic achievement, metacognition, and student retention. The role of social networks has especially been emphasized in research due to the availability of theoretical foundations and analytic methods to investigate their effects in higher education. The increased use of technologies in education allows for the collection of large and rich datasets about social networks which call for the use of novel analytics methods. This talk will first give a brief overview of the existing work on and lessons learned from some well-known studies on social networks in higher education in diverse situations from face-to-face to massive open online courses. The talk will then identify critical challenges that require immediate attention in order for the study of social networks to make a sustainable impact on learning and teaching. The most important take away from the talk will be that
- computational aspects of the study of social networks need to be integrated deeply with theory, research and practice,
- novel methods for the study of critical dimensions (discourse, structure and dynamics) that shape network formation and network effects are necessary, and
- innovative instructional approaches are essential to address the changing conditions created by contemporary educational and technological contexts.
Trends in deep learning in 2020 - International Journal of Artificial Intelli...gerogepatton
The International Journal of Artificial Intelligence & Applications (IJAIA) is a bi monthly open access peer-reviewed journal that publishes articles which contribute new results in all areas of the Artificial Intelligence & Applications (IJAIA). It is an international journal intended for professionals and researchers in all fields of AI for researchers, programmers, and software and hardware manufacturers. The journal also aims to publish new attempts in the form of special issues on emerging areas in Artificial Intelligence and applications.
Thank You for referencing this work, if you find it useful!
Citation of a related scientific paper:
Manea, V., Wac, K., (2018). mQoL: Mobile Quality of Life Lab: From Behavior Change to QoL, Mobile Human Contributions: Opportunities and Challenges (MHC) Workshop in conjunction with ACM UBICOMP, Singapore, October 2018.
Katarzyna Wac, From Quantified Self to Quality of Life, Book Chapter in "Digital Health", Health Informatics, Springer Nature, p. 83-108, Dordrecht, The Netherlands, 2018.
The talk details:
Katarzyna Wac, “Quality of Life Technologies: From Cure to Care”, Société Suisse des Pharmaciens Hospitaliers (GSASA), November 2018, Switzerland
This presentation was on Empathic Mixed Reality, which we applied Mixed Reality technology to Empathic Computing in our studies. We shared an overview of our research and selected findings. This talk was given at ETRI and KAIST in Daejeon, South Korea, on the 24th of May 2017.
Speakers: Yuanyuan Lin, Dana Reijerkerk
Mastery of the Chinese characters could probably be considered as one of the most difficult and strenuous tasks for Chinese language learners. The present research is designed to address how Chinese characters are processed and organized in the cognitive approaches between memory and reasoning. In this session, presenters will share the findings, which divulge how fuzzy-trace theory benefits Chinese character learning and helps students to become more independent and effective language learners. The research also suggests that providing assistance to the students to form traces and visual-spatial analysis of the Chinese characters would significantly increase students’ performance. The research procedure, method, data, and results will be shared during the session. This session has implications for the daily classroom practices of using certain techniques to best acquire vocabulary in a second language.
This was part of the Doctoral Consortium presentation in the ICMI Conference 2019 at Suzhou, China on 14th October, 2019. Collaboration is an important skill of the 21st century. It can take place in an online (or remote) setting or in a colocated
(or face-to-face) setting. With the large scale adoption
of sensor use, studies on co-located collaboration (CC) has
gained momentum. CC takes place in physical spaces where
the group members share each other’s social and epistemic
space. This involves subtle multimodal interactions such
as gaze, gestures, speech, discourse which are complex in
nature. The aim of this PhD is to detect these interactions
and then use these insights to build an automated real-time
feedback system to facilitate co-located collaboration
Towards reproducibility and maximally-open dataPablo Bernabeu
Presented at the Open Scholarship Prize Competition 2021, organised by Open Scholarship Community Galway.
Video of the presentation: https://nuigalway.mediaspace.kaltura.com/media/OSW2021A+OSCG+Open+Scholarship+Prize+-+The+Final!/1_d7ekd3d3/121659351#t=56:08
THE IMPORTANCE OF MARTIAN ATMOSPHERE SAMPLE RETURN.Sérgio Sacani
The return of a sample of near-surface atmosphere from Mars would facilitate answers to several first-order science questions surrounding the formation and evolution of the planet. One of the important aspects of terrestrial planet formation in general is the role that primary atmospheres played in influencing the chemistry and structure of the planets and their antecedents. Studies of the martian atmosphere can be used to investigate the role of a primary atmosphere in its history. Atmosphere samples would also inform our understanding of the near-surface chemistry of the planet, and ultimately the prospects for life. High-precision isotopic analyses of constituent gases are needed to address these questions, requiring that the analyses are made on returned samples rather than in situ.
Professional air quality monitoring systems provide immediate, on-site data for analysis, compliance, and decision-making.
Monitor common gases, weather parameters, particulates.
Observation of Io’s Resurfacing via Plume Deposition Using Ground-based Adapt...Sérgio Sacani
Since volcanic activity was first discovered on Io from Voyager images in 1979, changes
on Io’s surface have been monitored from both spacecraft and ground-based telescopes.
Here, we present the highest spatial resolution images of Io ever obtained from a groundbased telescope. These images, acquired by the SHARK-VIS instrument on the Large
Binocular Telescope, show evidence of a major resurfacing event on Io’s trailing hemisphere. When compared to the most recent spacecraft images, the SHARK-VIS images
show that a plume deposit from a powerful eruption at Pillan Patera has covered part
of the long-lived Pele plume deposit. Although this type of resurfacing event may be common on Io, few have been detected due to the rarity of spacecraft visits and the previously low spatial resolution available from Earth-based telescopes. The SHARK-VIS instrument ushers in a new era of high resolution imaging of Io’s surface using adaptive
optics at visible wavelengths.
This presentation explores a brief idea about the structural and functional attributes of nucleotides, the structure and function of genetic materials along with the impact of UV rays and pH upon them.
Cancer cell metabolism: special Reference to Lactate PathwayAADYARAJPANDEY1
Normal Cell Metabolism:
Cellular respiration describes the series of steps that cells use to break down sugar and other chemicals to get the energy we need to function.
Energy is stored in the bonds of glucose and when glucose is broken down, much of that energy is released.
Cell utilize energy in the form of ATP.
The first step of respiration is called glycolysis. In a series of steps, glycolysis breaks glucose into two smaller molecules - a chemical called pyruvate. A small amount of ATP is formed during this process.
Most healthy cells continue the breakdown in a second process, called the Kreb's cycle. The Kreb's cycle allows cells to “burn” the pyruvates made in glycolysis to get more ATP.
The last step in the breakdown of glucose is called oxidative phosphorylation (Ox-Phos).
It takes place in specialized cell structures called mitochondria. This process produces a large amount of ATP. Importantly, cells need oxygen to complete oxidative phosphorylation.
If a cell completes only glycolysis, only 2 molecules of ATP are made per glucose. However, if the cell completes the entire respiration process (glycolysis - Kreb's - oxidative phosphorylation), about 36 molecules of ATP are created, giving it much more energy to use.
IN CANCER CELL:
Unlike healthy cells that "burn" the entire molecule of sugar to capture a large amount of energy as ATP, cancer cells are wasteful.
Cancer cells only partially break down sugar molecules. They overuse the first step of respiration, glycolysis. They frequently do not complete the second step, oxidative phosphorylation.
This results in only 2 molecules of ATP per each glucose molecule instead of the 36 or so ATPs healthy cells gain. As a result, cancer cells need to use a lot more sugar molecules to get enough energy to survive.
Unlike healthy cells that "burn" the entire molecule of sugar to capture a large amount of energy as ATP, cancer cells are wasteful.
Cancer cells only partially break down sugar molecules. They overuse the first step of respiration, glycolysis. They frequently do not complete the second step, oxidative phosphorylation.
This results in only 2 molecules of ATP per each glucose molecule instead of the 36 or so ATPs healthy cells gain. As a result, cancer cells need to use a lot more sugar molecules to get enough energy to survive.
introduction to WARBERG PHENOMENA:
WARBURG EFFECT Usually, cancer cells are highly glycolytic (glucose addiction) and take up more glucose than do normal cells from outside.
Otto Heinrich Warburg (; 8 October 1883 – 1 August 1970) In 1931 was awarded the Nobel Prize in Physiology for his "discovery of the nature and mode of action of the respiratory enzyme.
WARNBURG EFFECT : cancer cells under aerobic (well-oxygenated) conditions to metabolize glucose to lactate (aerobic glycolysis) is known as the Warburg effect. Warburg made the observation that tumor slices consume glucose and secrete lactate at a higher rate than normal tissues.
A brief information about the SCOP protein database used in bioinformatics.
The Structural Classification of Proteins (SCOP) database is a comprehensive and authoritative resource for the structural and evolutionary relationships of proteins. It provides a detailed and curated classification of protein structures, grouping them into families, superfamilies, and folds based on their structural and sequence similarities.
Seminar of U.V. Spectroscopy by SAMIR PANDASAMIR PANDA
Spectroscopy is a branch of science dealing the study of interaction of electromagnetic radiation with matter.
Ultraviolet-visible spectroscopy refers to absorption spectroscopy or reflect spectroscopy in the UV-VIS spectral region.
Ultraviolet-visible spectroscopy is an analytical method that can measure the amount of light received by the analyte.
Comparing Evolved Extractive Text Summary Scores of Bidirectional Encoder Rep...University of Maribor
Slides from:
11th International Conference on Electrical, Electronics and Computer Engineering (IcETRAN), Niš, 3-6 June 2024
Track: Artificial Intelligence
https://www.etran.rs/2024/en/home-english/
Slide 1: Title Slide
Extrachromosomal Inheritance
Slide 2: Introduction to Extrachromosomal Inheritance
Definition: Extrachromosomal inheritance refers to the transmission of genetic material that is not found within the nucleus.
Key Components: Involves genes located in mitochondria, chloroplasts, and plasmids.
Slide 3: Mitochondrial Inheritance
Mitochondria: Organelles responsible for energy production.
Mitochondrial DNA (mtDNA): Circular DNA molecule found in mitochondria.
Inheritance Pattern: Maternally inherited, meaning it is passed from mothers to all their offspring.
Diseases: Examples include Leber’s hereditary optic neuropathy (LHON) and mitochondrial myopathy.
Slide 4: Chloroplast Inheritance
Chloroplasts: Organelles responsible for photosynthesis in plants.
Chloroplast DNA (cpDNA): Circular DNA molecule found in chloroplasts.
Inheritance Pattern: Often maternally inherited in most plants, but can vary in some species.
Examples: Variegation in plants, where leaf color patterns are determined by chloroplast DNA.
Slide 5: Plasmid Inheritance
Plasmids: Small, circular DNA molecules found in bacteria and some eukaryotes.
Features: Can carry antibiotic resistance genes and can be transferred between cells through processes like conjugation.
Significance: Important in biotechnology for gene cloning and genetic engineering.
Slide 6: Mechanisms of Extrachromosomal Inheritance
Non-Mendelian Patterns: Do not follow Mendel’s laws of inheritance.
Cytoplasmic Segregation: During cell division, organelles like mitochondria and chloroplasts are randomly distributed to daughter cells.
Heteroplasmy: Presence of more than one type of organellar genome within a cell, leading to variation in expression.
Slide 7: Examples of Extrachromosomal Inheritance
Four O’clock Plant (Mirabilis jalapa): Shows variegated leaves due to different cpDNA in leaf cells.
Petite Mutants in Yeast: Result from mutations in mitochondrial DNA affecting respiration.
Slide 8: Importance of Extrachromosomal Inheritance
Evolution: Provides insight into the evolution of eukaryotic cells.
Medicine: Understanding mitochondrial inheritance helps in diagnosing and treating mitochondrial diseases.
Agriculture: Chloroplast inheritance can be used in plant breeding and genetic modification.
Slide 9: Recent Research and Advances
Gene Editing: Techniques like CRISPR-Cas9 are being used to edit mitochondrial and chloroplast DNA.
Therapies: Development of mitochondrial replacement therapy (MRT) for preventing mitochondrial diseases.
Slide 10: Conclusion
Summary: Extrachromosomal inheritance involves the transmission of genetic material outside the nucleus and plays a crucial role in genetics, medicine, and biotechnology.
Future Directions: Continued research and technological advancements hold promise for new treatments and applications.
Slide 11: Questions and Discussion
Invite Audience: Open the floor for any questions or further discussion on the topic.
Richard's aventures in two entangled wonderlandsRichard Gill
Since the loophole-free Bell experiments of 2020 and the Nobel prizes in physics of 2022, critics of Bell's work have retreated to the fortress of super-determinism. Now, super-determinism is a derogatory word - it just means "determinism". Palmer, Hance and Hossenfelder argue that quantum mechanics and determinism are not incompatible, using a sophisticated mathematical construction based on a subtle thinning of allowed states and measurements in quantum mechanics, such that what is left appears to make Bell's argument fail, without altering the empirical predictions of quantum mechanics. I think however that it is a smoke screen, and the slogan "lost in math" comes to my mind. I will discuss some other recent disproofs of Bell's theorem using the language of causality based on causal graphs. Causal thinking is also central to law and justice. I will mention surprising connections to my work on serial killer nurse cases, in particular the Dutch case of Lucia de Berk and the current UK case of Lucy Letby.
PRESENTATION ABOUT PRINCIPLE OF COSMATIC EVALUATION
Journal club: Quantitative models of neural language representation
1. Quantitative models of neural
language representation
Journal club
2020.02.10
Takuya Koumura
2. 2020.02.10
Takuya KOUMURA
p. 2
Research paradigm
Assumption:
A good quantitative model of neural representation should
⚫be able to linearly encode
⚫be linearly decodable from
⚫have high similarity with
the brain activities.
But see: Raman, R. & Hosoya, H. CNN explains
tuning properties of anterior, but not middle, face-
processing areas in macaque IT. bioRxiv 1–33 (2019).
Corpus
Language
representation
Training
Evaluation
StimulusStimulus
Language
representation
✕
Linear encoding
Linear decoding
Representational
similarity
3. 2020.02.10
Takuya KOUMURA
p. 3
Papers
⚫ Mitchell, T. M. et al. Predicting human brain activity associated with the meanings of nouns. Science. 320,
1191–1195 (2008).
⚫ Huth, A. G., De Heer, W. A., Griffiths, T. L., Theunissen, F. E. & Gallant, J. L. Natural speech reveals the
semantic maps that tile human cerebral cortex. Nature. 532, 453–458 (2016).
⚫ Pereira, F. et al. Toward a universal decoder of linguistic meaning from brain activation. Nat. Commun. 9,
(2018).
⚫ Wehbe, L., Vaswani, A., Knight, K. & Mitchell, T. Aligning context-based statistical models of language
with brain activity during reading. EMNLP. 233–243 (2014).
⚫ Qian, P., Qiu, X. & Huang, X. Bridging LSTM Architecture and the Neural Dynamics during Reading. IJCAI.
1953–1959 (2016).
⚫ Jain, S. & Huth, A. Incorporating Context into Language Encoding Models for fMRI. NeurIPS. 6628–6637
(2018).
⚫ Abnar, S., Beinborn, L., Choenni, R. & Zuidema, W. Blackbox meets blackbox: Representational
Similarity and Stability Analysis of Neural Language Models and Brains. arxiv. (2019).
⚫ Gauthier, J. & Ivanova, A. Does the brain represent words? An evaluation of brain decoding studies of
language understanding. arxiv. (2018).
⚫ Sun, J., Wang, S., Zhang, J. & Zong, C. Towards Sentence-Level Brain Decoding with Distributed
Representations. AAAI. 33, 7047–7054 (2019).
⚫ Gauthier, J. & Levy, R. Linking artificial and human neural representations of language. arxiv. (2019).
4. 2020.02.10
Takuya KOUMURA
p. 4
Summary Language representation Brain activities
Paper Paradigm Model Training data Recording Stimulus Evaluation
Mitchell
2008
Encoding Co-occurrence frequency (25
words)
LDC2006T13 fMRI 60 noun-picture pairs
(visual)
Pairwise
classification
Huth
2016
Encoding Co-occurrence frequency
(985 words)
Moth stories, books,
Wikipedia pages,
reddit.com
fMRI The Moth Radio Hour (audio) Pearson
correlation for each
voxel
Pereira
2018
Decoding GloVe Pre-trained fMRI Sentence, word & picture, word cloud
(visual)
Pairwise
classification, Rank
accuracy
Wehbe
2014
Encoding RNN, CNN Harry Potter fan
fiction database
MEG Chapter 9 of Harry Potter and the
Philosopher’s Stone (visual, word-by-word)
Pairwise
classification
Qian
2016
Encoding LSTM Harry Porter and the
Philosopher’s Stone
fMRI chapter 9 from Harry Porter and the
Philosopher’s Stone (visual, word-by-word)
Cosine distance
Jain
2018
Encoding LSTM reddit.com Huth 2016 Huth 2016 Sum of r2 across
voxels
Abnar
2019
RSA, encoding GloVe, ELMO, GoogleLM,
UniSentEnc, BERT
Pre-trained fMRI (from
another
Wehbe 2014)
chapter 9 of Harry Potter and the Sorcerer’s
stone
Representational
similarity, r2
Gauthier
2018
Decoding GloVe, LSTM, BiLSTM,
CNN+attention
Pre-trained Pereira 2018 Pereira 2018 Average rank
Sun
2019
Decoding
(Similarity based,
linear, MLP)
Average, max, FastSent, SIF,
Skip-thought, Quick-Thought,
InferSent, GenSen
Pre-trained Pereira 2018 Pereira 2018 Pairwise matching,
Ranking
Gauthier
2019
Decoding BERT fine-tuned for various
tasks
Pre-trained Pereira 2018 Pereira 2018 MSE, average
ranking
6. 2020.02.10
Takuya KOUMURA
p. 6
My conclusions & impressions
⚫Contexts often improve modeling of neural representation
⚫Model complexity and intelligence does not always improve
modeling of neural representation
⚫No study tried raw stimulus reconstruction (as far as I read)
8. 2020.02.10
Takuya KOUMURA
p. 8
Methods
⚫ Encoding model
⚫ Language representation
⚪ Distributional word
representation (25 dimensional)
⚫ The frequency with which a
word co-occurs with the 25
chosen verbs: “see, hear, listen,
taste, smell, eat, touch, rub, lift,
manipulate, run, push, fill,
move, ride, say, fear, open,
approach, near, enter, drive,
wear, break, clean”
⚫ In a very large text corpus
(LDC2006T13)
⚫Brain activities
⚪fMRI
⚪60 noun-picture pairs
⚪Visually presented
11. 2020.02.10
Takuya KOUMURA
p. 11
Results
⚫“push” activates the right postcentral gyrus (premotor planning)
⚫“run” activates the posterior portion of the right superior
temporal sulcus (perception of biological motion)
14. 2020.02.10
Takuya KOUMURA
p. 14
Methods
⚫ Encoding model
⚪ L2-regularized linear regerssion
⚫ Language representation
⚪ Distributional word representation
(985 dimensional)
⚫ normalized co-occurrence between
each word and a set of 985 common
English words
⚫ Wikipedia’s List of 1000 Basic Words
(contrary to the title, this list contained only 985
unique words at the time it was accessed)
⚫ Dataset
⚪ 13 Moth stories (including the
stimuli for fMRI)
⚪ 604 popular books
⚪ 2,405,569 Wikipedia pages,
⚪ 36,333,459 user comments from
reddit.com
⚫ Brain activities
⚪ fMRI
⚪ > 2 hours of The Moth Radio Hour
⚫ Evaluation
⚪ Pearson correlation for each
voxel
17. 2020.02.10
Takuya KOUMURA
p. 17
Methods
⚫Decoding model
⚪L2-regularized linear regression
⚫Language representation
⚪GloVe (300 dimensional word representation)
⚫ Pennington, J., Socher, R. & Manning, C.D. GloVe: Global Vectors for Word Representation. Proc. Conf.
Emp. Meth. Nat. Lang. Proc. 1532–1543 (2014)
⚪For sentences: average of all words in the sentence
⚫Brain activities
⚪fMRI
⚪Stimuli
⚫Experiment 1: 180 manually selected words
⚫Experiment 2: 24 manually selected concepts
⚫Experiment 3: 24 manually selected concepts
18. 2020.02.10
Takuya KOUMURA
p. 18
Methods: Experiment 1
⚫180 manually selected
words
⚪Selected among 30,000
words by clustering
based on the word
representation
⚫ 30,000 words: Brysbaert, M.,
Warriner, A. B. & Kuperman, V.
Concreteness ratings for 40
thousand generally known English
word lemmas. Behav. Res.
Methods 46, 904–911 (2014).
⚪128 nouns, 22 verbs,
23 adjectives, 6
adverbs, 1 function
word
20. 2020.02.10
Takuya KOUMURA
p. 20
Methods: Experiment 2 & 3
⚫Experiment 2
⚪24 manually selected concepts
⚪A sentence that provided basic information about the concept
⚫Experiment 3
⚪24 manually selected concepts
⚪A passage related to the concept
24. 2020.02.10
Takuya KOUMURA
p. 24
Results
⚫Distribution of the informative voxels
⚪Language: the frontotemporal language-selective network
⚫ Fedorenko, E., Behr, M. K. & Kanwisher, N. Functional specificity for high- level linguistic processing in
the human brain. PNAS 108, 16248–16433 (2011)
⚪Default: the default mode network
⚫ Buckner, R. L., Andrews-Hanna, J. R. & Schacter, D. L. The brain’s default network: anatomy, function,
and relevance to disease. Ann. N. Y. Acad. Sci. 1124,1–38 (2008).
⚫ Binder, J. R., Desai, R. H., Graves, W. W. & Conant, L. L. Where is the semantic system? A critical review
and meta-analysis of 120 functional neuroimaging studies. Cereb. Cortex 19, 2767–2796 (2009).
⚪Task: the task-positive network
⚫ Power, J. D. et al. Functional network organization of the human brain. Neuron 72, 665–678 (2011).
⚫ Buckner 2008 (above)
⚫ Binder 2009 (above)
⚪Visual: the visual network
⚫ Power 2011 (above)
⚫ Buckner 2008 (above)
26. 2020.02.10
Takuya KOUMURA
p. 26
Methods
⚫Encoding model (linear, ridge)
⚫Language representation
⚪RNN
⚫w: one-hot
⚪CNN (they call it neural probabilistic LM)
⚫u: one-hot
⚪Dataset: Harry Potter fan fiction database
27. 2020.02.10
Takuya KOUMURA
p. 27
Methods
⚫Brain activities
⚪MEG
⚪Stimulus
⚫Chapter 9 of Harry Potter and the Philosopher’s Stone
⚫Words were presented one by one at the center of the screen for
0.5 s
⚫Evaluation
⚪Pairwise classification
31. 2020.02.10
Takuya KOUMURA
p. 31
Methods
⚫ Encoding model (linear)
⚫ Language representation
⚪ LSTM
⚪ Dataset: Harry Porter and the
Philosopher’s Stone (excluding
chapter 9)
⚫ Brain activities
⚪ fMRI
⚪ chapter 9 from Harry Porter and
the Philosopher’s Stone
⚪ Words presented one by one for
0.5 s
⚫ Evaluation
⚪ Cosine distance, transformed to [0,
1]
33. 2020.02.10
Takuya KOUMURA
p. 33
Results
⚫They also tested other language representation for comparison
⚪tf-idf: frequency-inverse document frequency, classical features
for document retrieval
⚪AveEmbedding: average embeddins of a word sequence
They also conducted ablation study (skipped today)
36. 2020.02.10
Takuya KOUMURA
p. 36
Methods
⚫Encoding model (linear, ridge)
⚫Language representation
⚪LSTM
⚪Dataset: reddit.com
⚫Brain activities
⚪fMRI
⚪> 2 hour of The Moth Radio Hour
⚪ Huth, A. G., De Heer, W. A., Griffiths, T. L., Theunissen, F. E. & Gallant, J. L. Natural speech
reveals the semantic maps that tile human cerebral cortex. Nature. 532, 453–458 (2016).
⚫Evaluation
⚪Sum of r2 across voxels
42. 2020.02.10
Takuya KOUMURA
p. 42
Methods
⚫Representational similarity analysis | encoding model
⚫Language representation
⚫Brain activities
⚪fMRI
⚪Stimulus: chapter 9 of Harry Potter and the Sorcerer’s stone
⚪ Wehbe L, Murphy B, Talukdar P, Fyshe A, Ramdas A, Mitchell T (2014) Simultaneously Uncovering the
Patterns of Brain Regions Involved in Different Story Reading Subprocesses. PLoS One 9:e112575
⚫Evaluation
⚪Representational similarity analysis
⚪R2
47. 2020.02.10
Takuya KOUMURA
p. 47
Methods
⚫Decoding model (L2-regularized)
⚫Language representation
⚫Brain activities
⚪fMRI
⚪ Pereira, F. et al. Toward a universal decoder of linguistic meaning from brain activation. Nat. Commun.
9, (2018).
⚫Evaluation
⚪Mean average rank
51. 2020.02.10
Takuya KOUMURA
p. 51
Methods
⚫Language representation
⚪Unstructured model
⚫Simple pooling of word representation
⚪Average
⚪Max-pooling
⚪Concatenation of averaging & max-pooling
⚫Parameterized pooling
⚪FastSent (Hill, F.; Cho, K.; and Korhonen, A. 2016. Learning distributed representations of
sentences from unlabelled data. NAACL-HLT)
⚪SIF (Arora, S.; Liang, Y.; and Ma, T. 2016. A simple but tough-to- beat baseline for sentence
embeddings. ICLR)
52. 2020.02.10
Takuya KOUMURA
p. 52
Methods
⚫Language representation
⚪Structured model
⚫Unsupervised methods
⚪Skip-thought (Kiros, R.; Zhu, Y.; Salakhutdinov, R. R.; Zemel, R.; Urtasun, R.; Torralba, A.;
and Fidler, S. 2015. Skip-thought vectors. NeurIPS, 3294–3302)
⚪Quick-Thought (Logeswaran, L., and Lee, H. 2018. An efficient framework for learning
sentence representations. arXiv:1803.02893)
⚫Supervised methods
⚪InferSent (Conneau, A.; Kiela, D.; Schwenk, H.; Barrault, L.; and Bordes, A. 2017. Supervised
learning of universal sentence representations from natural language inference data. EMNLP)
⚪GenSen (Subramanian, S.; Trischler, A.; Bengio, Y.; and Pal, C. J. 2018. Learning general purpose
distributed sentence representations via large scale multi-task learning. ICLR)
53. 2020.02.10
Takuya KOUMURA
p. 53
Methods
⚫Brain activities
⚪fMRI
⚪ Pereira, F. et al. Toward a universal decoder of linguistic meaning from brain activation. Nat.
Commun. 9, (2018).
⚫Evaluation
⚪Pairwise matching
⚪Ranking
57. 2020.02.10
Takuya KOUMURA
p. 57
Methods
⚫Decoding model (L2-regularized linear)
⚫Language representation
⚪Sentence representation in BERT
⚪Fine tuned on several tasks
⚪And custom tasks (modifications of masked
language model pre-training)
⚫Scrambled within sentences
⚫Scrambled within paragraphs
⚫Predicting only part-of-speech
58. 2020.02.10
Takuya KOUMURA
p. 58
BERT: Bidirectional Encoder Representations from Transformers
⚫ Devlin J, Chang M-W, Lee K, Toutanova K (2018) BERT: Pre-training of Deep
Bidirectional Transformers for Language Understanding.
⚫Architecture
⚪Stacked self-attention
⚫Pre-training
⚪Masked language model
⚪Next sentence prediction
59. 2020.02.10
Takuya KOUMURA
p. 59
BERT: Bidirectional Encoder Representations from Transformers
⚫Fine-tuning for various task
⚫Performance
⚪State-of-the-art on 11 tasks
60. 2020.02.10
Takuya KOUMURA
p. 60
Methods
⚫Brain activities
⚪fMRI (Pereira, F. et al. Toward a universal decoder of
linguistic meaning from brain activation. Nat. Commun. 9,
(2018).)
⚫Evaluation
⚪Mean squared error
⚪Average ranking
