Nature Language Processing for proteins by Amélie Héliou, Software Engineer @ Google Research

•

0 likes•111 views

Paris Women in Machine Learning and Data Science

"Nature Language Processing for proteins" by Amélie Héliou, Software Engineer @ Google Research Abstract: Over the past few months, Large Language Models have become very popular. We'll see how a simple LLM works, from input sentence to prediction. I'll then present an application of LLM to protein name prediction. Twitter: @Amelie_hel

Engineering

NLP for
Proteins
Encoder-Decoders
Transformers
Application to Proteins
Language representation

Application to
proteins
Predict proteins name
MRYRLAWLLHPALPSTFRSVLGARLPPPERLCGFQKKTYSKMNNP
AIKRIGNHITKSPEDKREYRGLELANGIKVLLISD …
P14735 Insulin-degrading enzyme
https://www.uniprot.org/uniprotkb/P14735/entry

Application to
proteins
Predict proteins name
● Names give an idea about the function of
proteins.
● Names are given by experts or rule based
annotations.
● ~ 50 millions of proteins without a name.
(20%).

Language
representation
Tokenization
Vocabulary
Embeddings
Input sentence:
Women are powerful.
Tokenization:
Women are powerful.
Vocabulary:
10 123 25
Embeddings:

Encoder
Example: BERT
Learnt token embeddings.
Can be trained on masked
tasks.
Input sentence:
Women are powerful.
Encoder

Decoder
Example: PaLM
Next token prediction
tasks.
Input sentence:
Women are powerful.
Decoder
Next token probabilities.

Encoder -
Decoder
Input sentence:
Women are powerful.
Encoder Decoder
Next token probabilities.

Autoregressive
tasks
Predict tokens one by
one.
Input sentence:
Women are powerful.
Encoder Decoder
Les
<s>

Autoregressive
tasks
Predict tokens one by
one.
Input sentence:
Women are powerful.
Encoder Decoder
femmes
<s> Les

Autoregressive
tasks
Predict tokens one by
one.
Input sentence:
Women are powerful.
Encoder Decoder
sont
<s> Les femmes

Autoregressive
tasks
Predict tokens one by
one.
Input sentence:
Women are powerful.
Encoder Decoder
puissantes
<s> Les femmes sont

Autoregressive
tasks
Predict tokens one by
one.
Input sentence:
Women are powerful.
Encoder Decoder
<eos>
<s> Les femmes sont puissantes

Transformers
Self-Attention
Nice explanation:
https://jalammar.github.io/illust
rated-transformer/

Transformers
Self-Attention
Nice explanation:
https://jalammar.github.io/illust
rated-transformer/
Women
are
powerful
Women are powerful
keys = XWk
queries
=
XWq

Transformers
Self-Attention
Nice explanation:
https://jalammar.github.io/illust
rated-transformer/
Women
are
powerful
Women are powerful
Women = Women x + are x
are = Women x + are x
powerful = Women x + powerful x
queries
=
XWq
keys = XWk
values = XWv

Transformers
Position
feed-forward
● Position is added to the initial
embedding.
● After attention:
○ Residual connection
(input+output)
○ Feed-forward

Transformers
Overall architecture.
Attention is all you
need, Vaswani et al.
2017

Application to
proteins
Predict proteins name
Input sentence:
MRYRLAWLLHPALPSTFRSVLGARLPPPERL
Encoder Decoder
Insulin
<s>

Application to
proteins
Predict proteins name
Input sentence:
MRYRLAWLLHPALPSTFRSVLGARLPPPERL
Encoder Decoder
degrading
<s> Insulin

Application to
proteins
Predict proteins name
Input sentence:
MRYRLAWLLHPALPSTFRSVLGARLPPPERL
Encoder Decoder
enzyme
<s> Insulin degrading

Application to
proteins
Predict proteins name
Input sentence:
MRYRLAWLLHPALPSTFRSVLGARLPPPERL
Encoder Decoder
<eos>
<s> Insulin degrading enzyme

As electricity is difficult to store, it is crucial to strictly maintain the balance between production and consumption. The integration of intermittent renewable energies into the production mix has made the management of the balance more complex. However, access to near real-time data and communication with consumers via smart meters suggest demand response. Specifically, sending signals would encourage users to adjust their consumption according to the production of electricity. The algorithms used to select these signals must learn consumer reactions and optimize them while balancing exploration and exploitation. Various sequential or reinforcement learning approaches are being considered.

How and why AI should fight cybersexism, by Chloe Daudier

Paris Women in Machine Learning and Data Science

Anomaly detection and data imputation within time series

Paris Women in Machine Learning and Data Science

Managing international tech teams, by Natasha Dimban

Paris Women in Machine Learning and Data Science

Optimizing GenAI apps, by N. El Mawass and Maria Knorps

Paris Women in Machine Learning and Data Science

Nour and Maria present the work they did at Tweag, Modus Create innovation arm, where the GenAI team developed an evaluation framework for Retrieval-Augmented Generation (RAG) systems. RAG systems provide an easy and low-cost way to extend the knowledge of Large Language Models (LLMs) but measuring their performance is not an easy task. The presentation will review existing evaluation frameworks, ranging from those based on the traditional ML approach of using groundtruth datasets, including Tweag's, to those that use LLMs to compute evaluation metrics. It will also delve into the practical implementation of Tweag's chatbot over two distinct documents datasets and provide insights on chunking, embedding and how open source and commercial LLMs compare.

Perspectives, by M. Pannegeon

Paris Women in Machine Learning and Data Science

Evaluation strategies for dealing with partially labelled or unlabelled data

Paris Women in Machine Learning and Data Science

Combinatorial Optimisation with Policy Adaptation using latent Space Search, ...

Paris Women in Machine Learning and Data Science

Abstract: Who hasn't heard of the "Pilot Syndrome"? 85% of Data Science Pilots remain pilots and do not make it to the production stage. Let's build a production-ready and end-user-friendly Data Science application. 100% python and 100% open source. Phase 1 | Building the GUI: create an interactive and powerful interface in a few lines of code Phase 2 | Integrated back end: Manage your models and pipelines and create scenarios the smart way

Sandrine Henry presents the BechdelAI project

Paris Women in Machine Learning and Data Science

Anastasiia Tryputen_War in Ukraine or how extraordinary courage reshapes geop...

Paris Women in Machine Learning and Data Science

Khrystyna Grynko WiMLDS - From marketing to Tech.pdf

Paris Women in Machine Learning and Data Science

Iana Iatsun_ML in production_20Dec2022.pdf

Paris Women in Machine Learning and Data Science

41 WiMLDS Kyiv Paris Poznan.pdf

Paris Women in Machine Learning and Data Science

Emergency plan to secure winter: what are the measures set up by RTE?

Paris Women in Machine Learning and Data Science

"Emergency plan to secure winter: what are the measures set up by RTE?" by Sophie Diakhate, Ingénieure Génie électrique, Consultante en énergie et utilities at Yélé Abstract: The french electric system is currently going through an exceptional crisis, threatening the electric supplies for this winter, and potentially the next ones. As the guarantor of the balance between supply and demand, RTE must assume the security of supply security in France. They set up an emergency plan to secure winters. Yélé helps RTE to carry on that plan. We are going to see what are the primary measures proposed by RTE for the winter 2022-2023, and the options for individuals and the industry to reduce the risks of network load shedding.

New edge prediction and anomaly-detection in large computer networks

Paris Women in Machine Learning and Data Science

"New edge prediction and anomaly-detection in large computer networks" by Dr Silvia Metelli, Marie Skołodowska-Curie Individual Fellow Abstract : Monitoring computer network traffic in search for anomalous behaviour is both a challenging and important task for cyber-security. New edges, i.e. connections from a host or user to a computer that has not been connected to before, provide potentially strong statistical evidence for detecting anomalies and in rare cases might suggest the presence of intruders or malicious activity. In this talk, I will introduce a robust Bayesian model and anomaly detection method for simultaneously characterising network structure and modelling likely new edge formation in a large computer network graph. What constitutes normal behaviour for some hosts might be very unusual for some others and thus examining existing network structure (e.g. clusters of similar clients and servers) is key for accurately predicting likely future interactions. Finally, the model is used to construct an anomaly detection method, which successfully identifies some of the machines known to be compromised when demonstrated on real computer network authentication data.

transformers_multimodal_ehr.pdf

Paris Women in Machine Learning and Data Science

Interest has increased in the use of prognosis factors as a cursor for breast cancer personalized treatment. For clinicians, early detection of those factors can be helpful for a good management of the disease and for the choice of an efficient treatment. Moreover, it exists a huge amount of meaningful information in pathological reports, biological measurements and clinical information in a patient journey that remain unexploited. In that context, I propose to develop and apply novel machine learning techniques to predict cancer outcome such as recurrence or survival from multi-modal breast cancer patient data (including medical notes in natural languages and the outcome of various lab analyses). For that, I use a deep neural sequence transduction for electronic health records called BEHRT1. This model is inspired from one of the most powerful transformer-based architecture in Natural Language Processing: BERT2.

meetup_jussieu_sept2022.pdf

Paris Women in Machine Learning and Data Science

During the joint meetup with Duchess France and PyLadies Paris, Deborah Boyenval, PhD Student at Université Côté d'Azur presented a part of her PhD work: “Formal modeling of biological cyclic behavior with control points: the case of the cell cycle”. The main limitation of biologists rooted in an experimental practice is the ability to perform rigorous proofs in the absence of a language for formalizing the biological knowledge extracted from their experiments. Biologists have identified numerous biochemical and genetic mechanisms involved in physiological functions or diseases, but once this knowledge is linked together it remains extremely difficult and expensive to predict the impact of genetic mechanisms on physiological functions. Déborah will present to us her thesis, which focuses firstly on a reasonable mathematical specification of complex biological functions such as cell cycle checkpoints, which represent the main barrier against cancer. Secondly, she focused on the development of an automated proof method, using the mentioned tools, proving whether a set of genetic regulations is sufficient to generate cell cycle checkpoints.

“NLP and Computer Vision Applications in Consumer Feedback Analysis” by Olesi...

Paris Women in Machine Learning and Data Science

"Blood flow simulation for clinical applications" by Dr Irene Vignon-Clemente...

Paris Women in Machine Learning and Data Science

COST OF WAR - EMPLOYMENT IMPACT

Paris Women in Machine Learning and Data Science

"Cost of war - employment impact" by Anastasiia Tryputen, AI enthusiast, researcher, entrepreneur, founder of NeuroHarb and Data unBlocked companies Abstract: The talk would give a background of the Ukrainian talent market, historical trends, the success of Ukrainians inside and outside of the country. The next part of the talk will focus on the truth of russian full-scale war and what it means to families, companies, and talents. The last part of the talk will focus on the statistics around people fleeing war, specifically children and women, and suggest data science professionals and business owners to consider helping those who arrived to pick up new skills to enter the technical field. Finally, the closing part will focus on the gender gap in tech field and how we can contribute to changing this situation for all the women in EU and globally.

CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx

R&R Consult

CFD analysis is incredibly effective at solving mysteries and improving the performance of complex systems! Here's a great example: At a large natural gas-fired power plant, where they use waste heat to generate steam and energy, they were puzzled that their boiler wasn't producing as much steam as expected. R&R and Tetra Engineering Group Inc. were asked to solve the issue with reduced steam production. An inspection had shown that a significant amount of hot flue gas was bypassing the boiler tubes, where the heat was supposed to be transferred. R&R Consult conducted a CFD analysis, which revealed that 6.3% of the flue gas was bypassing the boiler tubes without transferring heat. The analysis also showed that the flue gas was instead being directed along the sides of the boiler and between the modules that were supposed to capture the heat. This was the cause of the reduced performance. Based on our results, Tetra Engineering installed covering plates to reduce the bypass flow. This improved the boiler's performance and increased electricity production. It is always satisfying when we can help solve complex challenges like this. Do your systems also need a check-up or optimization? Give us a call! Work done in cooperation with James Malloy and David Moelling from Tetra Engineering. More examples of our work https://www.r-r-consult.dk/en/cases-en/

Investor-Presentation-Q1FY2024 investor presentation document.pptx

AmarGB2

Recently uploaded

CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx

R&R Consult

Investor-Presentation-Q1FY2024 investor presentation document.pptx

AmarGB2

Cosmetic shop management system project report.pdf

Kamal Acharya

Buying new cosmetic products is difficult. It can even be scary for those who have sensitive skin and are prone to skin trouble. The information needed to alleviate this problem is on the back of each product, but it's thought to interpret those ingredient lists unless you have a background in chemistry. Instead of buying and hoping for the best, we can use data science to help us predict which products may be good fits for us. It includes various function programs to do the above mentioned tasks. Data file handling has been effectively used in the program. The automated cosmetic shop management system should deal with the automation of general workflow and administration process of the shop. The main processes of the system focus on customer's request where the system is able to search the most appropriate products and deliver it to the customers. It should help the employees to quickly identify the list of cosmetic product that have reached the minimum quantity and also keep a track of expired date for each cosmetic product. It should help the employees to find the rack number in which the product is placed.It is also Faster and more efficient way.

ethical hacking in wireless-hacking1.ppt

Jayaprasanna4

J.Yang, ICLR 2024, MLILAB, KAIST AI.pdf

MLILAB

Nuclear Power Economics and Structuring 2024

Massimo Talia

Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)

MdTanvirMahtab2

ethical hacking-mobile hacking methods.ppt

Jayaprasanna4

Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf

fxintegritypublishin

Advancements in technology unveil a myriad of electrical and electronic breakthroughs geared towards efficiently harnessing limited resources to meet human energy demands. The optimization of hybrid solar PV panels and pumped hydro energy supply systems plays a pivotal role in utilizing natural resources effectively. This initiative not only benefits humanity but also fosters environmental sustainability. The study investigated the design optimization of these hybrid systems, focusing on understanding solar radiation patterns, identifying geographical influences on solar radiation, formulating a mathematical model for system optimization, and determining the optimal configuration of PV panels and pumped hydro storage. Through a comparative analysis approach and eight weeks of data collection, the study addressed key research questions related to solar radiation patterns and optimal system design. The findings highlighted regions with heightened solar radiation levels, showcasing substantial potential for power generation and emphasizing the system's efficiency. Optimizing system design significantly boosted power generation, promoted renewable energy utilization, and enhanced energy storage capacity. The study underscored the benefits of optimizing hybrid solar PV panels and pumped hydro energy supply systems for sustainable energy usage. Optimizing the design of solar PV panels and pumped hydro energy supply systems as examined across diverse climatic conditions in a developing country, not only enhances power generation but also improves the integration of renewable energy sources and boosts energy storage capacities, particularly beneficial for less economically prosperous regions. Additionally, the study provides valuable insights for advancing energy research in economically viable areas. Recommendations included conducting site-specific assessments, utilizing advanced modeling tools, implementing regular maintenance protocols, and enhancing communication among system components.

Top 10 Oil and Gas Projects in Saudi Arabia 2024.pdf

Teleport Manpower Consultant

space technology lecture notes on satellite

ongomchris

road safety engineering r s e unit 3.pdf

VENKATESHvenky89705

AP LAB PPT.pdf ap lab ppt no title specific

BrazilAccount1

power quality voltage fluctuation UNIT - I.pptx

ViniHema

Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...

Dr.Costas Sachpazis

Terzaghi's soil bearing capacity theory, developed by Karl Terzaghi, is a fundamental principle in geotechnical engineering used to determine the bearing capacity of shallow foundations. This theory provides a method to calculate the ultimate bearing capacity of soil, which is the maximum load per unit area that the soil can support without undergoing shear failure. The Calculation HTML Code included.

CME397 Surface Engineering- Professional Elective

karthi keyan

Student information management system project report ii.pdf

Kamal Acharya

RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...

thanhdowork

Design and Analysis of Algorithms-DP,Backtracking,Graphs,B&B

Sreedhar Chowdam

Water Industry Process Automation and Control Monthly - May 2024.pdf

Water Industry Process Automation & Control

Welcome to WIPAC Monthly the magazine brought to you by the LinkedIn Group Water Industry Process Automation & Control. In this month's edition, along with this month's industry news to celebrate the 13 years since the group was created we have articles including A case study of the used of Advanced Process Control at the Wastewater Treatment works at Lleida in Spain A look back on an article on smart wastewater networks in order to see how the industry has measured up in the interim around the adoption of Digital Transformation in the Water Industry.

Recently uploaded (20)

CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx

Investor-Presentation-Q1FY2024 investor presentation document.pptx

Cosmetic shop management system project report.pdf

ethical hacking in wireless-hacking1.ppt

J.Yang, ICLR 2024, MLILAB, KAIST AI.pdf

Nuclear Power Economics and Structuring 2024

Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)

ethical hacking-mobile hacking methods.ppt

Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf

Top 10 Oil and Gas Projects in Saudi Arabia 2024.pdf

space technology lecture notes on satellite

road safety engineering r s e unit 3.pdf

AP LAB PPT.pdf ap lab ppt no title specific

power quality voltage fluctuation UNIT - I.pptx

Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...

CME397 Surface Engineering- Professional Elective

Student information management system project report ii.pdf

RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...

Design and Analysis of Algorithms-DP,Backtracking,Graphs,B&B

Water Industry Process Automation and Control Monthly - May 2024.pdf

Nature Language Processing for proteins by Amélie Héliou, Software Engineer @ Google Research

1. Amélie Héliou NLP for Proteins

2. NLP for Proteins Encoder-Decoders Transformers Application to Proteins Language representation

3. Application to proteins Predict proteins name MRYRLAWLLHPALPSTFRSVLGARLPPPERLCGFQKKTYSKMNNP AIKRIGNHITKSPEDKREYRGLELANGIKVLLISD … P14735 Insulin-degrading enzyme https://www.uniprot.org/uniprotkb/P14735/entry

4. Application to proteins Predict proteins name ● Names give an idea about the function of proteins. ● Names are given by experts or rule based annotations. ● ~ 50 millions of proteins without a name. (20%).

5. Language representation Tokenization Vocabulary Embeddings Input sentence: Women are powerful. Tokenization: Women are powerful. Vocabulary: 10 123 25 Embeddings:

6. Encoder Example: BERT Learnt token embeddings. Can be trained on masked tasks. Input sentence: Women are powerful. Encoder

7. Decoder Example: PaLM Next token prediction tasks. Input sentence: Women are powerful. Decoder Next token probabilities.

8. Encoder - Decoder Input sentence: Women are powerful. Encoder Decoder Next token probabilities.

9. Autoregressive tasks Predict tokens one by one. Input sentence: Women are powerful. Encoder Decoder Les <s>

10. Autoregressive tasks Predict tokens one by one. Input sentence: Women are powerful. Encoder Decoder femmes <s> Les

11. Autoregressive tasks Predict tokens one by one. Input sentence: Women are powerful. Encoder Decoder sont <s> Les femmes

12. Autoregressive tasks Predict tokens one by one. Input sentence: Women are powerful. Encoder Decoder puissantes <s> Les femmes sont

13. Autoregressive tasks Predict tokens one by one. Input sentence: Women are powerful. Encoder Decoder <eos> <s> Les femmes sont puissantes

14. Transformers Self-Attention Nice explanation: https://jalammar.github.io/illust rated-transformer/

15. Transformers Self-Attention Nice explanation: https://jalammar.github.io/illust rated-transformer/ Women are powerful Women are powerful keys = XWk queries = XWq

16. Transformers Self-Attention Nice explanation: https://jalammar.github.io/illust rated-transformer/ Women are powerful Women are powerful Women = Women x + are x are = Women x + are x powerful = Women x + powerful x queries = XWq keys = XWk values = XWv

17. Transformers Position feed-forward ● Position is added to the initial embedding. ● After attention: ○ Residual connection (input+output) ○ Feed-forward

18. Transformers Overall architecture. Attention is all you need, Vaswani et al. 2017

19. Application to proteins Predict proteins name Input sentence: MRYRLAWLLHPALPSTFRSVLGARLPPPERL Encoder Decoder Insulin <s>

20. Application to proteins Predict proteins name Input sentence: MRYRLAWLLHPALPSTFRSVLGARLPPPERL Encoder Decoder degrading <s> Insulin

21. Application to proteins Predict proteins name Input sentence: MRYRLAWLLHPALPSTFRSVLGARLPPPERL Encoder Decoder enzyme <s> Insulin degrading

22. Application to proteins Predict proteins name Input sentence: MRYRLAWLLHPALPSTFRSVLGARLPPPERL Encoder Decoder <eos> <s> Insulin degrading enzyme

23. Application to proteins

24. Thank you

Nature Language Processing for proteins by Amélie Héliou, Software Engineer @ Google Research

Recommended

Recommended

More Related Content

More from Paris Women in Machine Learning and Data Science

More from Paris Women in Machine Learning and Data Science (20)

Recently uploaded

Recently uploaded (20)

Nature Language Processing for proteins by Amélie Héliou, Software Engineer @ Google Research