The document provides an overview of deep learning concepts and techniques for natural language processing tasks. It includes the following:
1. A schedule for a deep learning workshop covering fundamentals of deep learning for machine translation, word embeddings, neural language models, and neural machine translation.
2. Descriptions of neural networks, activation functions, backpropagation, and word embeddings.
3. Details about feedforward neural network language models, recurrent neural network language models, and how they are applied to tasks like language modeling and machine translation.
4. An explanation of attention-based encoder-decoder models for neural machine translation.
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train.
Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.0 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
Slides by Víctor Garcia about the paper:
Reed, Scott, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. "Generative adversarial text to image synthesis." ICML 2016.
A Review of Deep Contextualized Word Representations (Peters+, 2018)Shuntaro Yada
A brief review of the paper:
Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. In NAACL-HLT (pp. 2227–2237)
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train.
Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.0 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
Slides by Víctor Garcia about the paper:
Reed, Scott, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. "Generative adversarial text to image synthesis." ICML 2016.
A Review of Deep Contextualized Word Representations (Peters+, 2018)Shuntaro Yada
A brief review of the paper:
Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. In NAACL-HLT (pp. 2227–2237)
GPT-2: Language Models are Unsupervised Multitask LearnersYoung Seok Kim
Review of paper
Language Models are Unsupervised Multitask Learners
(GPT-2)
by Alec Radford et al.
Paper link: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
YouTube presentation: https://youtu.be/f5zULULWUwM
(Slides are written in English, but the presentation is done in Korean)
https://telecombcn-dl.github.io/2017-dlsl/
Winter School on Deep Learning for Speech and Language. UPC BarcelonaTech ETSETB TelecomBCN.
The aim of this course is to train students in methods of deep learning for speech and language. Recurrent Neural Networks (RNN) will be presented and analyzed in detail to understand the potential of these state of the art tools for time series processing. Engineering tips and scalability issues will be addressed to solve tasks such as machine translation, speech recognition, speech synthesis or question answering. Hands-on sessions will provide development skills so that attendees can become competent in contemporary data analytics tools.
Word embedding, Vector space model, language modelling, Neural language model, Word2Vec, GloVe, Fasttext, ELMo, BERT, distilBER, roBERTa, sBERT, Transformer, Attention
This is a presentation I gave as a short overview of LSTMs. The slides are accompanied by two examples which apply LSTMs to Time Series data. Examples were implemented using Keras. See links in slide pack.
Introduction For seq2seq(sequence to sequence) and RNNHye-min Ahn
This is my slides for introducing sequence to sequence model and Recurrent Neural Network(RNN) to my laboratory colleagues.
Hyemin Ahn, @CPSLAB, Seoul National University (SNU)
Over the last two years, the field of Natural Language Processing (NLP) has witnessed the emergence of transfer learning methods and architectures which significantly improved upon the state-of-the-art on pretty much every NLP tasks.
The wide availability and ease of integration of these transfer learning models are strong indicators that these methods will become a common tool in the NLP landscape as well as a major research direction.
In this talk, I'll present a quick overview of modern transfer learning methods in NLP and review examples and case studies on how these models can be integrated and adapted in downstream NLP tasks, focusing on open-source solutions.
Website: https://fwdays.com/event/data-science-fwdays-2019/review/transfer-learning-in-nlp
An introduction to the Transformers architecture and BERTSuman Debnath
The transformer is one of the most popular state-of-the-art deep (SOTA) learning architectures that is mostly used for natural language processing (NLP) tasks. Ever since the advent of the transformer, it has replaced RNN and LSTM for various tasks. The transformer also created a major breakthrough in the field of NLP and also paved the way for new revolutionary architectures such as BERT.
最近のNLP×DeepLearningのベースになっている"Transformer"について、研究室の勉強会用に作成した資料です。参考資料の引用など正確を期したつもりですが、誤りがあれば指摘お願い致します。
This is a material for the lab seminar about "Transformer", which is the base of recent NLP x Deep Learning research.
Recurrent Neural Networks have shown to be very powerful models as they can propagate context over several time steps. Due to this they can be applied effectively for addressing several problems in Natural Language Processing, such as Language Modelling, Tagging problems, Speech Recognition etc. In this presentation we introduce the basic RNN model and discuss the vanishing gradient problem. We describe LSTM (Long Short Term Memory) and Gated Recurrent Units (GRU). We also discuss Bidirectional RNN with an example. RNN architectures can be considered as deep learning systems where the number of time steps can be considered as the depth of the network. It is also possible to build the RNN with multiple hidden layers, each having recurrent connections from the previous time steps that represent the abstraction both in time and space.
It’s long ago, approx. 30 years, since AI was not only a topic for Science-Fiction writers, but also a major research field surrounded with huge hopes and investments. But the over-inflated expectations ended in a subsequent crash and followed by a period of absent funding and interest – the so-called AI winter. However, the last 3 years changed everything – again. Deep learning, a machine learning technique inspired by the human brain, successfully crushed one benchmark after another and tech companies, like Google, Facebook and Microsoft, started to invest billions in AI research. “The pace of progress in artificial general intelligence is incredible fast” (Elon Musk – CEO Tesla & SpaceX) leading to an AI that “would be either the best or the worst thing ever to happen to humanity” (Stephen Hawking – Physicist).
What sparked this new Hype? How is Deep Learning different from previous approaches? Are the advancing AI technologies really a threat for humanity? Let’s look behind the curtain and unravel the reality. This talk will explore why Sundar Pichai (CEO Google) recently announced that “machine learning is a core transformative way by which Google is rethinking everything they are doing” and explain why "Deep Learning is probably one of the most exciting things that is happening in the computer industry” (Jen-Hsun Huang – CEO NVIDIA).
Either a new AI “winter is coming” (Ned Stark – House Stark) or this new wave of innovation might turn out as the “last invention humans ever need to make” (Nick Bostrom – AI Philosoph). Or maybe it’s just another great technology helping humans to achieve more.
While academic research is more and more focusing on integration of deep learning approaches for machine translation, also called Neural Machine Translation, and shows promising and exciting results – the resulting systems still have important pragmatic limitations compared to the current generation of translation engine. We will be discussing how SYSTRAN is integrating these new techniques into production systems, the results and benefits for the end users, and our vision for the next versions.
GPT-2: Language Models are Unsupervised Multitask LearnersYoung Seok Kim
Review of paper
Language Models are Unsupervised Multitask Learners
(GPT-2)
by Alec Radford et al.
Paper link: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
YouTube presentation: https://youtu.be/f5zULULWUwM
(Slides are written in English, but the presentation is done in Korean)
https://telecombcn-dl.github.io/2017-dlsl/
Winter School on Deep Learning for Speech and Language. UPC BarcelonaTech ETSETB TelecomBCN.
The aim of this course is to train students in methods of deep learning for speech and language. Recurrent Neural Networks (RNN) will be presented and analyzed in detail to understand the potential of these state of the art tools for time series processing. Engineering tips and scalability issues will be addressed to solve tasks such as machine translation, speech recognition, speech synthesis or question answering. Hands-on sessions will provide development skills so that attendees can become competent in contemporary data analytics tools.
Word embedding, Vector space model, language modelling, Neural language model, Word2Vec, GloVe, Fasttext, ELMo, BERT, distilBER, roBERTa, sBERT, Transformer, Attention
This is a presentation I gave as a short overview of LSTMs. The slides are accompanied by two examples which apply LSTMs to Time Series data. Examples were implemented using Keras. See links in slide pack.
Introduction For seq2seq(sequence to sequence) and RNNHye-min Ahn
This is my slides for introducing sequence to sequence model and Recurrent Neural Network(RNN) to my laboratory colleagues.
Hyemin Ahn, @CPSLAB, Seoul National University (SNU)
Over the last two years, the field of Natural Language Processing (NLP) has witnessed the emergence of transfer learning methods and architectures which significantly improved upon the state-of-the-art on pretty much every NLP tasks.
The wide availability and ease of integration of these transfer learning models are strong indicators that these methods will become a common tool in the NLP landscape as well as a major research direction.
In this talk, I'll present a quick overview of modern transfer learning methods in NLP and review examples and case studies on how these models can be integrated and adapted in downstream NLP tasks, focusing on open-source solutions.
Website: https://fwdays.com/event/data-science-fwdays-2019/review/transfer-learning-in-nlp
An introduction to the Transformers architecture and BERTSuman Debnath
The transformer is one of the most popular state-of-the-art deep (SOTA) learning architectures that is mostly used for natural language processing (NLP) tasks. Ever since the advent of the transformer, it has replaced RNN and LSTM for various tasks. The transformer also created a major breakthrough in the field of NLP and also paved the way for new revolutionary architectures such as BERT.
最近のNLP×DeepLearningのベースになっている"Transformer"について、研究室の勉強会用に作成した資料です。参考資料の引用など正確を期したつもりですが、誤りがあれば指摘お願い致します。
This is a material for the lab seminar about "Transformer", which is the base of recent NLP x Deep Learning research.
Recurrent Neural Networks have shown to be very powerful models as they can propagate context over several time steps. Due to this they can be applied effectively for addressing several problems in Natural Language Processing, such as Language Modelling, Tagging problems, Speech Recognition etc. In this presentation we introduce the basic RNN model and discuss the vanishing gradient problem. We describe LSTM (Long Short Term Memory) and Gated Recurrent Units (GRU). We also discuss Bidirectional RNN with an example. RNN architectures can be considered as deep learning systems where the number of time steps can be considered as the depth of the network. It is also possible to build the RNN with multiple hidden layers, each having recurrent connections from the previous time steps that represent the abstraction both in time and space.
It’s long ago, approx. 30 years, since AI was not only a topic for Science-Fiction writers, but also a major research field surrounded with huge hopes and investments. But the over-inflated expectations ended in a subsequent crash and followed by a period of absent funding and interest – the so-called AI winter. However, the last 3 years changed everything – again. Deep learning, a machine learning technique inspired by the human brain, successfully crushed one benchmark after another and tech companies, like Google, Facebook and Microsoft, started to invest billions in AI research. “The pace of progress in artificial general intelligence is incredible fast” (Elon Musk – CEO Tesla & SpaceX) leading to an AI that “would be either the best or the worst thing ever to happen to humanity” (Stephen Hawking – Physicist).
What sparked this new Hype? How is Deep Learning different from previous approaches? Are the advancing AI technologies really a threat for humanity? Let’s look behind the curtain and unravel the reality. This talk will explore why Sundar Pichai (CEO Google) recently announced that “machine learning is a core transformative way by which Google is rethinking everything they are doing” and explain why "Deep Learning is probably one of the most exciting things that is happening in the computer industry” (Jen-Hsun Huang – CEO NVIDIA).
Either a new AI “winter is coming” (Ned Stark – House Stark) or this new wave of innovation might turn out as the “last invention humans ever need to make” (Nick Bostrom – AI Philosoph). Or maybe it’s just another great technology helping humans to achieve more.
While academic research is more and more focusing on integration of deep learning approaches for machine translation, also called Neural Machine Translation, and shows promising and exciting results – the resulting systems still have important pragmatic limitations compared to the current generation of translation engine. We will be discussing how SYSTRAN is integrating these new techniques into production systems, the results and benefits for the end users, and our vision for the next versions.
This was a pitch for Iconic's neural machine translation technology given at the TAUS Annual Conference in Portland, Oregan on October 24th, 2016.
There has been a lot of talk, and a lot of hype about neural machine translation in the press. But not a lot of practical application. Let's change the conversation
While academic research is more and more focusing on integration of deep learning approaches for machine translation, also called Neural Machine Translation, and shows promising and exciting results – the resulting systems still have important pragmatic limitations compared to the current generation of translation engine. We will be discussing how SYSTRAN is integrating these new techniques into production systems, the results and benefits for the end users, and our vision for the next versions.
http://imatge-upc.github.io/telecombcn-2016-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.
FaceBook のAIチームが研究の発表論文である "Memory networks"とその拡張である"Towards AI-complete question answering: A set of prerequisite toy tasks."を簡単に紹介します。
[1] Weston, J., Chopra, S., and Bordes, A. Memory networks. In International Conference on Learning Representations (ICLR), 2015a.
[2] Weston, J., Bordes, A., Chopra, S., and Mikolov, T. Towards AI-complete question answering: A set of prerequisite toy tasks. arXiv preprint: 1502.05698, 2015b.
In this presentation, I address two major questions:
1. What is Intelligent Agent Perception?
2. How do Intelligent Agents Perceive?
In addressing the meaning of agent perception, I highlight impediments to the perceptual process and the process of situation assessment.
In addressing how agents perceive, I highlight traditional approaches to robotic perception and then the next step after sensor input, which is how sensor data (vision sensor, tactile, olfactory, haptic, etc.) is interpreted. Methods for interpretation include solutions based on Bayes' Theorem, the underpinning of many robotics algorithms; Probabilistic Algorithms; and Artificial Neural Networks.
I also discuss a current system for robotic perception, designed to accommodate more robust and complex robotic needs: using sensors in tandem with machine learning. This method is closer to mimicking the human perceptual process than previous methods. I discuss some examples of this: 1) a study in which researchers used visual sensors and an artificial neural network (ANN) for robotic perception; 2) a study in which researchers used haptic sensors and a classification algorithm, called a boosting algorithm for robotic perception; and 3) a study in which researchers used pressure sensors, an ANN, and intended to add pre-programmed models in order to facilitate robotic perception.
Este documento trata de mostrar el origen, ventajas, desventajas, que no es, tendencias, cifras de los MOOC debido a su gran viralización a nivel mundial
This presentation nlp classifiers, the different types of models tfidf, word2vec & DL models such as feed forward NN , CNN & siamese networks. Details on important metrics such as precision, recall AUC are also given
Deep Learning Architectures for NLP (Hungarian NLP Meetup 2016-09-07)Márton Miháltz
A brief survey of current deep learning/neural network methods currently used in NLP: recurrent networks (LSTM, GRU), recursive networks, convolutional networks, hybrid architectures, attention models. We will look at specific papers in the literature, targeting sentiment analysis, text classification and other tasks.
From Hours to Minutes: The Journey of Optimizing Mask-RCNN and BERT Using MXNetEric Haibin Lin
Training large deep learning models like Mask R-CNN and BERT takes lots of time and compute resources. Using MXNet, the Amazon Web Services deep learning framework team has been working with NVIDIA to optimize many different areas to cut the training time from hours to minutes.
This slides explains how Convolution Neural Networks can be coded using Google TensorFlow.
Video available at : https://www.youtube.com/watch?v=EoysuTMmmMc
Much of data is sequential – think speech, text, DNA, stock prices, financial transactions and customer action histories. Modern methods for modelling sequence data are often deep learning-based, composed of either recurrent neural networks (RNNs) or attention-based Transformers. A tremendous amount of research progress has recently been made in sequence modelling, particularly in the application to NLP problems. However, the inner workings of these sequence models can be difficult to dissect and intuitively understand.
This presentation/tutorial will start from the basics and gradually build upon concepts in order to impart an understanding of the inner mechanics of sequence models – why do we need specific architectures for sequences at all, when you could use standard feed-forward networks? How do RNNs actually handle sequential information, and why do LSTM units help longer-term remembering of information? How can Transformers do such a good job at modelling sequences without any recurrence or convolutions?
In the practical portion of this tutorial, attendees will learn how to build their own LSTM-based language model in Keras. A few other use cases of deep learning-based sequence modelling will be discussed – including sentiment analysis (prediction of the emotional valence of a piece of text) and machine translation (automatic translation between different languages).
The goals of this presentation are to provide an overview of popular sequence-based problems, impart an intuition for how the most commonly-used sequence models work under the hood, and show that quite similar architectures are used to solve sequence-based problems across many domains.
Dr. Erin LeDell, Machine Learning Scientist, H2O.ai at MLconf SEA - 5/20/16MLconf
Multi-algorithm Ensemble Learning at Scale: Software, Hardware and Algorithmic Approaches: Multi-algorithm ensemble machine learning methods are often used when the true prediction function is not easily approximated by a single algorithm. The Super Learner algorithm, also known as stacking, combines multiple, typically diverse, base learning algorithms into a single, powerful prediction function through a secondary learning process called metalearning. Although ensemble methods offer superior performance over their singleton counterparts, there is an implicit computational cost to ensembles, as it requires training and cross-validating multiple base learning algorithms.
We will demonstrate a variety of software- and hardware-based approaches that lead to more scalable ensemble learning software, including a highly scalable implementation of stacking called “H2O Ensemble”, built on top of the open source, distributed machine learning platform, H2O. H2O Ensemble scales across multi-node clusters and allows the user to create ensembles of deep neural networks, Gradient Boosting Machines, Random Forest, and others. As for algorithm-based approaches, we will present two algorithmic modifications to the original stacking algorithm that further reduce computation time — Subsemble algorithm and the Online Super Learner algorithm. This talk will also include benchmarks of the implementations of these new stacking variants.
Building a Neural Machine Translation System From ScratchNatasha Latysheva
Human languages are complex, diverse and riddled with exceptions – translating between different languages is therefore a highly challenging technical problem. Deep learning approaches have proved powerful in modelling the intricacies of language, and have surpassed all statistics-based methods for automated translation. This session begins with an introduction to the problem of machine translation and discusses the two dominant neural architectures for solving it – recurrent neural networks and transformers. A practical overview of the workflow involved in training, optimising and adapting a competitive neural machine translation system is provided. Attendees will gain an understanding of the internal workings and capabilities of state-of-the-art systems for automatic translation, as well as an appreciation of the key challenges and open problems in the field.
Erin LeDell, Machine Learning Scientist, H2O.ai at MLconf ATL 2016MLconf
Multi-algorithm Ensemble Learning at Scale: Software, Hardware and Algorithmic Approaches: Multi-algorithm ensemble machine learning methods are often used when the true prediction function is not easily approximated by a single algorithm. The Super Learner algorithm, also known as stacking, combines multiple, typically diverse, base learning algorithms into a single, powerful prediction function through a secondary learning process called metalearning. Although ensemble methods offer superior performance over their singleton counterparts, there is an implicit computational cost to ensembles, as it requires training and cross-validating multiple base learning algorithms.
We will demonstrate a variety of software- and hardware-based approaches that lead to more scalable ensemble learning software, including a highly scalable implementation of stacking called “H2O Ensemble”, built on top of the open source, distributed machine learning platform, H2O. H2O Ensemble scales across multi-node clusters and allows the user to create ensembles of deep neural networks, Gradient Boosting Machines, Random Forest, and others. As for algorithm-based approaches, we will present two algorithmic modifications to the original stacking algorithm that further reduce computation time — Subsemble algorithm and the Online Super Learner algorithm. This talk will also include benchmarks of the implementations of these new stacking variants.
Similar to Deep Learning for Machine Translation (20)
How Masterly Are People at Playing with Their Vocabulary?Matīss
In this paper, we describe adaptation of a simple word guessing game that occupied the hearts and minds of people around the world. There are versions for all three Baltic countries and even several versions of each. We specifically pay attention to the Latvian version and look into how people form their guesses given any already uncovered hints. The paper analyses guess patterns, easy and difficult word characteristics, and player behaviour and response.
Processing of multi-word expressions (MWEs) is a known problem for any natural language processing task. Even neural machine translation (NMT) struggles to overcome it. This paper presents results of experiments on investigating NMT attention allocation to the MWEs and improving automated translation of
sentences that contain MWEs in English -> Latvian and English -> Czech NMT systems. Two improvement strategies were
explored - (1) bilingual pairs of automatically extracted MWE candidates were added to the parallel corpus used to train the NMT system, and (2) full sentences containing the automatically extracted MWE candidates were added to the parallel corpus. Both approaches allowed to increase automated evaluation
results. The best result - 0.99 BLEU point increase - has been reached with the first approach, while with the second approach minimal improvements achieved. We also provide open-source software and tools used for MWE extraction and alignment inspection.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
4. Schedule
• Kevin Duh - Fundamentals for DL4MT I & II
• Lab 1 - Prep and setup. Compare logistic regression, MLP,
and stacked auto-encoders on data
• Lab 2 - word embedding (SGNS), visualization.
• Hermann Ney - Neural LMs and TMs for SMT I & II
• Lab 1 - Rescore n-best lists using RNN LM
• Lab 2 - n-best rescoring using uni/bidirectional translation and joint models
• Kyunghyun Cho - Neural MT I & II
• Lab 1 - Data Preparation: Basic preprocessing; Encoder-Decoder with Theano
(without attention)
• Lab 2 - Attention-based Encoder-Decoder with Theano
5. Neural Networks (NN) – brief introduction
• An NN consists of:
• Multiple layers (an input layer, zero
or more hidden layers, and an
output layer) that consist of a set of
neurones (xi, hj, and y)
• Interconnections between nodes of
different layers that have weights
assigned (wij and wj)
• Activation functions for each
neurone that convert weighted
input of neurons into output values
• Deep neural networks –NNs with
one or more hidden layers
6. Activation
functions
• Many different
types
• Most common
functions in NLP –
logistic sigmoid
and hyperbolic
tangent (because
of their non-
linearity)
http://blog.sciencenet.cn/blog-457187-878461.html
7. Training
• Neural networks are trained
with the backpropagation
algorithm
• Aim – reduce cost/error of the
model by iteratively performing a
forward pass, calculating the
error, and adjusting weights
based on the “direction” (read -
derivative) of the error. The error
is “backpropagated” from the
output layer back to the input
layer
• A long description with a lot of
theory:
http://neuralnetworksanddeep
learning.com/chap2.html
8. Word Embeddings
• A continuous space representation of words
• Multi-dimensional vectors
• Decimal values
• By-product of neural networks where words are vectorised
• Skip-gram models
• neural network language models
• neural machine translation models
• etc.
9. Word Embeddings –
Skip-Gram model
• Input – word (one of N vector)
• Output – context of the word
• Embeddings – the trained weight
matrix W between the input layer
and the hidden layer
• Each row represents the embedding
of a single word
• Implementation: word2vec
http://alexminnaar.com/word2vec-tutorial-part-i-the-skip-gram-model.html
10. Word embeddings are different
• The distributional representation of words has to be trained for specific tasks
(e.g., Skip-gram word embeddings are not good for translation)
• Similar words (according to cosine similarity) of the given words using different
word embedding models
http://arxiv.org/pdf/1412.6448.pdf
11. Feedforward Neural Net Language Model (NNLM)
• Y. Bengio, R.
Ducharme, P. Vincent.
A neural probabilistic
language model.
Journal of Machine
Learning Research,
3:1137-1155, 2003.
12. Feedforward Neural Net Language Model (NNLM)
• The same,
but with a
simpler
figure
/ http://www.cs.cmu.edu/~mfaruqui/talks/nn-clab.pdf /
13. What NNLMs are (supposedly) good at
(… what n-gram models never will)?
14. Language Modelling and Machine
Translation using Neural Networks
Hermann Ney
http://ej.uz/NNLM
15. Language Modelling
• Conventional Language Modelling
• Measure the quality of an LM with perplexity
• Problem: most of the events are never seen in training data
16. Language Modelling Using Neural Networks
• non ANN: count models (Markov chain):
• limited history of predecessor words
• smooth relative frequencies
• feedforward multi-layer perceptron (FF MLP):
• limited history too
• use predecessor words as input to MLP
• recurrent neural networks (RNN):
• advantage: unlimited history
19. Experiments Results
with JRC-ACQUIS EN-LV
Approach Perplexity CPU time Size
5-gram count language model 48.0376 5 minutes + 2 minutes (binarize) 1118MB
4-gram Feedforward Neural Net
Language Model with 2 layers
• 1000 wrord classes;
• batch-size 64;
• learning-rate 5e-3;
• 200 nodes per input word, and a
subsequent layer of 200 nodes
with a sigmoid activation
function.
126.9841 1 week 43MB
20.
21. Practicalities of ANN LM Training
(Implementation and Software)
• no regularization, no momentum term, no drop-out (so far!)
• no pre-training (so far!)
• vocabulary reduction: remove singletons, or keep most frequent words
• random initialization of weights: Gaussian of mean 0, variance 0.01
• training criterion: cross-entropy (perplexity)
• stopping: cross-validation, perplexity on a development text
• initial learning rate: typically between 1 · e−3 and 1 · e−2
• learning rate: halved when the dev perplexity is worse than the best of
previous epochs
• use of mini-batches: 4 to 64
• low level implementation in C++
• GPUs (typically) not used for the results presented in this talk
22. Language Modelling Using Neural Networks lab
Exercice Sheet: http://ej.uz/NNLMlab
https://www-i6.informatik.rwth-aachen.de/web/Software/rwthlm.php
• Data preparation
• Training on small data
• Tuning of hyper-parameters
• Modifying the network architecture
28. Experiments Results
Syntax-based Multi-System Hybrid Translator + NNLM on JRC-ACQUIS EN-LV
Approach BLEU
MHyT with 5-gram count language model 22.69
SyMHyT with 5-gram count language model 24.72
SyMHyT with 4-gram Feedforward Neural Net Language Model with 2 layers 23.71
29. Machine Translation using Neural Networks lab
• Exercice Sheet: http://ej.uz/NNMTlab
• Part 1: N-best Reranking using Neural Network Language Models
• Obtain the new 1-best hypotheses
• Measure the translation quality
• Do reranking and compare to the results obtained before reranking.
• Part 2: Neural Network Translation Models
• Train a unidirectional translation model
• Train a unidirectional joint model
• Train a bidirectional translation model
• Train a bidirectional joint model
• Try to obtain better perplexity values by changing the batch size and learning rate
• Part 3: N-best Reranking using Neural Network Translation Models
• Apply rescoring using each of the unidirectional and bidirectional translation and joint models
• Optimize the model weights with MERT to achieve the best BLEU score on the dev dataset
• Evaluate the translation hypotheses for each of the rescoring experiments
30. Neural Machine Translation
• Encoder-decoder model
https://github.com/nyu-dl/NLP_DL_Lecture_Note/blob/master/lecture_note.pdf
34. Neural Machine Translation
• Attention-based models
• Due to its sequential nature, a
recurrent neural network
tends to remember recent
symbols better
• The attention mechanism
allows to focus at each time
step at the relevant symbols by
selecting the appropriate
vector that summarises the
input sentence
http://devblogs.nvidia.com/parallelforall/introduction-neural-machine-translation-gpus-part-3/
35. Attention-based encoder-decoder NMT
• English-Latvian
• Vocabulary – 30K
• Embedding dimensions – 400
• Hidden layer dimensions – 1,024
• Batch size - 14
• Training corpus – DGT-TM (2,401,815 unique sentences)
• Trainined on an NVIDIA GeForce GTX 960 (2GB) GPU
• Training time - ~4 days and 2 hours (crashed due to out of memory exception)
• But ... Luckily it saves models iteratively
• During training uses ~40GB of virtual memory
• Translation time of 512 sentence test set
(however, it includes also the model loading time)
• NMT: 19 minutes and 3 seconds (translation with CPU on 6 cores)
• LetsMT: 1 minute and 39 seconds (translation with CPU on 1 core)
https://github.com/kyunghyuncho/dl4mt-material/tree/master/session2
36. Attention-based encoder-decoder NMT
• Comparison with LetsMT (LetsMT – 13.93 BLEU, NMT – 12.42 BLEU)
• Translations more fluent (even if not always correct «according to reference»)
43. Attention-based encoder-decoder NMT
• Model trained on 50K vocabulary and a bath size of 12
• After 300,000 updates (or 3,600,000 observed sentences)
• I.e., model is not yet fully trained
• Results:
• LetsMT – 13.93 BLEU, NMT – 12.48 BLEU(+0.06)
• Not good, but it may improve since the model has not finished training…
• After 520,000 updates (6,240,000 observed sentences)
• LetsMT – 13.93 BLEU, NMT – 11.88 BLEU(-0.54)
44. Attention-based encoder-decoder NMT
• Lessons learned (from the tiny but long experiments):
• You need to have a «good» GPU (6GB GDDR may not be enough for systems of a decent
vocabulary size)
• A 2GB card will not allow building models with a vocabulary that is larger than 30-50K
• The «good» GPUs are expensive (>1K€)
• Only Nvidia GPUs are currently usable (the existing libraries are built/tuned for CUDA), OpenCL is an
alternative that is under-supported
• If training with a GPU takes up to a week, training with a CPU is a no go
• 30K is miles away from a decent vocabulary
• You need to have means to handle unknown words
• From the translation quality positive tendencies are evident, but an experiment with a
more decent data set and a larger vocabulary is necessary to make more justified
judgements
47. Theano
http://deeplearning.net/software/theano/
• Built for python
• Tight integration with NumPy – Use numpy.ndarray in Theano-compiled functions.
• Transparent use of a GPU – Perform data-intensive calculations up to 140x faster than with CPU.(float32 only)
• Efficient symbolic differentiation – Theano does your derivatives for function with one or many inputs.
• Speed and stability optimizations – Get the right answer for log(1+x) even when x is really tiny.
• Dynamic C code generation – Evaluate expressions faster.
• Extensive unit-testing and self-verification – Detect and diagnose many types of mistake.
• The EN-LV NMT model was trained using Theano
• Speed comparison of different NN libraries: https://github.com/soumith/convnet-benchmarks