A short and fast journey through some of the profiling options available in the Ruby 2.x world, including a look at flamegraphs and new ways of tracking memory usage in the MRI.
Simple, fast, and scalable torch7 tutorialJin-Hwa Kim
A tutorial based on basic information of Torch7. It covers installation, simple runable codes, tensor manipulations, sweep out key-packages and post-hoc audience q&a.
Distributed implementation of a lstm on spark and tensorflowEmanuel Di Nardo
Academic project based on developing a LSTM distributing it on Spark and using Tensorflow for numerical operations.
Source code: https://github.com/EmanuelOverflow/LSTM-TensorSpark
Profiling PyTorch for Efficiency & Sustainabilitygeetachauhan
From my talk at the Data & AI summit - latest update on the PyTorch Profiler and how you can use it for optimizations for efficiency. Talk also dives into the future and what we need to do together as an industry to move towards Sustainable AI
下記論文を扱った研究室内輪読用の資料です
This is slides for group reading in Lab.
Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau, "Operating Systems: Three Easy Pieces"
http://pages.cs.wisc.edu/~remzi/OSTEP/
Introduction to Deep Learning, Keras, and TensorFlowSri Ambati
This meetup was recorded in San Francisco on Jan 9, 2019.
Video recording of the session can be viewed here: https://youtu.be/yG1UJEzpJ64
Description:
This fast-paced session starts with a simple yet complete neural network (no frameworks), followed by an overview of activation functions, cost functions, backpropagation, and then a quick dive into CNNs. Next, we'll create a neural network using Keras, followed by an introduction to TensorFlow and TensorBoard. For best results, familiarity with basic vectors and matrices, inner (aka "dot") products of vectors, and rudimentary Python is definitely helpful. If time permits, we'll look at the UAT, CLT, and the Fixed Point Theorem. (Bonus points if you know Zorn's Lemma, the Well-Ordering Theorem, and the Axiom of Choice.)
Oswald's Bio:
Oswald Campesato is an education junkie: a former Ph.D. Candidate in Mathematics (ABD), with multiple Master's and 2 Bachelor's degrees. In a previous career, he worked in South America, Italy, and the French Riviera, which enabled him to travel to 70 countries throughout the world.
He has worked in American and Japanese corporations and start-ups, as C/C++ and Java developer to CTO. He works in the web and mobile space, conducts training sessions in Android, Java, Angular 2, and ReactJS, and he writes graphics code for fun. He's comfortable in four languages and aspires to become proficient in Japanese, ideally sometime in the next two decades. He enjoys collaborating with people who share his passion for learning the latest cool stuff, and he's currently working on his 15th book, which is about Angular 2.
A short and fast journey through some of the profiling options available in the Ruby 2.x world, including a look at flamegraphs and new ways of tracking memory usage in the MRI.
Simple, fast, and scalable torch7 tutorialJin-Hwa Kim
A tutorial based on basic information of Torch7. It covers installation, simple runable codes, tensor manipulations, sweep out key-packages and post-hoc audience q&a.
Distributed implementation of a lstm on spark and tensorflowEmanuel Di Nardo
Academic project based on developing a LSTM distributing it on Spark and using Tensorflow for numerical operations.
Source code: https://github.com/EmanuelOverflow/LSTM-TensorSpark
Profiling PyTorch for Efficiency & Sustainabilitygeetachauhan
From my talk at the Data & AI summit - latest update on the PyTorch Profiler and how you can use it for optimizations for efficiency. Talk also dives into the future and what we need to do together as an industry to move towards Sustainable AI
下記論文を扱った研究室内輪読用の資料です
This is slides for group reading in Lab.
Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau, "Operating Systems: Three Easy Pieces"
http://pages.cs.wisc.edu/~remzi/OSTEP/
Introduction to Deep Learning, Keras, and TensorFlowSri Ambati
This meetup was recorded in San Francisco on Jan 9, 2019.
Video recording of the session can be viewed here: https://youtu.be/yG1UJEzpJ64
Description:
This fast-paced session starts with a simple yet complete neural network (no frameworks), followed by an overview of activation functions, cost functions, backpropagation, and then a quick dive into CNNs. Next, we'll create a neural network using Keras, followed by an introduction to TensorFlow and TensorBoard. For best results, familiarity with basic vectors and matrices, inner (aka "dot") products of vectors, and rudimentary Python is definitely helpful. If time permits, we'll look at the UAT, CLT, and the Fixed Point Theorem. (Bonus points if you know Zorn's Lemma, the Well-Ordering Theorem, and the Axiom of Choice.)
Oswald's Bio:
Oswald Campesato is an education junkie: a former Ph.D. Candidate in Mathematics (ABD), with multiple Master's and 2 Bachelor's degrees. In a previous career, he worked in South America, Italy, and the French Riviera, which enabled him to travel to 70 countries throughout the world.
He has worked in American and Japanese corporations and start-ups, as C/C++ and Java developer to CTO. He works in the web and mobile space, conducts training sessions in Android, Java, Angular 2, and ReactJS, and he writes graphics code for fun. He's comfortable in four languages and aspires to become proficient in Japanese, ideally sometime in the next two decades. He enjoys collaborating with people who share his passion for learning the latest cool stuff, and he's currently working on his 15th book, which is about Angular 2.
Workshop about TensorFlow usage for AI Ukraine 2016. Brief tutorial with source code example. Described TensorFlow main ideas, terms, parameters. Example related with linear neuron model and learning using Adam optimization algorithm.
Rajat Monga at AI Frontiers: Deep Learning with TensorFlowAI Frontiers
In this talk at AI Frontiers Conference, Rajat Monga shares about TensorFlow that has enabled cutting-edge machine learning research at the top AI labs in the world. At the same time it has made the technology accessible to a large audience leading to some amazing uses. TensorFlow is used for classification, recommendation, text parsing, sentiment analysis and more. This talk goes over the design that makes it fast, flexible, and easy to use, and describe how we continue to make it better.
A quick review and demonstration on how to get started on parallel computing with R. Includes an example of SNOW cluster set up in the departmental lab.
SIMD machines — machines capable of evaluating the same instruction on several elements of data in parallel — are nowadays commonplace and diverse, be it in supercomputers, desktop computers or even mobile ones. Numerous tools and libraries can make use of that technology to speed up their computations, yet it could be argued that there is no library that provides a satisfying minimalistic, high-level and platform-agnostic interface for the C++ developer.
Oracle Solaris 11 is the first operating system engineered with cloud computing in mind. So what's new in Oracle Solaris 11, and how does that connect to the cloud? If you`re involved in Application Life-cycle Management, Configuration Management,
Cloud Deployment, Big Data Design and Application or Infrastructure Scaling - You will learn how to leverage the Solaris 11 technologies in order to build your Cloud infrastructure.
For more information see: http://www.oracle.com/technetwork/systems/hands-on-labs/hol-oracle-solaris-remote-lab-1894053.html
Oracle Solaris 11 as a BIG Data Platform Apache Hadoop Use CaseOrgad Kimchi
The following are benefits of using Oracle Solaris Zones for a Hadoop cluster:
Fast provision of new cluster members using the zone cloning feature
Very high network throughput between the zones for data node replication
Optimized disk I/O utilization for better I/O performance with ZFS built-in compression
Secure data at rest using ZFS encryption
For more information see: http://www.oracle.com/technetwork/articles/servers-storage-admin/howto-setup-hadoop-zones-1899993.html
Workshop about TensorFlow usage for AI Ukraine 2016. Brief tutorial with source code example. Described TensorFlow main ideas, terms, parameters. Example related with linear neuron model and learning using Adam optimization algorithm.
Rajat Monga at AI Frontiers: Deep Learning with TensorFlowAI Frontiers
In this talk at AI Frontiers Conference, Rajat Monga shares about TensorFlow that has enabled cutting-edge machine learning research at the top AI labs in the world. At the same time it has made the technology accessible to a large audience leading to some amazing uses. TensorFlow is used for classification, recommendation, text parsing, sentiment analysis and more. This talk goes over the design that makes it fast, flexible, and easy to use, and describe how we continue to make it better.
A quick review and demonstration on how to get started on parallel computing with R. Includes an example of SNOW cluster set up in the departmental lab.
SIMD machines — machines capable of evaluating the same instruction on several elements of data in parallel — are nowadays commonplace and diverse, be it in supercomputers, desktop computers or even mobile ones. Numerous tools and libraries can make use of that technology to speed up their computations, yet it could be argued that there is no library that provides a satisfying minimalistic, high-level and platform-agnostic interface for the C++ developer.
Oracle Solaris 11 is the first operating system engineered with cloud computing in mind. So what's new in Oracle Solaris 11, and how does that connect to the cloud? If you`re involved in Application Life-cycle Management, Configuration Management,
Cloud Deployment, Big Data Design and Application or Infrastructure Scaling - You will learn how to leverage the Solaris 11 technologies in order to build your Cloud infrastructure.
For more information see: http://www.oracle.com/technetwork/systems/hands-on-labs/hol-oracle-solaris-remote-lab-1894053.html
Oracle Solaris 11 as a BIG Data Platform Apache Hadoop Use CaseOrgad Kimchi
The following are benefits of using Oracle Solaris Zones for a Hadoop cluster:
Fast provision of new cluster members using the zone cloning feature
Very high network throughput between the zones for data node replication
Optimized disk I/O utilization for better I/O performance with ZFS built-in compression
Secure data at rest using ZFS encryption
For more information see: http://www.oracle.com/technetwork/articles/servers-storage-admin/howto-setup-hadoop-zones-1899993.html
Java EE 6 Adoption in One of the World’s Largest Online Financial SystemsArshal Ameen
Financial companies need Java EE to power their business today. Rakuten Card, one of the largest credit card companies in Japan, adopted Java EE 6 for its online systems rearchitecture. Learn why it chose Java EE, and hear about its experiences and lessons learned. This is the first time a large credit card company in Japan is sharing its story. How do you start such a big project? Why did it choose Java EE? How did it select the in-house development policies, educate itself, and develop the additional libraries? How did it launch within only six months? What is the key factor driving 24/7 critical financial systems successfully? How do you migrate to Java EE 7 in the future? This presentation answers these questions and any others you may have.
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak PROIDEA
Speaker: Andrzej Dyjak
Language: English
In recent years security industry started to grow fond of Apple’s iOS and OS X platforms. This talk will cover one of XNU's flagship debugging utilities: DTrace, a dynamic tracing framework for troubleshooting kernel and application problems on production systems in real time. It will be shown how it can be used in order to ease various tasks within the realm of dynamic binary analysis and beyond.
CONFidence: http://confidence.org.pl/
Brief introduction to DTrace technologies within OpenSolaris/Solaris 10 and DTrace probes within Apache, PHP and MySQL can provide end to end dynamic tracing of your Drupal based web site..
No more struggles with Apache Spark workloads in productionChetan Khatri
Paris Scala Group Event May 2019, No more struggles with Apache Spark workloads in production.
Apache Spark
Primary data structures (RDD, DataSet, Dataframe)
Pragmatic explanation - executors, cores, containers, stage, job, a task in Spark.
Parallel read from JDBC: Challenges and best practices.
Bulk Load API vs JDBC write
An optimization strategy for Joins: SortMergeJoin vs BroadcastHashJoin
Avoid unnecessary shuffle
Alternative to spark default sort
Why dropDuplicates() doesn’t result consistency, What is alternative
Optimize Spark stage generation plan
Predicate pushdown with partitioning and bucketing
Why not to use Scala Concurrent ‘Future’ explicitly!
TensorFrames: Google Tensorflow on Apache SparkDatabricks
Presentation at Bay Area Spark Meetup by Databricks Software Engineer and Spark committer Tim Hunter.
This presentation covers how you can use TensorFrames with Tensorflow to distributed computing on GPU.
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17spark-project
Slides from Tathagata Das's talk at the Spark Meetup entitled "Deep Dive with Spark Streaming" on June 17, 2013 in Sunnyvale California at Plug and Play. Tathagata Das is the lead developer on Spark Streaming and a PhD student in computer science in the UC Berkeley AMPLab.
DTrace and SystemTap are dynamic tracing frameworks available for Solaris and Linux respectively. This session will give an overview of the static DTrace probes available in both Drizzle and MySQL and show numerous examples of scripts that utilize these probes. Mixing dynamic and static probes will also be discussed.
Automating Analysis and Exploitation of Embedded Device FirmwareMalachi Jones
Dynamic binary analysis tools utilize a combination of techniques that include fuzzing, symbolic execution, and concolic execution to discover exploitable code in sophisticated binaries. Much work has been dedicated to developing automated analysis tools to target mainstream processor architectures (e.g. x86 and x86_64. ). An often overlooked and inadequately addressed area is the development of tools that target embedded systems processors that include PowerPC, MIPS, and SuperH. Historically, a challenge with targeting multiple embedded architectures was that it was often necessary to write an analysis tool for each architecture.
In this talk, we'll discuss an approach for decoupling the architecture specifics from the analysis by utilizing intermediate representation (IR) languages. Intermediate representation languages provide a method to abstract out machine specifics in order to aid in the analysis of computer programs. In particular, the LLVM IR language provides an extensive set of analysis and optimization libraries, along with a JIT engine, that can be collectively utilized to develop architecture-independent automated analysis and exploitation tools.
Building Deep Reinforcement Learning Applications on Apache Spark with Analyt...Databricks
Deep Reinforcement Learning (DRL) is a thriving area in the current AI battlefield. AlphaGO by DeepMind is a very successful application of DRL which has drawn the attention of the entire world. Besides playing games, DRL also has many practical use in industry, e.g. autonomous driving, chatbots, financial investment, inventory management, and even recommendation systems. Although DRL applications has something in common with supervised Computer Vision or Natural Language Processing tasks, they are unique in many ways.
For example, they have to interact (explore) with the environment to obtain training samples along the optimization, and the method to improve the model is usually different from common supervised applications. In this talk we will share our experience of building Deep Reinforcement Learning applications on BigDL/Spark. BigDL is a well-developed deep learning library on Spark which is handy for Big Data users, but it has been mostly used for supervised and unsupervised machine learning. We have made extensions particularly for DRL algorithms (e.g. DQN, PG, TRPO and PPO, etc.), implemented classical DRL algorithms, built applications with them and did performance tuning. We are happy to share what we have learnt during this process.
We hope our experience will help our audience learn how to build a RL application on their own for in their production business.
Red Hat Enteprise Linux Open Stack Platfrom DirectorOrgad Kimchi
Red Hat Enterprise Linux OpenStack Platform director is a toolset for installing and managing a complete OpenStack environment. It is based primarily on the OpenStack project TripleO, which is an abbreviation for "OpenStack-On-OpenStack". This project takes advantage of OpenStack components to install a fully operational OpenStack environment. This includes new OpenStack components that provision and control bare metal systems to use as OpenStack nodes. This provides a simple method for installing a complete Red Hat Enterprise Linux OpenStack Platform environment that is both lean and robust.
Oracle Solaris 11.2 - Engineered for Cloud
Oracle Solaris provides an efficient, secure and compliant, simple, open, and affordable solution for
deploying your enterprise-grade clouds. More than just an operating system, Oracle Solaris 11.2 includes
features and enhancements that deliver no-compromise virtualization, application-driven software-defined
networking, and a complete OpenStack distribution for creating and managing an enterprise cloud, enabling
you to meet IT demands and redefine your business.
For more information: http://www.oracle.com/technetwork/server-storage/solaris11/overview/beta-2182985.html
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...Orgad Kimchi
Analyzing the performance of a virtualized multitenant cloud environment can be challenging because of the layers of abstraction. This article shows how to use Oracle Solaris 11 to overcome those limitations.
For more information see:
http://www.oracle.com/technetwork/articles/servers-storage-admin/perf-analysis-multitenant-cloud-2082193.html
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Cloud Observation and Performance Analysis using Solaris 11 DTrace
1. DTrace, for Solaris (zones
inside) and Java
Amit Hurvitz, ISV Engineering, Oracle
2. Program Agenda
Introductory Demo
What's Dtrace?
Enabling Dtrace from Inside the Zone
Java Statically Defined Tracing
TCP Client and Server Flow Tracing - Demo
Future Thoughts
4. What's DTrace? - cont.
Zero performance impact when not in use
Completely safe; no way to cause panics, crashes, data corruption or
pathological performance degradation
Powerful data management primitives eliminate need for most postprocessing
5. DTrace – D Language
probe description
/predicate/
{
actions
}
Probes Which events we are interested in monitoring
Predicates (optional) When do we want to monitor the events
Actions (optional) What do we want to do when the above
happens
One liner
# dtrace -n 'probe/predicate/{actions}'
{actions}
6. DTrace – Probes
Programmable sensors (points of instrumentation) made available by
providers placed all over the Solaris system
provider:module:function:name
−
tcp:ip:tcp_send:entry
−
Syscall:::
Providers: syscall,io,pid,profile, hotspot, tcp, udp, ip, iscsi,...
Modules: nfs, zfs, cpc, …
Names: entry,return
Listing Probes
# dtrace -l [-P provider | -m module | -f function name | -n name]
11. Java Statically Defined Tracing (JSDT)
Make points of interest in your application easily monitored
Insert your own DTrace probes in desired locations inside your
methods
– Use them in conjunction with any other probes
#!/usr/sbin/dtrace -s
MyProvider:::start
{
self->start_time = timestamp;
}
syscall:::entry
/self->start_time/
{
@[probefunc] = quantize();
}
15. Next Thoughts
A special ‘Java-DTrace’ utility to do implicit instrumentation
Probes look like native DTrace PID provider:
−
−
JDDT$target:class-name:method-name:entry
JDDT$target:class-name:method-name:return
# jdtrace java-dtrace-script.d -p <process-id>
jdtrace will take care of all required dynamic instrumentation
−
Clean instrumented code on script end
Any suggestions?