This document provides an overview of effective numerical computation in NumPy and SciPy. It discusses how Python can be used for numerical computation tasks like differential equations, simulations, and machine learning. While Python is initially slower than languages like C, libraries like NumPy and SciPy allow Python code to achieve sufficient speed through techniques like broadcasting, indexing, and using sparse matrix representations. The document provides examples of how to efficiently perform tasks like applying functions element-wise to sparse matrices and calculating norms. It also presents a case study for efficiently computing a formula that appears in a machine learning paper using different sparse matrix representations in SciPy.
In this power point presentation i have explained about Seaborn Library in Data Visualization.
I have touched the topics like Introduction, what is Seaborn types etc.
Hope this ppt will help you & you will like it.
Thank You
All the best
In this power point presentation i have explained about Seaborn Library in Data Visualization.
I have touched the topics like Introduction, what is Seaborn types etc.
Hope this ppt will help you & you will like it.
Thank You
All the best
Part B CS8391 Data Structures Part B Questions compiled from R2008 & R2013 to help the students of Affiliated Colleges appearing for Ann University Examination
Programmers love Python because of how fast and easy it is to use. Python cuts development time in half with its simple to read syntax and easy compilation feature. Debugging your programs is a breeze in Python with its built in debugger. Using Python makes Programmers more productive and their programs ultimately better. Python is continued to be a favorite option for data scientists who use it for building and using Machine learning applications and other scientific computations.
Python runs on Windows, Linux/Unix, Mac OS and has been ported to Java and .NET virtual machines. Python is free to use, even for the commercial products, because of its OSI-approved open source license.
Python has evolved as the most preferred Language for Data Analytics and the increasing search trends on python also indicates that Python is the next "Big Thing" and a must for Professionals in the Data Analytics domain.
this is a briefer overview about the Big O Notation. Big O Notaion are useful to check the Effeciency of an algorithm and to check its limitation at higher value. with big o notation some examples are also shown about its cases and some functions in c++ are also described.
Algorithm Complexity presentation slides. Time Complexity and Space Complexity analysis using Big-O notation with examples that demonstrates how a function complexity effects to algorithm efficiency.
A presentation about the ideas of recursion and recursive functions.
This is my lecture presentation during A. Paruj Ratanaworabhan’s basic preparatory programming course for freshmen: Introduction to Programming: A Tutorial for New Comers Using Python
Part B CS8391 Data Structures Part B Questions compiled from R2008 & R2013 to help the students of Affiliated Colleges appearing for Ann University Examination
Programmers love Python because of how fast and easy it is to use. Python cuts development time in half with its simple to read syntax and easy compilation feature. Debugging your programs is a breeze in Python with its built in debugger. Using Python makes Programmers more productive and their programs ultimately better. Python is continued to be a favorite option for data scientists who use it for building and using Machine learning applications and other scientific computations.
Python runs on Windows, Linux/Unix, Mac OS and has been ported to Java and .NET virtual machines. Python is free to use, even for the commercial products, because of its OSI-approved open source license.
Python has evolved as the most preferred Language for Data Analytics and the increasing search trends on python also indicates that Python is the next "Big Thing" and a must for Professionals in the Data Analytics domain.
this is a briefer overview about the Big O Notation. Big O Notaion are useful to check the Effeciency of an algorithm and to check its limitation at higher value. with big o notation some examples are also shown about its cases and some functions in c++ are also described.
Algorithm Complexity presentation slides. Time Complexity and Space Complexity analysis using Big-O notation with examples that demonstrates how a function complexity effects to algorithm efficiency.
A presentation about the ideas of recursion and recursive functions.
This is my lecture presentation during A. Paruj Ratanaworabhan’s basic preparatory programming course for freshmen: Introduction to Programming: A Tutorial for New Comers Using Python
Recommendation System --Theory and PracticeKimikazu Kato
Survey on recommendation systems presented at IMI Colloquium, Kyushu University, Feb 18, 2015.
レコメンデーションシステムの最新の研究動向に関する解説です。2015年2月18日に九州大学IMIコロキアムで講演したものです。資料は英語ですが、講演は日本語でやりました。
Introducton to Convolutional Nerural Network with TensorFlowEtsuji Nakai
Explaining basic mechanism of the Convolutional Neural Network with sample TesnsorFlow codes.
Sample codes: https://github.com/enakai00/cnn_introduction
Numerical tour in the Python eco-system: Python, NumPy, scikit-learnArnaud Joly
We first present the Python programming language and the NumPy package for scientific computing. Then, we devise a digit recognition system highlighting the scikit-learn package.
Python for R developers and data scientistsLambda Tree
This is an introductory talk aimed at data scientists who are well versed with R but would like to work with Python as well. I will cover common workflows in R and how they translate into Python. No Python experience necessary.
Time Series Analysis:Basic Stochastic Signal RecoveryDaniel Cuneo
Simple case of a recovering a stochastic signal from a time series with a linear combination of nuisance signals.
Errata:
corrected error in the Gaussian fit.
corrected the JackKnife example and un-centers data.
Corrected sig fig language and rationale
removed jk calculation of mean reformatted cells
In this chapter we will learn about arrays as a way to work with sequences of elements of the same type. We will explain what arrays are, how we declare, create, instantiate and use them. We will examine one-dimensional and multidimensional arrays. We will learn different ways to iterate through the array, read from the standard input and write to the standard output. We will give many example exercises, which can be solved using arrays and we will show how useful they really are.
Pythran: Static compiler for high performance by Mehdi Amini PyData SV 2014PyData
Pythran is a an ahead of time compiler that turns modules written in a large subset of Python into C++ meta-programs that can be compiled into efficient native modules. It targets mainly compute intensive part of the code, hence it comes as no surprise that it focuses on scientific applications that makes extensive use of Numpy. Under the hood, Pythran inter-procedurally analyses the program and performs high level optimizations and parallel code generation. Parallelism can be found implicitly in Python intrinsics or Numpy operations, or explicitly specified by the programmer using OpenMP directives directly in the Python source code. Either way, the input code remains fully compatible with the Python interpreter. While the idea is similar to Parakeet or Numba, the approach differs significantly: the code generation is not performed at runtime but offline. Pythran generates C++11 heavily templated code that makes use of the NT2 meta-programming library and relies on any standard-compliant compiler to generate the binary code. We propose to walk through some examples and benchmarks, exposing the current state of what Pythran provides as well as the limit of the approach.
Did you know that Python preallocates integers from -5 to 257? Reusing them 1000 times, instead of allocating memory for a bigger integer, can save you a couple milliseconds of code’s execution time. If you want to learn more about this kind of optimizations then, … well, probably this presentation is not for you :) Instead of going into such small details, I will talk about more “sane” ideas for writing faster code.
After a brief overview of how you can speed up your Python code in general, we will dig into source code optimization. I will show you some simple and fast ways of measuring the execution time of your code, and then we will discuss examples of how to improve some common code structures.
You will see:
* The fastest way of removing duplicates from a list
* How much faster your code is when you reuse the built-in functions instead of trying to reinvent the wheel
* What is faster than the “for loop”
* If the lookup is faster in a list or a set
* When it’s better to beg for forgiveness than to ask for permission
Introduction to behavior based recommendation systemKimikazu Kato
Material presented at Tokyo Web Mining Meetup, March 26, 2016.
The source code is here:
https://github.com/hamukazu/tokyo.webmining.2016-03-26
東京ウェブマイニング(2016年3月27)の発表資料です。すべて英語です。
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
2. About Myself
Kimikazu Kato
Chief Scientists at Silver Egg Technology Co., Ltd.
Ph.D in Computer Science
Background in Mathematics, Numerical Computation, Algorithms, etc.
<2 year experience in Python
>10 year experience in numerical computation
Now designing algorithms for recommendation system, and doing research
about machine learning and data analysis.
2 / 35
3. This talk...
is about effective usage of NumPy/SciPy
is NOT exhaustive introduction of capabilities, but shows some case
studies based on my experience and interest
3 / 35
4. Table of Contents
Introduction
Basics about NumPy
Broadcasting
Indexing
Sparse matrix
Usage of scipy.sparse
Internal structure
Case studies
Conclusion
4 / 35
5. Numerical Computation
Differential equations
Simulations
Signal processing
Machine Learning
etc...
Why Numerical Computation in Python?
Productivity
Easy to write
Easy to debug
Connectivity with visualization tools
Matplotlib
IPython
Connectivity with web system
Many frameworks (Django, Pyramid, Flask, Bottle, etc.)
5 / 35
6. But Python is Very Slow!
Code in C
#include <stdio.h>
int main() {
int i; double s=0;
for (i=1; i<=100000000; i++) s+=i;
printf("%.0fn",s);
}
Code in Python
s=0.
for i in xrange(1,100000001):
s+=i
print s
Both of the codes compute the sum of integers from 1 to 100,000,000.
Result of benchmark in a certain environment:
Above: 0.109 sec (compiled with -O3 option)
Below: 8.657 sec
(80+ times slower!!)
6 / 35
7. Better code
import numpy as np
a=np.arange(1,100000001)
print a.sum()
Now it takes 0.188 sec. (Measured by "time" command in Linux, loading time
included)
Still slower than C, but sufficiently fast as a script language.
7 / 35
8. Lessons
Python is very slow when written badly
Translate C (or Java, C# etc.) code into Python is often a bad idea.
Python-friendly rewriting sometimes result in drastic performance
improvement
8 / 35
9. Basic rules for better performance
Avoid for-sentence as far as possible
Utilize libraries' capabilities instead
Forget about the cost of copying memory
Typical C programmer might care about it, but ...
9 / 35
11. Broadcasting
>>> import numpy as np
>>> a=np.array([0,1,2])
>>> a*3
array([0, 3, 6])
>>> b=np.array([1,4,9])
>>> np.sqrt(b)
array([ 1., 2., 3.])
A function which is applied to each element when applied to an array is called
a universal function.
11 / 35
14. Refernces
Gabriele Lanaro, "Python High Performance Programming," Packt
Publishing, 2013.
Stéfan van der Walt, Numpy Medkit
14 / 35
15. Sparse matrix
Defined as a matrix in which most elements are zero
Compressed data structure is used to express it, so that it will be...
Space effective
Time effective
15 / 35
16. scipy.sparse
The class scipy.sparse has mainly three types as expressions of a sparse
matrix. (There are other types but not mentioned here)
lil_matrix : convenient to set data; setting a[i,j] is fast
csr_matrix : convenient for computation, fast to retrieve a row
csc_matrix : convenient for computation, fast to retrieve a column
Usually, set the data into lil_matrix, and then, convert it to csc_matrix or
csr_matrix.
For csr_matrix, and csc_matrix, calcutaion of matrices of the same type is fast,
but you should avoid calculation of different types.
16 / 35
20. Merit of knowing the internal structure
Setting csr_matrix or csc_matrix with its internal structure is much faster than
setting lil_matrix with indices.
See the benchmark of setting
ý ý
ý
20 / 35
21. from scipy.sparse import lil_matrix, csr_matrix
import numpy as np
from timeit import timeit
def set_lil(n):
a=lil_matrix((n,n))
for i in xrange(n):
a[i,i]=2.
if i+1n:
a[i,i+1]=1.
return a
def set_csr(n):
data=np.empty(2*n-1)
indices=np.empty(2*n-1,dtype=np.int32)
indptr=np.empty(n+1,dtype=np.int32)
# to be fair, for-sentence is intentionally used
# (using indexing technique is faster)
for i in xrange(n):
indices[2*i]=i
data[2*i]=2.
if in-1:
indices[2*i+1]=i+1
data[2*i+1]=1.
indptr[i]=2*i
indptr[n]=2*n-1
a=csr_matrix((data,indices,indptr),shape=(n,n))
return a
print lil:,timeit(set_lil(10000),
number=10,setup=from __main__ import set_lil)
print csr:,timeit(set_csr(10000),
number=10,setup=from __main__ import set_csr)
21 / 35
22. Result:
lil: 11.6730761528
csr: 0.0562081336975
Remark
When you deal with already sorted data, setting csr_matrix or csc_matrix
with data, indices, indptr is much faster than setting lil_matrix
But the code tend to be more complicated if you use the internal structure
of csr_matrix or csc_matrix
22 / 35
24. Case 1: Norms
If 2
is dense:
norm=np.dot(v,v)
Ï2 Ï %
2%
Expressed as product of matrices. (dot means matrix product, but you don't
have to take transpose explicitly.)
When is sparse, suppose that is expressed as matrix:
2 2 g *
norm=v.multiply(v).sum()
(multiply() is element-wise product)
This is because taking transpose of a sparse matrix changes the type.
24 / 35
26. Case 2: Applying a function to all of the elements of a
sparse matrix
A universal function can be applied to a dense matrix:
import numpy as np
a=np.arange(9).reshape((3,3))
a
array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
np.tanh(a)
array([[ 0. , 0.76159416, 0.96402758],
[ 0.99505475, 0.9993293 , 0.9999092 ],
[ 0.99998771, 0.99999834, 0.99999977]])
This is convenient and fast.
However, we cannot do the same thing for a sparse matrix.
26 / 35
27. from scipy.sparse import lil_matrix
a=lil_matrix((3,3))
a[0,0]=1.
a[1,0]=2.
b=a.tocsr()
np.tanh(b)
3x3 sparse matrix of type 'type 'numpy.float64''
with 2 stored elements in Compressed Sparse Row format
This is because, for an arbitrary function, its application to a sparse matrix is
not necessarily sparse.
However, if a universal function satisfies
, the density is
preserved.
Then, how can we compute it?
27 / 35
28. Use the internal structure!!
The positions of the non-zero elements are not changed after application of
the function.
Keep indices and indptr, and just change data.
Solution:
b = csr_matrix((np.tanh(a.data), a.indices, a.indptr), shape=a.shape)
28 / 35
29. Case 3: Formula which appears in a paper
In the algorithm for recommendation system [1], the following formula
appears:
øø
* g
where is dense matrix, and D is a diagonal matrix defined from a
given array as:
%
ý
*
Here, (which corresponds to the number of users or items) is big and
(which means the number of latent factors) is small.
[1] Hu et al. Collaborative Filtering for Implicit Feedback Datasets, ICDM,
2008.
*
29 / 35
30. Solution 1:
There is a special class dia_matrix to deal with a diagonal sparse matrix.
import scipy.sparse as sparse
import numpy as np
def f(a,d):
a: 2d array of shape (n,f), d: 1d array of length n
dd=sparse.diags([d],[0])
return np.dot(a.T,dd.dot(a))
30 / 35
32. Solution 3:
û
)
û
)
g g
û
)
û
)
This is equivalent to the broadcasting!
def h(a,d):
return np.dot(a.T*d,a)
ü
ü
ü
*
*
û
*)
ý
*
ü
ü
g
ü
* *
* *
û
*) *
32 / 35
34. Conclusion
Try not to use for-sentence, but use libraries' capabilities instead.
Knowledge about the internal structure of the sparse matrix is useful to
extract further performance.
Mathematical derivation is important. The key is to find a mathematically
equivalent and Python-friendly formula.
Computational speed does not necessarily matter. Finding a better code in
a short time is valuable. Otherwise, you shouldn't pursue too much.
34 / 35
35. Acknowledgment
I would like to thank
(@shima__shima)
who gave me useful advice in Twitter.
35 / 35