This document discusses using SQLAlchemy to access relational databases from Python. It provides an overview of SQLAlchemy, describing its core SQL expression language and object-relational mapper (ORM). SQLAlchemy provides tools and components to assist with database access while maintaining a consistent interface over the Python DB-API. It allows generating SQL statements and mapping database rows to Python objects for a more object-oriented programming experience.
Presented by Rafal Kuć, Consultant and Software engineer, , Sematext Group, Inc.
Even though Solr can run without causing any troubles for long periods of time it is very important to monitor and understand what is happening in your cluster. In this session you will learn how to use various tools to monitor how Solr is behaving at a high level, but also on Lucene, JVM, and operating system level. You'll see how to react to what you see and how to make changes to configuration, index structure and shards layout using Solr API. We will also discuss different performance metrics to which you ought to pay extra attention. Finally, you'll learn what to do when things go awry - we will share a few examples of troubleshooting and then dissect what was wrong and what had to be done to make things work again.
This information is outdated now. For an up to date look at using Cassandra from Python see this presentation: https://speakerdeck.com/tylerhobbs/intro-to-cassandra-and-the-python-driver
Using Apache Cassandra from Python. Given at PyCon 2012.
DataStax: An Introduction to DataStax Enterprise SearchDataStax Academy
1) Why We Built DSE Search
2) Basics of the Read and Write Paths
3) Fault-tolerance and Adaptive Routing
4) Analytics with Search and Spark
5) Live Indexing
Preview of Cassandra 2.2 and 3.0 features. Materialized views, user defined functions, user defined aggregations, new storage engine, rewritten hints, improved vnodes, native JSON support, updated garbage collector.
Sqlite Database | SQLite Tutorial | SQLite3 with Python | Database CRUD | Sqlite Python Tutorial
In this tutorial, you can learn the Database CRUD operations using SQLite3 with Python Language. Here demo of following database CRUD operations are given. Beginners can easily understand how to work with database using sqlite database in python.
Topics Covered
1. Database Creation, Table Creation, Data Insertion
2. Data Retrieval
3. Data Updation
4. Data Deletion
Hope this will be helpful for those who are searching the database tutorial in python language...
For Demo Visit the link below:
https://youtu.be/0yxhJ8amoi8 (Elangovan Tech Notes-YouTube Channel)
Effective testing for spark programs Strata NY 2015Holden Karau
This session explores best practices of creating both unit and integration tests for Spark programs as well as acceptance tests for the data produced by our Spark jobs. We will explore the difficulties with testing streaming programs, options for setting up integration testing with Spark, and also examine best practices for acceptance tests.
Unit testing of Spark programs is deceptively simple. The talk will look at how unit testing of Spark itself is accomplished, as well as factor out a number of best practices into traits we can use. This includes dealing with local mode cluster creation and teardown during test suites, factoring our functions to increase testability, mock data for RDDs, and mock data for Spark SQL.
Testing Spark Streaming programs has a number of interesting problems. These include handling of starting and stopping the Streaming context, and providing mock data and collecting results. As with the unit testing of Spark programs, we will factor out the common components of the tests that are useful into a trait that people can use.
While acceptance tests are not always part of testing, they share a number of similarities. We will look at which counters Spark programs generate that we can use for creating acceptance tests, best practices for storing historic values, and some common counters we can easily use to track the success of our job.
Relevant Spark Packages & Code:
https://github.com/holdenk/spark-testing-base / http://spark-packages.org/package/holdenk/spark-testing-base
https://github.com/holdenk/spark-validator
Python and EM CLI: The Enterprise Management Super ToolsSeth Miller
Release 3 of Enterprise Manager gives the command line interface for EM a distinct advantage by moving the EMCLI functionality into Jython, a Java implementation of the Python programming language.
This session will provide an introduction to Python and give attendees a crash course in the newest version of EMCLI so they can get started using this powerful tool in their environments right away.
Learning Objectives:
-- Evaluate where Python can provide solutions in other aspects of the DBA's responsibility including automating password changes and backups.
-- Understand how to implement and use release 3 of EMCLI. Differentiate between the Jython architecture of release 3 versus earlier versions of EMCLI.
-- Have a basic understanding of and be able to construct simple scripts in Python.
Presented by Rafal Kuć, Consultant and Software engineer, , Sematext Group, Inc.
Even though Solr can run without causing any troubles for long periods of time it is very important to monitor and understand what is happening in your cluster. In this session you will learn how to use various tools to monitor how Solr is behaving at a high level, but also on Lucene, JVM, and operating system level. You'll see how to react to what you see and how to make changes to configuration, index structure and shards layout using Solr API. We will also discuss different performance metrics to which you ought to pay extra attention. Finally, you'll learn what to do when things go awry - we will share a few examples of troubleshooting and then dissect what was wrong and what had to be done to make things work again.
This information is outdated now. For an up to date look at using Cassandra from Python see this presentation: https://speakerdeck.com/tylerhobbs/intro-to-cassandra-and-the-python-driver
Using Apache Cassandra from Python. Given at PyCon 2012.
DataStax: An Introduction to DataStax Enterprise SearchDataStax Academy
1) Why We Built DSE Search
2) Basics of the Read and Write Paths
3) Fault-tolerance and Adaptive Routing
4) Analytics with Search and Spark
5) Live Indexing
Preview of Cassandra 2.2 and 3.0 features. Materialized views, user defined functions, user defined aggregations, new storage engine, rewritten hints, improved vnodes, native JSON support, updated garbage collector.
Sqlite Database | SQLite Tutorial | SQLite3 with Python | Database CRUD | Sqlite Python Tutorial
In this tutorial, you can learn the Database CRUD operations using SQLite3 with Python Language. Here demo of following database CRUD operations are given. Beginners can easily understand how to work with database using sqlite database in python.
Topics Covered
1. Database Creation, Table Creation, Data Insertion
2. Data Retrieval
3. Data Updation
4. Data Deletion
Hope this will be helpful for those who are searching the database tutorial in python language...
For Demo Visit the link below:
https://youtu.be/0yxhJ8amoi8 (Elangovan Tech Notes-YouTube Channel)
Effective testing for spark programs Strata NY 2015Holden Karau
This session explores best practices of creating both unit and integration tests for Spark programs as well as acceptance tests for the data produced by our Spark jobs. We will explore the difficulties with testing streaming programs, options for setting up integration testing with Spark, and also examine best practices for acceptance tests.
Unit testing of Spark programs is deceptively simple. The talk will look at how unit testing of Spark itself is accomplished, as well as factor out a number of best practices into traits we can use. This includes dealing with local mode cluster creation and teardown during test suites, factoring our functions to increase testability, mock data for RDDs, and mock data for Spark SQL.
Testing Spark Streaming programs has a number of interesting problems. These include handling of starting and stopping the Streaming context, and providing mock data and collecting results. As with the unit testing of Spark programs, we will factor out the common components of the tests that are useful into a trait that people can use.
While acceptance tests are not always part of testing, they share a number of similarities. We will look at which counters Spark programs generate that we can use for creating acceptance tests, best practices for storing historic values, and some common counters we can easily use to track the success of our job.
Relevant Spark Packages & Code:
https://github.com/holdenk/spark-testing-base / http://spark-packages.org/package/holdenk/spark-testing-base
https://github.com/holdenk/spark-validator
Python and EM CLI: The Enterprise Management Super ToolsSeth Miller
Release 3 of Enterprise Manager gives the command line interface for EM a distinct advantage by moving the EMCLI functionality into Jython, a Java implementation of the Python programming language.
This session will provide an introduction to Python and give attendees a crash course in the newest version of EMCLI so they can get started using this powerful tool in their environments right away.
Learning Objectives:
-- Evaluate where Python can provide solutions in other aspects of the DBA's responsibility including automating password changes and backups.
-- Understand how to implement and use release 3 of EMCLI. Differentiate between the Jython architecture of release 3 versus earlier versions of EMCLI.
-- Have a basic understanding of and be able to construct simple scripts in Python.
What is Python? An overview of Python for science.Nicholas Pringle
A brief introduction on the use of Python for scientists. Python is fast becoming a popular programming language for scientists. It is free, open source and constantly improving. Being an easy language to learn, it has a large a community of users. Its many favourable qualities make it the perfect language for scientific collaboration.
Big data analysis in python @ PyCon.tw 2013Jimmy Lai
Big data analysis involves several processes: collecting, storage, computing, analysis and visualization. In this slides, the author demonstrates these processes by using python tools to build a data product. The example is based on text-analyzing an online forum.
The Agenda for the Webinar:
1. Introduction to Python.
2. Python and Big Data.
3. Python and Data Science.
4. Key features of Python and their usage in Business Analytics.
5. Business Analytics with Python – Real world Use Cases.
The basics of Python are rather straightforward. In a few minutes you can learn most of the syntax. There are some gotchas along the way that might appear tricky. This talk is meant to bring programmers up to speed with Python. They should be able to read and write Python.
http://imatge-upc.github.io/telecombcn-2016-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.
Spark SQL Catalyst Code Optimization using Function Outlining with Kavana Bha...Databricks
Spark SQL Catalyst optimizer, post query plan optimization, compiles the SQL query to Java code. Without code generation, such query expressions would have to be interpreted for each row of data, by walking down a tree of nodes. This introduces large amounts of branches and virtual function calls that slow down execution. With code generation, a query is collapsed into a single optimized function that eliminates multiple function calls and leverages CPU registers for intermediate data.
This code is then compiled in runtime to Java bytecode using Janino compiler. This presentation focuses on further catalyst code generation optimizations possible using function outlining. Automatic code generation tools generally tend to generate huge optimized functions. Large functions that are frequently executed might degrade runtime performance by preventing JVM optimizations such as function inlining. To avoid this, code generation tools should try to contain independent logic into separate functions.
This presentation will take the audience through the Spark Catalyst Code generation, how automatic split of large functions into smaller functions was achieved and the performance benefits associated with it
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...Amazon Web Services
“Infrastructure as Code” has changed not only how we think about configuring infrastructure, but about the infrastructure itself. AWS has been at the core of this movement, enabling your infrastructure teams to benefit from software engineering best practices such as CI/CD, automated testing, and repeatable deployments. Now that you have mastered the art of managing your infrastructure as code, it’s time to leverage these same lessons for monitoring and metrics. In this session, we dive into how you can leverage tooling such as AWS, Terraform, and Datadog to programmatically define your monitoring so that you that you can scale your organizational observability along with your infrastructure, and attain consistency from local development all the way through production.
Session sponsored by Datadog, Inc.
Automating with NX-OS: Let's Get Started!Cisco DevNet
A session in the DevNet Zone at Cisco Live, Berlin. Cisco's flagship data center platform, the Nexus series of switches, has a variety of programming protocols to offer. This session will provide participants with an overview and code examples on various protocols: * NX-API * XMPP * Netconf
Finding target for hacking on internet is now easierDavid Thomas
Finding target on internet for penetration testing involves searching internet using google or using Google Hacking/Dorking. There are google hacking queries available on internet, according to ethical hacking researcher of International Institute of Cyber Security it is the main source of passive attacks on internet. This whole process of finding target on internet using GHDB is automated using python based framework named as Katana framework.
Having used Jenkins with more than 30 projects during 8 years, it was time to switch to another build manager. Written in Ada, Porion is a new build manager intended to replace Jenkins by providing more security, safety and performance.
The presentation describes the complexity of designing a build manager. It highlights some security issues that apply to a build manager and its implementation. It explains the overall architecture that was chosen and the reason of the choice. Finally it presents the Ada generation tools that have been used in this project.
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak PROIDEA
Speaker: Andrzej Dyjak
Language: English
In recent years security industry started to grow fond of Apple’s iOS and OS X platforms. This talk will cover one of XNU's flagship debugging utilities: DTrace, a dynamic tracing framework for troubleshooting kernel and application problems on production systems in real time. It will be shown how it can be used in order to ease various tasks within the realm of dynamic binary analysis and beyond.
CONFidence: http://confidence.org.pl/
Similar to Relational Database Access with Python (20)
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
2. Is This Your Current Relational Database Access
Style?
# Django ORM
>>> from ip2country.models import Ip2Country
>>> Ip2Country.objects.all()
[<Ip2Country: Ip2Country object>, <Ip2Country: Ip2Country
object>, '...(remaining elements truncated)...']
>>> myp = Ip2Country.objects.filter(assigned__year=2015)
... .filter(countrycode2=’MY')
>>> myp[0].ipfrom
736425984.0
3. Is This Your Current Relational Database Access
Style?
# SQLAlchemy ORM
>>> from sqlalchemy import create_engine, extract
>>> from sqlalchemy.orm import sessionmaker
>>> from models import Ip2Country
>>> engine =
create_engine('postgresql://ip2country_rw:secret@localhost/i
p2country')
>>> Session = sessionmaker(bind=engine)
>>> session = Session()
>>> all_data = session.query(Ip2Country).all()
>>> myp = session.query(Ip2Country).
... filter(extract('year',Ip2Country.assigned) == 2015).
... filter(Ip2Country.countrycode2 == ’MY')
print(myp[0].ipfrom)
736425984.0
4. SQL Relational Database Access
SELECT * FROM ip2country;
“id”,"ipfrom";"ipto";"registry";"assigned";"countrycode2";"countrycode3";"countryname"
1,1729522688;1729523711;"apnic";"2011-08-05";"CN";"CHN";"China"
2,1729523712;1729524735;"apnic";"2011-08-05";"CN";"CHN";"China”
. . .
SELECT * FROM ip2country
WHERE date_part('year', assigned) = 2015
AND countrycode2 = ’MY';
“id”,"ipfrom";"ipto";"registry";"assigned";"countrycode2";"countrycode3";"countryname"
5217;736425984;736427007;"apnic";"2015-01-13 00:00:00";"MY";"MYS";"Malaysia”
5218;736427008;736428031;"apnic";"2015-01-13 00:00:00";"MY";"MYS";"Malaysia”
. . .
SELECT ipfrom FROM ip2country
WHERE date_part('year', assigned) = 2015
AND countrycode2 = ’MY';
"ipfrom"
736425984
736427008
. . .
5. Python + SQL == Python DB-API 2.0
• The Python standard for a consistent
interface to relational databases is the
Python DB-API (PEP 249)
• The majority of Python database interfaces
adhere to this standard
7. Python DB-API Connection Object
Access the database via the connection object
• Use connect constructor to create a
connection with database
conn = psycopg2.connect(parameters…)
• Create cursor via the connection
cur = conn.cursor()
• Transaction management (implicit begin)
conn.commit()
conn.rollback()
• Close connection (will rollback current
transaction)
conn.close()
• Check module capabilities by globals
psycopg2.apilevel psycopg2.threadsafety
psycopg2.paramstyle
8. Python DB-API Cursor Object
A cursor object is used to represent a database
cursor, which is used to manage the context of
fetch operations.
• Cursors created from the same connection
are not isolated
cur = conn.cursor()
cur2 = conn.cursor()
• Cursor methods
cur.execute(operation, parameters)
cur.executemany(op,seq_of_parameters)
cur.fetchone()
cur.fetchmany([size=cursor.arraysize])
cur.fetchall()
cur.close()
9. Python DB-API Cursor Object
• Optional cursor methods
cur.scroll(value[,mode='relative'])
cur.next()
cur.callproc(procname[,parameters])
cur.__iter__()
• Results of an operation
cur.description
cur.rowcount
cur.lastrowid
• DB adaptor specific “proprietary” cursor
methods
10. Python DB-API Parameter Styles
Allows you to keep SQL separate from parameters
Improves performance & security
Warning Never, never, NEVER use Python string
concatenation (+) or string parameters
interpolation (%) to pass variables to a SQL query
string. Not even at gunpoint.
From http://initd.org/psycopg/docs/usage.html#query-parameters
11. Python DB-API Parameter Styles
Global paramstyle gives supported style for the
adaptor
qmark Question mark style
WHERE countrycode2 = ?
numeric Numeric positional style
WHERE countrycode2 = :1
named Named style
WHERE countrycode2 = :code
format ANSI C printf format style
WHERE countrycode2 = %s
pyformat Python format style
WHERE countrycode2 = %(name)s
12. Python + SQL: INSERT
import csv, datetime, psycopg2
conn = psycopg2.connect("dbname=ip2countrydb user=ip2country_rw
password=secret")
cur = conn.cursor()
with open("IpToCountry.csv", "rt") as f:
reader = csv.reader(f)
try:
for row in reader:
if row[0][0] != "#":
row[3] =
datetime.datetime.utcfromtimestamp(float(row[3]))
cur.execute("""INSERT INTO ip2country(
ipfrom, ipto, registry, assigned,
countrycode2, countrycode3, countryname)
VALUES (%s, %s, %s, %s, %s, %s, %s)""", row)
except (Exception) as error:
print(error)
conn.rollback()
else:
conn.commit()
finally:
cur.close()
conn.close()
13. Python + SQL: SELECT
# Find ipv4 address ranges assigned to Malaysia
import psycopg2, socket, struct
def num_to_dotted_quad(n):
"""convert long int to dotted quad string
http://code.activestate.com/recipes/66517/"""
return socket.inet_ntoa(struct.pack('!L', n))
conn = psycopg2.connect("dbname=ip2countrydb user=ip2country_rw
password=secret")
cur = conn.cursor()
cur.execute("""SELECT * FROM ip2country
WHERE countrycode2 = 'MY'
ORDER BY ipfrom""")
for row in cur:
print("%s - %s" % (num_to_dotted_quad(int(row[0])),
num_to_dotted_quad(int(row[1]))))
20. ODBC
• mxODBC
• CPython 2.3+
• DB-API 2.0 interfaces
• http://www.egenix.com/products/pytho
n/mxODBC/doc
• Commercial product
• PyODBC
• CPython 2 & 3
• DB-API 2.0 interfaces with extensions
• https://github.com/mkleehammer/pyod
bc
• ODBC interfaces not limited to Windows
thanks to iODBC and unixODBC
21. Jython + SQL
• zxJDBC
• DB-API 2.0 Written in Java using JDBC API
so can utilize JDBC drivers
• Support for connection pools and JNDI
lookup
• Included with standard Jython
installation http://www.jython.org/
• jyjdbc
• DB-API 2.0 compliant
• Written in Python/Jython so can utilize
JDBC drivers
• Decimal data type support
• https://bitbucket.org/clach04/jyjdbc/
22. IronPython + SQL
• adodbapi
• IronPython 2+
• Also works with CPython 2.3+ with
pywin32
• http://adodbapi.sourceforge.net/
23. Gerald, the half a schema
import gerald
s1 = gerald.PostgresSchema(’public',
'postgres://ip2country_rw:secret@localhost/ip2country')
s2 = gerald.PostgresSchema(’public',
'postgres://ip2country_rw:secret@localhost/ip2countryv4')
print s1.schema['ip2country'].compare(s2.schema['ip2country'])
DIFF: Definition of assigned is different
DIFF: Column countryname not in ip2country
DIFF: Definition of registry is different
DIFF: Column countrycode3 not in ip2country
DIFF: Definition of countrycode2 is different
• Database schema toolkit
• via DB-API currently supports
• PostgreSQL
• MySQL
• Oracle
• http://halfcooked.com/code/gerald/
24. SQLPython
$ sqlpython --postgresql ip2country ip2country_rw
Password:
0:ip2country_rw@ip2country> select * from ip2country where countrycode2='SG';
...
1728830464.0 1728830719.0 apnic 2011-11-02 SG SGP Singapore
551 rows selected.
0:ip2country_rw@ip2country> select * from ip2country where countrycode2='SG'j
[...
{"ipfrom": 1728830464.0, "ipto": 1728830719.0, "registry": "apnic”,"assigned":
"2011-11-02", "countrycode2": "SG", "countrycode3": "SGP", "countryname":
"Singapore"}]
• A command-line interface to relational
databases
• via DB-API currently supports
• PostgreSQL
• MySQL
• Oracle
• http://packages.python.org/sqlpython/
25. SQLPython, batteries included
0:ip2country_rw@ip2country> select * from ip2country where countrycode2 =’MY’;
...
1728830464.0 1728830719.0 apnic 2011-11-02 MY MYS Malaysia
551 rows selected.
0:ip2country_rw@ip2country> py
Python 2.6.6 (r266:84292, May 20 2011, 16:42:25)
[GCC 4.4.5 20110214 (Red Hat 4.4.5-6)] on linux2
py <command>: Executes a Python command.
py: Enters interactive Python mode.
End with `Ctrl-D` (Unix) / `Ctrl-Z` (Windows), `quit()`, 'exit()`.
Past SELECT results are exposed as list `r`;
most recent resultset is `r[-1]`.
SQL bind, substitution variables are exposed as `binds`, `substs`.
Run python code from external files with ``run("filename.py")``
>>> r[-1][-1]
(1728830464.0, 1728830719.0, 'apnic', datetime.date(2011, 11, 2), ’MY', ’MYS',
’Malaysia')
>>> import socket, struct
>>> def num_to_dotted_quad(n):
... return socket.inet_ntoa(struct.pack('!L',n))
...
>>> num_to_dotted_quad(int(r[-1][-1].ipfrom))
'103.11.220.0'
26. SpringPython – Database Templates
# Find ipv4 address ranges assigned to Malaysia
# using SpringPython DatabaseTemplate & DictionaryRowMapper
from springpython.database.core import *
from springpython.database.factory import *
conn_factory = PgdbConnectionFactory(
user="ip2country_rw", password="secret",
host="localhost", database="ip2countrydb")
dt = DatabaseTemplate(conn_factory)
results = dt.query(
"SELECT * FROM ip2country WHERE countrycode2=%s",
(”MY",), DictionaryRowMapper())
for row in results:
print("%s - %s" % (num_to_dotted_quad(int(row['ipfrom'])),
num_to_dotted_quad(int(row['ipto']))))
27. SQLAlchemy
http://www.sqlalchemy.org/
First release in 2005
Now at version 1.0.8
What is it
• Provides helpers, tools & components to
assist with database access
• Provides a consisdent and full featured
façade over the Python DBAPI
• Provides an optional object relational
mapper(ORM)
• Foundation for many Python third party
libraries & tools
• It doesn’t hide the database, you need
understand SQL
29. SQLAlchemy Core – The Engine
from sqlalchemy import create_engine
engine =
create_engine('postgresql://ip2country_rw:secret@localhost/i
p2countrydb')
engine.execute("""
create table registry (
id serial primary key,
name text
)
""")
engine.execute("""
insert into registry(name) values('apnic')
""")
engine.execute("""
insert into registry(name) values('aprn')
""”)
engine.execute("""
insert into registry(name) values('lacnic')
""")
30. SQLAlchemy Core – SQL Expression Language
from sqlalchemy import create_engine, Table, Column, Integer, String,
MetaData
engine =
create_engine('postgresql://ip2country_rw:secret@localhost/ip2countrydb',
echo=True)
metadata = MetaData()
registry = Table('registry', metadata,
Column('id', Integer,
autoincrement=True,
primary_key=True),
Column('name', String(10)))
metadata.create_all(engine) # create table if it doesn't exist
# auto construct insert statement with binding parameters
ins = registry.insert().values(name='dummy’)
conn = engine.connect() # get database connection
# insert multiple rows with explicit commit
conn.execute(ins, [{'name': 'apnic'},
{'name': 'aprn'}, {'name': 'lacnic'}])
31. SQLAlchemy Core – SQL Expression Language
from sqlalchemy import create_engine, Table, Column, Integer, String,
MetaData
from sqlalchemy.sql import select
engine =
create_engine('postgresql://ip2country_rw:secret@localhost/ip2countrydb',
echo=True)
metadata = MetaData()
registry = Table('registry', metadata,
Column('id', Integer, autoincrement=True,
primary_key=True,
Column('name', String(10)))
# auto create select statement
s = select([registry])
conn = engine.connect()
result = conn.execute(s)
for row in result:
print(row)
32. SQLAlchemy Core – SQL Expression Language
from sqlalchemy import create_engine, Table, Column, Integer, String,
MetaData
from sqlalchemy.sql import select
engine =
create_engine('postgresql://ip2country_rw:secret@localhost/ip2countrydb',
echo=True)
metadata = MetaData()
registry = Table('registry', metadata,
Column('id', Integer, autoincrement=True,
primary_key=True,
Column('name', String(10)))
# auto create select statement
s = select([registry])
conn = engine.connect()
result = conn.execute(s)
for row in result:
print(row)
33. SQLAlchemy ORM
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import create_engine, Table, Column, Integer, String
Base = declarative_base()
class Registry(Base):
__tablename__ = 'registry'
id = Column(Integer, autoincrement=True, primary_key=True)
name = Column(String(10))
def __repr__(self):
return "<Registry(%r, %r)>" % (
self.id, self.name
)
engine =
create_engine('postgresql://ip2country_rw:secret@localhost/ip2countrydb',
echo=True)
Base.metadata.create_all(engine)
from sqlalchemy.orm import Session
session = Session(bind=engine)
apnic = session.query(Registry).filter_by(name='apnic').first()
print(apnic)
34. SQLAlchemy ORM
. . .
Base = declarative_base()
class Registry(Base):
__tablename__ = 'registry'
id = Column(Integer, autoincrement=True, primary_key=True)
name = Column(String(10))
def __repr__(self):
return "<Registry(%r, %r)>" % (
self.id, self.name
)
engine =
create_engine('postgresql://ip2country_rw:secret@localhost/ip2countrydb',
echo=True)
Base.metadata.create_all(engine)
from sqlalchemy.orm import Session
session = Session(bind=engine)
mynic = Registry(name='mynic')
session.add(mynic)
36. Andy Todd’s OSDC paper
http://halfcooked.com/presentations/osdc2006/p
ython_databases.html
Source of csv data used in examples from
WebNet77 licensed under GPLv3
http://software77.net/geo-ip/
Attributions
37. Mark Rees
mark at censof dot com
+Mark Rees
@hexdump42
hex-dump.blogspot.com
Contact Details
Editor's Notes
If you were at PyCon APAC 2012, the first part of this talk will a case of déjà vu if you attended my talk there.
For some Python programmers, their only exposure to accessing relational data is via a object relational mapper (ORM). As powerful is the concept of mapping objects to data, sometimes it is much simpler to manipulate your relational data using SQL. This talk will be about using the DB-API, Python’s standard mechanism for accessing relational databases.
Or maybe you prefer sqlalchemy to abstract away the database. Both the Django ORM and SQLAlchemy need a lower level API to access databases. So the first part of this talk will be about using the DB-API, Python’s standard mechanism for accessing relational databases.
SQL (Structured Query Language) is a DSL and we can achieve the same results as the previous two slides. This what DBA’s program in.
This diagram no longer seems to exist on Travis’s site
Always use parameter binding. Why?
* you normally get better performance from some database engines due to to SQL query caching
* reduce the chance of SQL injection
Always use parameter binding. Why?
* you normally get better performance from some database engines due to to SQL query caching
* reduce the chance of SQL injection
Gerald is a general purpose database schema toolkit written in Python. It can be used for cataloguing, managing and deploying database schemas. It is designed to allow you to easily identify the differences between databases.
SQLPython is a command-line interface to relational databases written in Python. It was created as an alternative to Oracle’s SQL\*Plus, and can likewise be used instead of postgres’ psql or mysql’s mysql text clients. In addition, it offers several extra features inspired by other command-line clients: Neatened output, smart prompt, tab completion, history, scripting, output to file, paste buffer & os command, unix like commands – ls cat grep, data dictionary exploration. Another feature is special output formats. By replacing the ; that terminates a SELECT statement with a backslash-character sequence, you can get output in a number of useful formats like xml, json, csv etc
One of the most powerful features is the py command. The py command allows the user to execute Python commands, either one-at-a-time (with py {command}) or in an interactive environment (beginning with a bare py statement, and continuing until Ctrl-D, quit(), or exit() is entered). A history of result sets from each query is exposed to the python session as the list r; the most recent result set is r[-1]. Each row can be references as a tuple, or as an object with an attribute for each column.
Spring Python takes the concepts of the Spring Framework and Spring Security, and brings them to the world of Python. It isn't a simple line-by-line port of the code. Instead, it takes some powerful ideas that were discovered in the realm of Java, and pragmatically applies them in the world of Python.
One of these paradigms is a Portable Service Abstraction called DatabaseTemplate.
* It is portable because it uses Python's standardized API, not tying us to any database vendor. Instead, in our example, we injected in an instance of Sqlite3ConnectionFactory
* It provides the useful service of easily accessing information stored in a relational database, but letting us focus on the query, not the plumbing code
* It offers a nice abstraction over Python's low level database API with reduced code noise. This allows us to avoid the cost and risk of writing code to manage cursors and exception handling
DatabaseTemplate handles exceptions by catching and holding them, then properly closing the cursor. It then raises it wrapped inside a Spring Python DataAccessException. This way, database resources are properly disposed of without losing the exception stack trace.
The Database Template can be used in isolation from the SpringPython framework.
Core
Engine – a registry which provides connectivity to a particular database server
Dialect – interprets generic SQL and database commands to match a specific DBAPI & database backend
Connection Pool – a collection of database connections in memory for fast re-use
SQL Expression Language – Allows SQL statements to be written using Python expressions
Schema/Types – Uses objects to represent tables, columns and data types
ORM
Allows construction of Python objects which can be mapped to relational database tables
Transparently persists objects into their corresponding database tables using the unit of work pattern
Provides a query system which loads objects and attributes using SQL generated from mappings
Built of top of the Core, uses core to generate SQL and talk to DB
Not much different from using the DBAPI directly
But using the SQL Expression Language it is more pythonic
You can do the same for selects
You can do the same for selects
You use a session to use the unit of work pattern
The session caches the add and doesn’t do the insert until the session is committed.