• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Writing infinite scalability web applications with PHP and PostgreSQL

Writing infinite scalability web applications with PHP and PostgreSQL



PostgreSQL 9.2 introduced native support for the JSON data type, as well as V8/Javascript and Coffeescript procedural languages. ...

PostgreSQL 9.2 introduced native support for the JSON data type, as well as V8/Javascript and Coffeescript procedural languages.
Learn how you can write web applications in PHP using an intelligent and horizontally sharded cluster of PostgreSQL databases, bringing you infinite scalability and parallel processing.
This talk will guide you through the development lifecycle of the application, focusing on architecture, technologies, testing and deployment.



Total Views
Views on SlideShare
Embed Views



2 Embeds 442

https://twitter.com 324
http://www.scoop.it 118



Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.


11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Writing infinite scalability web applications with PHP and PostgreSQL Writing infinite scalability web applications with PHP and PostgreSQL Presentation Transcript

    • Writing infinite scalability webapplications with PHP andPostgreSQLPHPDay 2013 - VeronaGabriele Bartolini2ndQuadrant Italiagabriele.bartolini@2ndQuadrant.it@_GBartolini_sabato 18 maggio 13
    • Gabriele Bartolini•Co-Founder and Manager of 2ndQuadrant Italia• Data Architect, Business critical environments (OLTP)• Data warehousing•Co-Founder Italian PostgreSQL Users Group•Co-Founder PostgreSQL Europe•PostgreSQL Contributor and Advocatesabato 18 maggio 13
    • Gabriele Bartolini 1.0 (1995-2005)•Web mining•Web programmer• C, C++ and PHP• Web accessibility• IWA• W3C (EARL)•Open source developer• ht://Dig (C++)• ht://Check (C++, MySQL,PHP)• ht://Miner (C++,PostgreSQL)sabato 18 maggio 13
    • PostgreSQL = Postgres100% open source, available at www.postgresql.orgsabato 18 maggio 13
    • Who’s using Postgres?... in a production environmentsabato 18 maggio 13
    • Who’s planning to use Postgres?sabato 18 maggio 13
    • Who’s familiar with sharding?sabato 18 maggio 13
    • Traditional PHP applicationHTML 5JavascriptCSS 3PHPPostgresJSONResultSetsabato 18 maggio 13
    • Postgres 9.2+ PHP ApplicationHTML 5JavascriptCSS 3PHPPostgresJSONsabato 18 maggio 13
    • HTML 5JavascriptCSS 3PHPPostgresPL/ProxyPostgres PostgresPostgres“NoSQL”......NoSQL with Postgressabato 18 maggio 13
    • Sharding•Horizontal partitioningtechnique•Logically and physicallysplit a table by distributingdata across differentdatabases/servers• Reduced number of rows ineach shard• Reduced index size•Some DBMS have the"DISTRIBUTED BY" clausefor "CREATE TABLE"•Query distribution• Parallel processing• Shared nothing architecturesabato 18 maggio 13
    • Table 1Table 1Shard 1Table 1Shard 2Table 1Shard 3Table 1Shard ...Table 1Shard nTable 1SELECT * FROM Table1;Parallel execution on the ‘n’ shardssabato 18 maggio 13
    • Part IPostgre(s|SQL)sabato 18 maggio 13
    • PostgreSQL•Started in 1995 (Postgres started in 1986)•Current major version is 9.2• Supported major versions are 8.4, 9.0, 9.1 and 9.2•The PostgreSQL License (BSD style)• Not subject to monopoly nor acquisitions•Platform independent (C language)sabato 18 maggio 13
    • Some major features•Client/Server•Concurrency (MVCC)•Referential integrity•ACID transactions•Views•Functions•Triggers•Tablespaces•Partitioning•Schemas•Point In Time Recovery•Master/Slave Replicationsabato 18 maggio 13
    • Features for developers•Standard compliant (e.g.SQL, XML, UTF8, JSON, ...)•Client interfaces: C/C++,PHP, JDBC, ODBC, .Net,Python, Perl, Ruby, ...•Vast gamma of ORMs•Several procedurallanguages•Versatile and extensible• custom data types• intelligence in the database• extensions (from 9.1)• PostGIS•Database Unit testingsabato 18 maggio 13
    • PostgreSQL 9.3•Autumn 2013•JSON operators andfunctions•Updatable views•Materialised views•Writable foreign tables•Support for LATERAL•...•http://wiki.postgresql.org/wiki/What%27s_new_in_PostgreSQL_9.3sabato 18 maggio 13
    • NO ... DISTRIBUTED BY•PostgreSQL does not support transparent distribution ofdata and queries (yet)•Sharding can be implemented through PL/Proxy usingfunctionssabato 18 maggio 13
    • Part IIArchitecture and scalabilitysabato 18 maggio 13
    • ScalabilityProperty of a database to grow according to bothtechnical and business requirementssabato 18 maggio 13
    • ServerdbdbSingleSMP ServerVertical scalabilitysabato 18 maggio 13
    • ServerdbCommodityHardwareServer ServerdbHorizontal scalabilitysabato 18 maggio 13
    • DisclaimerFor the sake of simplicity, every database serverdescribed in the following architecture diagrams,deliberately represents a single point of failure.Due to its high availability/disaster recovery features andopen source tools, PostgreSQL is widely used in missioncritical environments.sabato 18 maggio 13
    • ServerpostgresPhysical/VirtualserverSingle server architectureapache/phpPros: costCons: performance (CPU, RAM, I/O), scalability, businesscontinuity, security, no db connection poolingsabato 18 maggio 13
    • Web serverTwo-server architectureapache/phpPros: flexibility (hardware, OS), cost optimisation, security,performance (CPU, RAM, I/O), db connection poolingCons: business continuity, scalabilityDB ServerpostgrespgbouncerVirtual, +CPU, +RAM, -I/OPhysical, =CPU, =RAM, +I/Osabato 18 maggio 13
    • Benchmark•2 x Intel Xeon E5630 CPUs• 48 GB RAM• Storage:• 2 SAS 7.2k rpm, RAID 1• 2 SAS 15k rpm, RAID 1• 6 SAS 15k rpm, RAID 1+0•850 MB/s seq read•13k UPDATE/sec 32 clients• 1.1k/sec for “large” dbs•PostgreSQL 9.2/CentOS 6•Rental costs:• 1.8k € setup + 350 €/monthsabato 18 maggio 13
    • Web serverMultiple web servers architectureapache/phpDB ServerpostgrespgbouncerpgbouncerWeb serverapache/phppgbouncerHA ProxyHTTP BalancingPros: scalability andcontinuity (Web)Cons: scalabilityand continuity (DB)sabato 18 maggio 13
    • Web serverRead only database serversapache/phpDB MasterpostgrespgbouncerpgbouncerWeb serverapache/phppgbouncerHA ProxyHTTP BalancingDB Hot StandbypostgrespgbouncerStreamingReplicationRORORW RWsabato 18 maggio 13
    • Web serverDatabase sharding architectureDB Access ServerpostgrespgbouncerWeb serverHA ProxyWeb layerpl/proxyDatabase Access LayerDB Shard ...postgrespgbouncerDB Shard npostgrespgbouncerDB Shard 1postgrespgbouncerDatabaseData Layer“NoSQL”“NoSQL”“Infinite” horizontal scalabilitySharednothingarchitecturesabato 18 maggio 13
    • Architecture recap•Do not get carried away by current trends and buzzwords•Make your decisions about architecture based on:• business continuity requirements• results of both database and application benchmarks•Always start with a simple architecture (two-server)•Enhance gradually up to sharding for infinite scalabilitysabato 18 maggio 13
    • Part III(Open Source) Technologiessabato 18 maggio 13
    • PgBouncer•Lightweight connectionpooler for PostgreSQL•Written in C by Skype•Recycling connections on:• sessions• transactions• statements (for PL/Proxy)•“Virtual databases”• Real databases can be ondifferent backend servers•Online reconfiguration,restart and upgrade• Online statistics• Supports PAUSE/RESUMEsabato 18 maggio 13
    • Configuration example[databases]db1_ro = host=pg2 dbname=db1 port=5432* = host=pg1 port=6432[pgbouncer]listen_addr = *listen_port = 6432pool_mode = session...sabato 18 maggio 13
    • StatementTransactionSession/Transaction/StatementConnection/SessionTransactionINSERTUPDATETransactionINSERTSELECTUPDATEBEGINCOMMITsabato 18 maggio 13
    • hstore•Data type in PostgreSQL•Key/value pairs as a singlevalue•Useful for storing:• unstructured data• many attributes for arecord•Keys and values are storedas text•Set of operators• ->, @>, ?, ...•Indexes•Integrates with JSON in 9.3sabato 18 maggio 13
    • JSON Support•Introduced in PostgreSQL 9.2•Data type•Generators (9.3)•Operators (9.3)•Integration with hstore (9.3)sabato 18 maggio 13
    • CREATE TABLE pictures (id SERIAL PRIMARY KEY,user_id INTEGER NOT NULLREFERENCES users(id),title TEXT,metadata JSON,thumbnail BYTEA,picture BYTEA,created_at TIMESTAMP DEFAULT now(),updated_at TIMESTAMP DEFAULT now());sabato 18 maggio 13
    • INSERT INTO pictures(user_id, title, metadata)VALUES (100,A picture of my Stratocaster,{"latitude":43.883,"longitude":11.1,"width":2240,"height":1680});Storing a JSON valuesabato 18 maggio 13
    • WITH m AS (SELECT 43.883 AS latitude,11.1 AS longitude,2240 AS width, 1680 AS height)INSERT INTO pictures(user_id, title, metadata)SELECT 100,Another picture of my Stratocaster,to_json(m) FROM m;Storing a JSON value #2sabato 18 maggio 13
    • WITH p AS (SELECT title, metadata, created_atFROM pictures WHERE id = 1)SELECT to_json(p) AS json FROM p;{"title":"A picture of my Stratocaster","metadata":{"latitude":43.883,"longitude":11.1,"width":2240,"height":1680},"created_at":"2013-05-1113:11:37.788636"}9.3 Returning a JSON objectsabato 18 maggio 13
    • WITH p AS (SELECT id, title, metadata, created_atFROM pictures WHERE user_id = 100)SELECT json_agg(p) AS json FROM p;[{"id":1,"title":"A picture of my Stratocaster","metadata":{"latitude":43.883,"longitude":11.1,"width":2240,"height":1680},"created_at":"2013-05-11 13:15:21.078747"}, +{"id":2,"title":"Another picture of my Stratocaster","metadata":{"latitude":43.883,"longitude":11.1,"width":2240,"height":1680},"created_at":"2013-05-11 13:15:21.078747"}]9.3 Returning an array of JSONssabato 18 maggio 13
    • 9.3 A taste of JSON operatorsSELECT id, (metadata->longitude,metadata->latitude) AS coordsFROM picturesWHERE user_id = 100;id | coords----+---------------1 | (11.1,43.883)2 | (11.1,43.883)...sabato 18 maggio 13
    • PL/Javascript•Procedural Language•Allows to implement functions inside PostgreSQL using theJavascript Language•Available as an extension•Powered by V8 JavaScript Enginesabato 18 maggio 13
    • CREATE FUNCTION plv8_test(keys TEXT[],vals TEXT[]) RETURNS TEXTLANGUAGE plv8 IMMUTABLE STRICT AS$BODY$var o = {};for(var i=0; i<keys.length; i++) {o[keys[i]] = vals[i];}return JSON.stringify(o);$BODY$;SELECT plv8_test(ARRAY[name, age],ARRAY[Angus, 58]);sabato 18 maggio 13
    • PL/Proxy•Procedural language fordefining proxy functions•Written in C by Skype•The body specifies:• remote function (target)• destination (shard)•Execution can happen onone, some or all shards• Execution is parallel ifshards > 1•Queries are run in auto-commit mode• PgBouncer (statement)sabato 18 maggio 13
    • The need for an API•PL/Proxy forces to workwith functions•One database cluster with:• access/proxy layer• data layer (shards)•Database cluster ~Singleton class•Business logic in thedatabase•Programmers need todefine an API with access/authorisation rules• Proxy functions (PL/Proxy)• Remote functions (any)•Supports SQL/MEDsabato 18 maggio 13
    • Database sharding architecturepg-access-1postgrespl/proxypg-data-2postgrespg-data-1postgresDatabaseAccessLayerDatabaseDataLayers00 s01 s02 s03- Definition of the cluster- Data/Queries distributed by user (hash)- Functions using the ‘plproxy’ language- User data tables- Functions (any language)sabato 18 maggio 13
    • Define the cluster with SQL/MEDCREATE SERVER myclusterFOREIGN DATA WRAPPER plproxyOPTIONS (connection_lifetime 1800,s0 dbname=shard00 host=pg-data-1,s1 dbname=shard01 host=pg-data-1,s2 dbname=shard02 host=pg-data-2,s3 dbname=shard03 host=pg-data-2);sabato 18 maggio 13
    • Proxy function for GETCREATE OR REPLACE FUNCTIONget_pictures(i_uid INTEGER)RETURNS jsonLANGUAGE plproxyAS$BODY$CLUSTER mycluster;RUN ON hashint4(i_uid);$BODY$;sabato 18 maggio 13
    • Data layer function for GETCREATE OR REPLACE FUNCTIONget_pictures(i_uid INTEGER)RETURNS jsonLANGUAGE SQL AS$BODY$WITH p AS (SELECT id, title, metadata, created_atFROM pictures WHERE user_id = $1) SELECT json_agg(p) AS json FROM p;$BODY$;sabato 18 maggio 13
    • Proxy function for INSERTCREATE OR REPLACE FUNCTIONinsert_picture(i_uid INTEGER, i_titleTEXT, i_metadata JSON, i_thumbnail BYTEA,i_picture BYTEA)RETURNS INTEGER LANGUAGE plproxyAS $BODY$CLUSTER mycluster;RUN ON hashint4(i_uid);$BODY$;sabato 18 maggio 13
    • Data function for INSERTCREATE OR REPLACE FUNCTIONinsert_picture(i_uid INTEGER, i_titleTEXT, i_metadata JSON, i_thumbnail BYTEA,i_picture BYTEA)RETURNS INTEGER LANGUAGE SQLAS $BODY$INSERT INTO pictures (user_id, title,metadata, thumbnail, picture) VALUES ($1,$2, $3, $4, $5) RETURNING id;$BODY$;sabato 18 maggio 13
    • Data representation in PHP// Obviously all of this will come from a form$user_id = 100;$title = A picture of my Stratocaster;$metadata = array (latitude => 43.883,longitude => 11.1,width => 2240,height => 1680, // ... EXIF?);$thumbnail = ... // Binary stream$picture = ... // Binary streamsabato 18 maggio 13
    • INSERT using PDO// $access_layer_conninfo = pgsql:...$connection = new PDO($access_layer_conninfo);$stmt = $connection->prepare(SELECTinsert_picture(?, ?, ?, ?, ?)); // NOSQL!$stmt->bindParam(1, $user_id, PDO::PARAM_INT);$stmt->bindParam(2, $title, PDO::PARAM_STR);$stmt->bindParam(3, json_encode($metadata),PDO::PARAM_STR);$stmt->bindParam(4, $thumbnail, PDO::PARAM_LOB);$stmt->bindParam(5, $picture, PDO::PARAM_LOB);$stmt->execute(); // TODO: Check and get the IDsabato 18 maggio 13
    • Memcached•Distributed system for caching of objects in memory• key/value pairs• volatile data•pgmemcache extension•Integration of database and applications (read scaling)sabato 18 maggio 13
    • PostgreSQLMemcachedm[k] = vPHPtrigger+pgmemcachesabato 18 maggio 13
    • Part IVTesting and deploymentsabato 18 maggio 13
    • Test-Driven DB development•Database developmenttechnique•Write tests before actualfeatures•pgTAP is an extension forTDDD in PostgreSQL•Allows to check for objects,privileges, functions,triggers, data, queries, etc.•Transactional•Provides with statistics• via SQL or pg_prove•Supports “Test AnythingProtocol” (TAP)• Continuous integrationsabato 18 maggio 13
    • BEGIN;SELECT plan(3);-- insert test 1 here-- insert test 2 here-- insert test 3 hereSELECT * FROM finish();ROLLBACK;pgTAP group test examplesabato 18 maggio 13
    • SELECT results_eq(SELECT * FROM active_users(),SELECT * FROM users WHERE active,active_users() should return activeusers);pgTAP example with ‘results_eq’sabato 18 maggio 13
    • Deployment with EXTENSION•Extensibility•Deployment is a crucial partof any ICT organisation•PostgreSQL 9.1 introducesEXTENSION for packagingin-database applications•Fully transactional (atomic)•Allows for upgrades anddowngrades•Simplifies logical dump/restore of objects•DevOps with EXTENSION:• Devs write extensions• Ops deploy themsabato 18 maggio 13
    • Extensions in detail•Extensions require developer to write:• Control file (metadata, settings, dependencies, ...)• Installation scripts (SQL statements)• Update scripts (SQL statements, optional)• Source code / binaries (optional)sabato 18 maggio 13
    • Extension management-- INSTALLATIONCREATE EXTENSION pictures_app;-- REMOVALDROP EXTENSION pictures_app;-- UPDATE (fully atomic)ALTER EXTENSION pictures_app UPDATE;sabato 18 maggio 13
    • Part VConclusionssabato 18 maggio 13
    • Conclusions•Postgres is much more than a “traditional” database•PHP integrates perfectly with it, in multiple flavours (nativeinterface, PDO, Doctrine, etc.)•Start exploiting “vertical scalability”•Always perform benchmark!•Evaluate costs and complexity of “horizontal scalability”•Postgres is “business friendly” in terms of licensingsabato 18 maggio 13
    • Feel free to approach me about•Open Source Community•License•Total Cost of Ownership•Development life cycle•Quality of code anddocumentation•Business Continuity•Security and data protection•PostgreSQL Eco-system•Performance and scalability•Software developmentsabato 18 maggio 13
    • Community•Next events:• PGDay.IT (Autumn 2013)• PGConf.EU (Dublin, Oct 29- Nov 1, 2013)•Italian PostgreSQL UsersGroup (ITPUG)• www.itpug.org•PostgreSQL Europe• www.postgresql.eusabato 18 maggio 13
    • References•PostgreSQL: www.postgresql.org•PgBouncer: http://wiki.postgresql.org/wiki/PgBouncer•PL/Proxy: http://wiki.postgresql.org/wiki/PL/Proxy•PL/V8: http://pgxn.org/dist/plv8/•pgTAP: www.pgtap.orgsabato 18 maggio 13
    • Questions?E-mail: gabriele.bartolini@2ndQuadrant.itTwitter: @_GBartolini_sabato 18 maggio 13
    • Thank you!Copyright (c) 2013 2ndQuadrant ItaliaLicense: Creative Commons BY-NC-SA 3.0http://creativecommons.org/licenses/by-nc-sa/3.0/it/deed.ensabato 18 maggio 13