SlideShare a Scribd company logo
1 of 42
Data Abstraction for
Large Web Applications
       By Brandon Savage
Who Am I?

•   Software developer at Mozilla working on Socorro

•   Author of the PHP Playbook

•   Former frequent blogger on PHP topics

•   Private pilot in my spare time
Data Abstraction For LARGE
      Web Applications
No magic bullets
Once upon a time...

In a galaxy far far away...
Eventually the web grew up.

     And grew larger.
Most webapps still start as
though they’ll always use a
        database.
We need to change our
      thinking.
Socorro
Socorro Data Sources
•   Postgres

•   REST API (Middleware)

•   Elastic Search

•   Hbase

•   Bugzilla REST API

•   Memcache
A database-centric model just
   doesn’t work anymore.
Solving the problem

•   Separate the use of data from the retrieval of data.

•   Think in terms of actions.

•   Build our applications to be storage agnostic.

•   Use the correct data storage medium.
#1 Separate the use of data
 from the retrieval of data
<?php
class MainPage_Controller {
/* ... */
public function do_something(){
    /* ... */
    $sql = ‘SELECT * FROM database”;
    $results = $this->execute($sql);
    return $this->executeView(‘index’, array(‘results’ => $results));
  }
}
<?php
class Data_Model {
/* ... */
public function get_some_data() {
    $sql = ‘SELECT * FROM database”;
    $results = $this->execute($sql);
    /** process results **/
    return $processedResults
  }
}
Processing the data is a
    separate layer.
<?php
class Data_Model {
/* ... */
  public function getSomeData() {
    $data = $this->adapter->queryData();
!   /** process data here **/
!   return $processedData;
  }
}

class Data_Model_Adapter extends MySQL_Adapter implements Adapter
{
  public function queryData() {
      $sql = ‘SELECT * FROM table’;
      /** turn into common format **/
      return $commonFormatData;
  }
}
Swapping out data sources
  becomes very simple.
A cautionary tale
Move to middleware in Socorro
Make life easier on yourself: do
    it right the first time!
#2 Think in terms of actions.
Actions move beyond SELECT,
INSERT, UPDATE and DELETE.
Domain Modeling:
“What are you modeling?”
What do I want?
      What do I need?
What does this data represent?
Django Models:
   One model per table.
All methods relate to SQL.
       That sucks.
<?php
abstract class User_Model {

public function loadUser();

public function authenticateUser();

public function showUserPhones();

}
#3 Build our applications to be
       storage agnostic
Use a standard data format
stdClass()
Create custom objects for
 typehinting or additional
         methods
Avoid expecting built-ins like
    PDOStatement and
  MongoCursor outside
       retrieval layer
#4 Use the correct
 storage medium.
Example: memcache isn’t for
    long-term storage.
Example: MongoDB is not for
   relational data storage.
Relational data goes in relational
           databases!
Choose the correct NoSQL
 database for your needs.
Availability, reliability, and
      consistency.

         Pick two.
Consider data storage that isn’t
      a database at all.
Alternative data options

•   Elastic Search

•   Redis

•   S3

•   The File System (Yes! It still exists!)
Fix it now or fix it later.

But you will have to fix it.
Question time

More Related Content

What's hot

Advance java session 16
Advance java session 16Advance java session 16
Advance java session 16Smita B Kumar
 
Web Programming - 5 Passing and Request Data
Web Programming - 5 Passing and Request DataWeb Programming - 5 Passing and Request Data
Web Programming - 5 Passing and Request DataAndiNurkholis1
 
Entities in Drupal 8 - Drupal Tech Talk - Bart Feenstra
Entities in Drupal 8 - Drupal Tech Talk - Bart FeenstraEntities in Drupal 8 - Drupal Tech Talk - Bart Feenstra
Entities in Drupal 8 - Drupal Tech Talk - Bart FeenstraTriquanta
 
Building nTier Applications with Entity Framework Services (Part 1)
Building nTier Applications with Entity Framework Services (Part 1)Building nTier Applications with Entity Framework Services (Part 1)
Building nTier Applications with Entity Framework Services (Part 1)David McCarter
 
An Introduction to Spring Data
An Introduction to Spring DataAn Introduction to Spring Data
An Introduction to Spring DataOliver Gierke
 
[Mas 500] Data Basics
[Mas 500] Data Basics[Mas 500] Data Basics
[Mas 500] Data Basicsrahulbot
 
Entity Framework Database and Code First
Entity Framework Database and Code FirstEntity Framework Database and Code First
Entity Framework Database and Code FirstJames Johnson
 
01 nosql and multi model database
01   nosql and multi model database01   nosql and multi model database
01 nosql and multi model databaseMahdi Atawneh
 
Big data technologies and databases
Big data technologies and databasesBig data technologies and databases
Big data technologies and databasesHariniA7
 
Spring Test DBUnit
Spring Test DBUnitSpring Test DBUnit
Spring Test DBUnitJaran Flaath
 
Do’s and don’ts of a hybrid environment
Do’s and don’ts of a hybrid environmentDo’s and don’ts of a hybrid environment
Do’s and don’ts of a hybrid environmentRick Vasquez
 
Tech Gupshup Meetup On MongoDB - 24/06/2016
Tech Gupshup Meetup On MongoDB - 24/06/2016Tech Gupshup Meetup On MongoDB - 24/06/2016
Tech Gupshup Meetup On MongoDB - 24/06/2016Mukesh Tilokani
 
Appache Cassandra
Appache Cassandra  Appache Cassandra
Appache Cassandra nehabsairam
 

What's hot (20)

Advance java session 16
Advance java session 16Advance java session 16
Advance java session 16
 
Web Programming - 5 Passing and Request Data
Web Programming - 5 Passing and Request DataWeb Programming - 5 Passing and Request Data
Web Programming - 5 Passing and Request Data
 
Survey on NoSQL integration
Survey on NoSQL integrationSurvey on NoSQL integration
Survey on NoSQL integration
 
Entities in Drupal 8 - Drupal Tech Talk - Bart Feenstra
Entities in Drupal 8 - Drupal Tech Talk - Bart FeenstraEntities in Drupal 8 - Drupal Tech Talk - Bart Feenstra
Entities in Drupal 8 - Drupal Tech Talk - Bart Feenstra
 
Building nTier Applications with Entity Framework Services (Part 1)
Building nTier Applications with Entity Framework Services (Part 1)Building nTier Applications with Entity Framework Services (Part 1)
Building nTier Applications with Entity Framework Services (Part 1)
 
An Introduction to Spring Data
An Introduction to Spring DataAn Introduction to Spring Data
An Introduction to Spring Data
 
[Mas 500] Data Basics
[Mas 500] Data Basics[Mas 500] Data Basics
[Mas 500] Data Basics
 
Entity Framework Database and Code First
Entity Framework Database and Code FirstEntity Framework Database and Code First
Entity Framework Database and Code First
 
#MongoDB indexes
#MongoDB indexes#MongoDB indexes
#MongoDB indexes
 
NOSQL vs SQL
NOSQL vs SQLNOSQL vs SQL
NOSQL vs SQL
 
Mysql
MysqlMysql
Mysql
 
SQL & NoSQL
SQL & NoSQLSQL & NoSQL
SQL & NoSQL
 
01 nosql and multi model database
01   nosql and multi model database01   nosql and multi model database
01 nosql and multi model database
 
Big data technologies and databases
Big data technologies and databasesBig data technologies and databases
Big data technologies and databases
 
Spring Test DBUnit
Spring Test DBUnitSpring Test DBUnit
Spring Test DBUnit
 
Do’s and don’ts of a hybrid environment
Do’s and don’ts of a hybrid environmentDo’s and don’ts of a hybrid environment
Do’s and don’ts of a hybrid environment
 
Tech Gupshup Meetup On MongoDB - 24/06/2016
Tech Gupshup Meetup On MongoDB - 24/06/2016Tech Gupshup Meetup On MongoDB - 24/06/2016
Tech Gupshup Meetup On MongoDB - 24/06/2016
 
Appache Cassandra
Appache Cassandra  Appache Cassandra
Appache Cassandra
 
SQL vs NoSQL
SQL vs NoSQLSQL vs NoSQL
SQL vs NoSQL
 
Multi model-databases
Multi model-databasesMulti model-databases
Multi model-databases
 

Viewers also liked

Applications for the Enterprise with PHP (CPEurope)
Applications for the Enterprise with PHP (CPEurope)Applications for the Enterprise with PHP (CPEurope)
Applications for the Enterprise with PHP (CPEurope)Robert Lemke
 
Beyond MVC: from Model to Domain
Beyond MVC: from Model to DomainBeyond MVC: from Model to Domain
Beyond MVC: from Model to DomainJeremy Cook
 
Software Engineering In PHP
Software Engineering In PHPSoftware Engineering In PHP
Software Engineering In PHPRalph Schindler
 
PHP deployment, 2016 flavor - cakefest 2016
PHP deployment, 2016 flavor - cakefest 2016PHP deployment, 2016 flavor - cakefest 2016
PHP deployment, 2016 flavor - cakefest 2016Quentin Adam
 
Advanced PHP: Design Patterns - Dennis-Jan Broerse
Advanced PHP: Design Patterns - Dennis-Jan BroerseAdvanced PHP: Design Patterns - Dennis-Jan Broerse
Advanced PHP: Design Patterns - Dennis-Jan Broersedpc
 
Proved PHP Design Patterns for Data Persistence
Proved PHP Design Patterns for Data PersistenceProved PHP Design Patterns for Data Persistence
Proved PHP Design Patterns for Data PersistenceGjero Krsteski
 
Taming the resource tiger
Taming the resource tigerTaming the resource tiger
Taming the resource tigerElizabeth Smith
 
Building Data Mapper PHP5
Building Data Mapper PHP5Building Data Mapper PHP5
Building Data Mapper PHP5Vance Lucas
 
Asynchronous I/O in PHP
Asynchronous I/O in PHPAsynchronous I/O in PHP
Asynchronous I/O in PHPThomas Weinert
 
Driving Design through Examples
Driving Design through ExamplesDriving Design through Examples
Driving Design through ExamplesCiaranMcNulty
 
PHP Strings and Patterns
PHP Strings and PatternsPHP Strings and Patterns
PHP Strings and PatternsHenry Osborne
 
Some REST Design Patterns (and Anti-Patterns) - SOA Symposium 2009
Some REST Design Patterns (and Anti-Patterns) - SOA Symposium 2009Some REST Design Patterns (and Anti-Patterns) - SOA Symposium 2009
Some REST Design Patterns (and Anti-Patterns) - SOA Symposium 2009Cesare Pautasso
 
Enterprise PHP: mappers, models and services
Enterprise PHP: mappers, models and servicesEnterprise PHP: mappers, models and services
Enterprise PHP: mappers, models and servicesAaron Saray
 
ORM: Object-relational mapping
ORM: Object-relational mappingORM: Object-relational mapping
ORM: Object-relational mappingAbhilash M A
 
Elegant Ways of Handling PHP Errors and Exceptions
Elegant Ways of Handling PHP Errors and ExceptionsElegant Ways of Handling PHP Errors and Exceptions
Elegant Ways of Handling PHP Errors and ExceptionsZendCon
 
Design Patterns avec PHP 5.3, Symfony et Pimple
Design Patterns avec PHP 5.3, Symfony et PimpleDesign Patterns avec PHP 5.3, Symfony et Pimple
Design Patterns avec PHP 5.3, Symfony et PimpleHugo Hamon
 
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...Aaron Saray
 
Database Design Patterns
Database Design PatternsDatabase Design Patterns
Database Design PatternsHugo Hamon
 
Patterns of Enterprise Application Architecture (by example)
Patterns of Enterprise Application Architecture (by example)Patterns of Enterprise Application Architecture (by example)
Patterns of Enterprise Application Architecture (by example)Paulo Gandra de Sousa
 
Writing and using php streams and sockets
Writing and using php streams and socketsWriting and using php streams and sockets
Writing and using php streams and socketsElizabeth Smith
 

Viewers also liked (20)

Applications for the Enterprise with PHP (CPEurope)
Applications for the Enterprise with PHP (CPEurope)Applications for the Enterprise with PHP (CPEurope)
Applications for the Enterprise with PHP (CPEurope)
 
Beyond MVC: from Model to Domain
Beyond MVC: from Model to DomainBeyond MVC: from Model to Domain
Beyond MVC: from Model to Domain
 
Software Engineering In PHP
Software Engineering In PHPSoftware Engineering In PHP
Software Engineering In PHP
 
PHP deployment, 2016 flavor - cakefest 2016
PHP deployment, 2016 flavor - cakefest 2016PHP deployment, 2016 flavor - cakefest 2016
PHP deployment, 2016 flavor - cakefest 2016
 
Advanced PHP: Design Patterns - Dennis-Jan Broerse
Advanced PHP: Design Patterns - Dennis-Jan BroerseAdvanced PHP: Design Patterns - Dennis-Jan Broerse
Advanced PHP: Design Patterns - Dennis-Jan Broerse
 
Proved PHP Design Patterns for Data Persistence
Proved PHP Design Patterns for Data PersistenceProved PHP Design Patterns for Data Persistence
Proved PHP Design Patterns for Data Persistence
 
Taming the resource tiger
Taming the resource tigerTaming the resource tiger
Taming the resource tiger
 
Building Data Mapper PHP5
Building Data Mapper PHP5Building Data Mapper PHP5
Building Data Mapper PHP5
 
Asynchronous I/O in PHP
Asynchronous I/O in PHPAsynchronous I/O in PHP
Asynchronous I/O in PHP
 
Driving Design through Examples
Driving Design through ExamplesDriving Design through Examples
Driving Design through Examples
 
PHP Strings and Patterns
PHP Strings and PatternsPHP Strings and Patterns
PHP Strings and Patterns
 
Some REST Design Patterns (and Anti-Patterns) - SOA Symposium 2009
Some REST Design Patterns (and Anti-Patterns) - SOA Symposium 2009Some REST Design Patterns (and Anti-Patterns) - SOA Symposium 2009
Some REST Design Patterns (and Anti-Patterns) - SOA Symposium 2009
 
Enterprise PHP: mappers, models and services
Enterprise PHP: mappers, models and servicesEnterprise PHP: mappers, models and services
Enterprise PHP: mappers, models and services
 
ORM: Object-relational mapping
ORM: Object-relational mappingORM: Object-relational mapping
ORM: Object-relational mapping
 
Elegant Ways of Handling PHP Errors and Exceptions
Elegant Ways of Handling PHP Errors and ExceptionsElegant Ways of Handling PHP Errors and Exceptions
Elegant Ways of Handling PHP Errors and Exceptions
 
Design Patterns avec PHP 5.3, Symfony et Pimple
Design Patterns avec PHP 5.3, Symfony et PimpleDesign Patterns avec PHP 5.3, Symfony et Pimple
Design Patterns avec PHP 5.3, Symfony et Pimple
 
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
 
Database Design Patterns
Database Design PatternsDatabase Design Patterns
Database Design Patterns
 
Patterns of Enterprise Application Architecture (by example)
Patterns of Enterprise Application Architecture (by example)Patterns of Enterprise Application Architecture (by example)
Patterns of Enterprise Application Architecture (by example)
 
Writing and using php streams and sockets
Writing and using php streams and socketsWriting and using php streams and sockets
Writing and using php streams and sockets
 

Similar to Data Abstraction for Large Web Applications

Spring data presentation
Spring data presentationSpring data presentation
Spring data presentationOleksii Usyk
 
Elements for an iOS Backend
Elements for an iOS BackendElements for an iOS Backend
Elements for an iOS BackendLaurent Cerveau
 
Minerva: Drill Storage Plugin for IPFS
Minerva: Drill Storage Plugin for IPFSMinerva: Drill Storage Plugin for IPFS
Minerva: Drill Storage Plugin for IPFSBowenDing4
 
Drupal performance and scalability
Drupal performance and scalabilityDrupal performance and scalability
Drupal performance and scalabilityTwinbit
 
Java Developers, make the database work for you (NLJUG JFall 2010)
Java Developers, make the database work for you (NLJUG JFall 2010)Java Developers, make the database work for you (NLJUG JFall 2010)
Java Developers, make the database work for you (NLJUG JFall 2010)Lucas Jellema
 
Staying Sane with Drupal NEPHP
Staying Sane with Drupal NEPHPStaying Sane with Drupal NEPHP
Staying Sane with Drupal NEPHPOscar Merida
 
BackboneJS Training - Giving Backbone to your applications
BackboneJS Training - Giving Backbone to your applicationsBackboneJS Training - Giving Backbone to your applications
BackboneJS Training - Giving Backbone to your applicationsJoseph Khan
 
Microsoft Entity Framework
Microsoft Entity FrameworkMicrosoft Entity Framework
Microsoft Entity FrameworkMahmoud Tolba
 
Machine Learning with ML.NET and Azure - Andy Cross
Machine Learning with ML.NET and Azure - Andy CrossMachine Learning with ML.NET and Azure - Andy Cross
Machine Learning with ML.NET and Azure - Andy CrossAndrew Flatters
 
[2015/2016] Local data storage for web-based mobile apps
[2015/2016] Local data storage for web-based mobile apps[2015/2016] Local data storage for web-based mobile apps
[2015/2016] Local data storage for web-based mobile appsIvano Malavolta
 
CrawlerLD - Distributed crawler for linked data
CrawlerLD - Distributed crawler for linked dataCrawlerLD - Distributed crawler for linked data
CrawlerLD - Distributed crawler for linked dataRaphael do Vale
 
Java Web Programming on Google Cloud Platform [2/3] : Datastore
Java Web Programming on Google Cloud Platform [2/3] : DatastoreJava Web Programming on Google Cloud Platform [2/3] : Datastore
Java Web Programming on Google Cloud Platform [2/3] : DatastoreIMC Institute
 
Being RDBMS Free -- Alternate Approaches to Data Persistence
Being RDBMS Free -- Alternate Approaches to Data PersistenceBeing RDBMS Free -- Alternate Approaches to Data Persistence
Being RDBMS Free -- Alternate Approaches to Data PersistenceDavid Hoerster
 
Core data WIPJam workshop @ MWC'14
Core data WIPJam workshop @ MWC'14Core data WIPJam workshop @ MWC'14
Core data WIPJam workshop @ MWC'14Diego Freniche Brito
 
Spring Data - Intro (Odessa Java TechTalks)
Spring Data - Intro (Odessa Java TechTalks)Spring Data - Intro (Odessa Java TechTalks)
Spring Data - Intro (Odessa Java TechTalks)Igor Anishchenko
 
Academy PRO: HTML5 Data storage
Academy PRO: HTML5 Data storageAcademy PRO: HTML5 Data storage
Academy PRO: HTML5 Data storageBinary Studio
 

Similar to Data Abstraction for Large Web Applications (20)

Spring data presentation
Spring data presentationSpring data presentation
Spring data presentation
 
Elements for an iOS Backend
Elements for an iOS BackendElements for an iOS Backend
Elements for an iOS Backend
 
Minerva: Drill Storage Plugin for IPFS
Minerva: Drill Storage Plugin for IPFSMinerva: Drill Storage Plugin for IPFS
Minerva: Drill Storage Plugin for IPFS
 
Data access
Data accessData access
Data access
 
Drupal performance and scalability
Drupal performance and scalabilityDrupal performance and scalability
Drupal performance and scalability
 
Java Developers, make the database work for you (NLJUG JFall 2010)
Java Developers, make the database work for you (NLJUG JFall 2010)Java Developers, make the database work for you (NLJUG JFall 2010)
Java Developers, make the database work for you (NLJUG JFall 2010)
 
Staying Sane with Drupal NEPHP
Staying Sane with Drupal NEPHPStaying Sane with Drupal NEPHP
Staying Sane with Drupal NEPHP
 
BackboneJS Training - Giving Backbone to your applications
BackboneJS Training - Giving Backbone to your applicationsBackboneJS Training - Giving Backbone to your applications
BackboneJS Training - Giving Backbone to your applications
 
Microsoft Entity Framework
Microsoft Entity FrameworkMicrosoft Entity Framework
Microsoft Entity Framework
 
Machine Learning with ML.NET and Azure - Andy Cross
Machine Learning with ML.NET and Azure - Andy CrossMachine Learning with ML.NET and Azure - Andy Cross
Machine Learning with ML.NET and Azure - Andy Cross
 
[2015/2016] Local data storage for web-based mobile apps
[2015/2016] Local data storage for web-based mobile apps[2015/2016] Local data storage for web-based mobile apps
[2015/2016] Local data storage for web-based mobile apps
 
CrawlerLD - Distributed crawler for linked data
CrawlerLD - Distributed crawler for linked dataCrawlerLD - Distributed crawler for linked data
CrawlerLD - Distributed crawler for linked data
 
Java Web Programming on Google Cloud Platform [2/3] : Datastore
Java Web Programming on Google Cloud Platform [2/3] : DatastoreJava Web Programming on Google Cloud Platform [2/3] : Datastore
Java Web Programming on Google Cloud Platform [2/3] : Datastore
 
La sql
La sqlLa sql
La sql
 
Being RDBMS Free -- Alternate Approaches to Data Persistence
Being RDBMS Free -- Alternate Approaches to Data PersistenceBeing RDBMS Free -- Alternate Approaches to Data Persistence
Being RDBMS Free -- Alternate Approaches to Data Persistence
 
Core data WIPJam workshop @ MWC'14
Core data WIPJam workshop @ MWC'14Core data WIPJam workshop @ MWC'14
Core data WIPJam workshop @ MWC'14
 
Spring Data - Intro (Odessa Java TechTalks)
Spring Data - Intro (Odessa Java TechTalks)Spring Data - Intro (Odessa Java TechTalks)
Spring Data - Intro (Odessa Java TechTalks)
 
Midao JDBC presentation
Midao JDBC presentationMidao JDBC presentation
Midao JDBC presentation
 
Dao example
Dao exampleDao example
Dao example
 
Academy PRO: HTML5 Data storage
Academy PRO: HTML5 Data storageAcademy PRO: HTML5 Data storage
Academy PRO: HTML5 Data storage
 

Recently uploaded

Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...AliaaTarek5
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesThousandEyes
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditSkynet Technologies
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rick Flair
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 

Recently uploaded (20)

Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance Audit
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 

Data Abstraction for Large Web Applications

Editor's Notes

  1. \n
  2. \n
  3. \n
  4. \n
  5. Years and years ago, when the web was young, state was maintained simply by the creation of a database. Web applications were mostly small, and databases could easily handle the traffic that was being sent their way. Most of us learned how to write web applications against a database. Most of us used the &amp;#x201C;LAMP stack&amp;#x201D; or Linux Apache MySQL PHP.\n
  6. As the web grew up, and grew bigger, methods for obtaining, storing and using data changed.\n\nDevelopers began using data sources provided by others, first over SOAP then REST. Other data stores like NoSQL, Redis, Elastic Search and Memcache came along to complicate things. \n\nIt was no longer all about the database. The database was just one piece of the puzzle.\n
  7. Yet if we take a good look at most of the frameworks available, they&amp;#x2019;re database-centric. For a long time, Doctrine support for other data layers was non-existent. Support for something other than a database in Django is non-existent. We still think in a database-centric way. Or data layers are still database-focused.\n
  8. The bottom line: we need to change our thinking.\n\nDatabases are not it. Even for applications that start against a database (and that&amp;#x2019;s most if not all of them), we need to think about the other ways that we&amp;#x2019;ll ingest data.\n
  9. This lesson was painful for those of us working on Socorro. Initially built as a database-centric application we&amp;#x2019;ve slowly expanded our technology stack as new needs have arisen. While much of our webapp data comes from Postgres, we&amp;#x2019;ve begun a process of moving our data layer to a more source-agnostic middleware layer.\n
  10. \n
  11. It&amp;#x2019;s clear for us that a database centric model doesn&amp;#x2019;t work anymore. We can&amp;#x2019;t think of data in concepts of rows and columns. It doesn&amp;#x2019;t work like that. \n\nSo how do we solve this problem?\n
  12. \n
  13. Large web applications don&amp;#x2019;t pursue abstraction as an art form. They pursue it as a necessity. Failing to properly abstract a large web application can result in catastrophic failure. It is therefore important to abstract the layer that gets data from a data storage unit from the layers that use the data.\n\nHere&amp;#x2019;s an example...\n
  14. When programmers are in a hurry they often don&amp;#x2019;t take the time to abstract their code in a way that makes it easy to come along later and make changes. I&amp;#x2019;ve seen this example hundreds of time in codebases I&amp;#x2019;ve worked on; many of you probably have too. But the problem here is that if ever the data source changes from some SQL-based database to something else, a programmer will have to rewrite the logic here and everywhere else all over again. This makes the cost of transition much higher than it has to be.\n
  15. When programmers are in a hurry they often don&amp;#x2019;t take the time to abstract their code in a way that makes it easy to come along later and make changes. I&amp;#x2019;ve seen this example hundreds of time in codebases I&amp;#x2019;ve worked on; many of you probably have too. But the problem here is that if ever the data source changes from some SQL-based database to something else, a programmer will have to rewrite the logic here and everywhere else all over again. This makes the cost of transition much higher than it has to be.\n
  16. It would make good sense to therefore abstract the process of \n
  17. We should instead use adapters to query the data and return it in an agreed upon format. The processing takes place elsewhere.\n
  18. \n
  19. NAP story. Data layer Postgres focused.\n
  20. When the retrieval and processing are combined, it makes it that much harder to remove one from the other in the future.\n
  21. \n
  22. \n
  23. \n
  24. \n
  25. \n
  26. \n
  27. When you think in terms of actions, rather than data sources, you don&amp;#x2019;t care what happens behind the scenes. Instead, you start caring about the finished product. In Socorro, we have reports that use both Hbase and Postgres data. If we cared about the data source, we&amp;#x2019;d have many more calls than we need.\n
  28. \n
  29. If we use JSON as a standard data format throughout our app, we can construct generic objects easily without worrying about what methods are automatically available to us.\n
  30. \n
  31. Rather than relying upon model-constructed or ORM-built objects, we should create our own when and if the need arises. \n
  32. It&amp;#x2019;s okay to process the results from a database query into some standard format or create an object using the data. But once the data has been retrieved, it should be pushed into a standard format that can be used in the app without caring about what the data source was.\n
  33. Developers are drawn to things that are new, cool, or otherwise unique and special. But it&amp;#x2019;s important to use the correct storage medium for development.\n
  34. \n
  35. \n
  36. \n
  37. Socorro uses ElasticSearch (not a NoSQL database) and Hbase. We should have used Cassandra, but we have Hbase instead.\n
  38. \n
  39. External APIs, the file system, all are valid data storage mechanisms. Just because we write database-driven applications doesn&amp;#x2019;t mean our data storage has to be entirely a database. A REST API to an external resource is a valid data storage mechanism, that isn&amp;#x2019;t database-driven (at least as far as your app is concerned).\n
  40. \n
  41. \n
  42. \n