Email
Like
Liked ×
Save
Private Content
Embed
Loading embed code…
We have emailed the verification/download link to "".
Login to your email and click the link to download the file directly.

To request the link at a different email address, update it here. Close
Validation messages. Success message. Fail message.

Check your bulk/spam folders if you can't find our mail.
Loading

+ Follow

Data Abstraction for Large Web Applications

by brandonsavage

1,443 views

Accessibility

View text version

Upload Details

Uploaded via SlideShare as Apple Keynote

Usage Rights

File a copyright complaint

2 Embeds 536

http://www.brandonsavage.net	530
http://www.scoop.it	6

Statistics

Likes: 0
Downloads: 6
Comments: 0
Embed Views: 536
Views on SlideShare: 907
Total Views: 1,443

\n
\n
\n
\n
Years and years ago, when the web was young, state was maintained simply by the creation of a database. Web applications were mostly small, and databases could easily handle the traffic that was being sent their way. Most of us learned how to write web applications against a database. Most of us used the “LAMP stack” or Linux Apache MySQL PHP.\n
As the web grew up, and grew bigger, methods for obtaining, storing and using data changed.\n\nDevelopers began using data sources provided by others, first over SOAP then REST. Other data stores like NoSQL, Redis, Elastic Search and Memcache came along to complicate things. \n\nIt was no longer all about the database. The database was just one piece of the puzzle.\n
Yet if we take a good look at most of the frameworks available, they’re database-centric. For a long time, Doctrine support for other data layers was non-existent. Support for something other than a database in Django is non-existent. We still think in a database-centric way. Or data layers are still database-focused.\n
The bottom line: we need to change our thinking.\n\nDatabases are not it. Even for applications that start against a database (and that’s most if not all of them), we need to think about the other ways that we’ll ingest data.\n
This lesson was painful for those of us working on Socorro. Initially built as a database-centric application we’ve slowly expanded our technology stack as new needs have arisen. While much of our webapp data comes from Postgres, we’ve begun a process of moving our data layer to a more source-agnostic middleware layer.\n
\n
It’s clear for us that a database centric model doesn’t work anymore. We can’t think of data in concepts of rows and columns. It doesn’t work like that. \n\nSo how do we solve this problem?\n
\n
Large web applications don’t pursue abstraction as an art form. They pursue it as a necessity. Failing to properly abstract a large web application can result in catastrophic failure. It is therefore important to abstract the layer that gets data from a data storage unit from the layers that use the data.\n\nHere’s an example...\n
When programmers are in a hurry they often don’t take the time to abstract their code in a way that makes it easy to come along later and make changes. I’ve seen this example hundreds of time in codebases I’ve worked on; many of you probably have too. But the problem here is that if ever the data source changes from some SQL-based database to something else, a programmer will have to rewrite the logic here and everywhere else all over again. This makes the cost of transition much higher than it has to be.\n
When programmers are in a hurry they often don’t take the time to abstract their code in a way that makes it easy to come along later and make changes. I’ve seen this example hundreds of time in codebases I’ve worked on; many of you probably have too. But the problem here is that if ever the data source changes from some SQL-based database to something else, a programmer will have to rewrite the logic here and everywhere else all over again. This makes the cost of transition much higher than it has to be.\n
It would make good sense to therefore abstract the process of \n
We should instead use adapters to query the data and return it in an agreed upon format. The processing takes place elsewhere.\n
\n
NAP story. Data layer Postgres focused.\n
When the retrieval and processing are combined, it makes it that much harder to remove one from the other in the future.\n
\n
\n
\n
\n
\n
\n
When you think in terms of actions, rather than data sources, you don’t care what happens behind the scenes. Instead, you start caring about the finished product. In Socorro, we have reports that use both Hbase and Postgres data. If we cared about the data source, we’d have many more calls than we need.\n
\n
If we use JSON as a standard data format throughout our app, we can construct generic objects easily without worrying about what methods are automatically available to us.\n
\n
Rather than relying upon model-constructed or ORM-built objects, we should create our own when and if the need arises. \n
It’s okay to process the results from a database query into some standard format or create an object using the data. But once the data has been retrieved, it should be pushed into a standard format that can be used in the app without caring about what the data source was.\n
Developers are drawn to things that are new, cool, or otherwise unique and special. But it’s important to use the correct storage medium for development.\n
\n
\n
\n
Socorro uses ElasticSearch (not a NoSQL database) and Hbase. We should have used Cassandra, but we have Hbase instead.\n
\n
External APIs, the file system, all are valid data storage mechanisms. Just because we write database-driven applications doesn’t mean our data storage has to be entirely a database. A REST API to an external resource is a valid data storage mechanism, that isn’t database-driven (at least as far as your app is concerned).\n
\n
\n
\n

Data Abstraction for Large Web Applications Presentation Transcript

Data Abstraction forLarge Web Applications By Brandon Savage
Who Am I?• Software developer at Mozilla working on Socorro• Author of the PHP Playbook• Former frequent blogger on PHP topics• Private pilot in my spare time
Data Abstraction For LARGE Web Applications
No magic bullets
Once upon a time...In a galaxy far far away...
Eventually the web grew up. And grew larger.
Most webapps still start asthough they’ll always use a database.
We need to change our thinking.
Socorro
Socorro Data Sources• Postgres• REST API (Middleware)• Elastic Search• Hbase• Bugzilla REST API• Memcache
A database-centric model just doesn’t work anymore.
Solving the problem• Separate the use of data from the retrieval of data.• Think in terms of actions.• Build our applications to be storage agnostic.• Use the correct data storage medium.
#1 Separate the use of data from the retrieval of data
<?phpclass MainPage_Controller {/* ... */public function do_something(){ /* ... */ $sql = ‘SELECT * FROM database”; $results = $this->execute($sql); return $this->executeView(‘index’, array(‘results’ => $results)); }}
<?phpclass Data_Model {/* ... */public function get_some_data() { $sql = ‘SELECT * FROM database”; $results = $this->execute($sql); /** process results **/ return $processedResults }}
Processing the data is a separate layer.
<?phpclass Data_Model {/* ... */ public function getSomeData() { $data = $this->adapter->queryData();! /** process data here **/! return $processedData; }}class Data_Model_Adapter extends MySQL_Adapter implements Adapter{ public function queryData() { $sql = ‘SELECT * FROM table’; /** turn into common format **/ return $commonFormatData; }}
Swapping out data sources becomes very simple.
A cautionary tale
Move to middleware in Socorro
Make life easier on yourself: do it right the ﬁrst time!
#2 Think in terms of actions.
Actions move beyond SELECT,INSERT, UPDATE and DELETE.
Domain Modeling:“What are you modeling?”
What do I want? What do I need?What does this data represent?
Django Models: One model per table.All methods relate to SQL. That sucks.
<?phpabstract class User_Model {public function loadUser();public function authenticateUser();public function showUserPhones();}
#3 Build our applications to be storage agnostic
Use a standard data format
stdClass()
Create custom objects for typehinting or additional methods
Avoid expecting built-ins like PDOStatement and MongoCursor outside retrieval layer
#4 Use the correct storage medium.
Example: memcache isn’t for long-term storage.
Example: MongoDB is not for relational data storage.
Relational data goes in relational databases!
Choose the correct NoSQL database for your needs.
Availability, reliability, and consistency. Pick two.
Consider data storage that isn’t a database at all.
Alternative data options• Elastic Search• Redis• S3• The File System (Yes! It still exists!)
Fix it now or ﬁx it later.But you will have to ﬁx it.
Question time

Data Abstraction for Large Web Applications

by brandonsavage

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 536

Statistics

Data Abstraction for Large Web Applications Presentation Transcript

Let LinkedIn power your SlideShare experience

Let LinkedIn power your SlideShare experience

Customize SlideShare content based on your interests

Keep up to date when your LinkedIn contacts post on SlideShare