Caching and data analysis will move your Symfony2 application to the next level

  • 6,918 views
Uploaded on

If you don't ignore data you could find a business value.

If you don't ignore data you could find a business value.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
6,918
On Slideshare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
28
Comments
0
Likes
10

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • OK the real title is zzz and blblabla will move your something to the next level
  • This is a not invented here talk, The topics I ‘ d like to discuss are all well known, I have not invented anything please go there to read the references such as the talk of Jordi etc. I’m here to share my knowledge SEE THE REFERENCES
  • I love the Symfony2 framework, G ave us: a breath of FRESHNESS to the whole php world , GREAT DOCUMENTATION/ NOT FRAMEWORK but BESTPRACTICES , great community / METHODOLOGIES gave as a real dark passenger, dark passenger means that there ’ s a little genious/geek that is on your shoulder when you develop B ut this dark passengere LOOK AT YOU not at the code at YOU M y dark passenger speaks FRANCE You ’ ave not Follow^ng coding standards, dependency injection. When I met this small dark passenger he grows up, I feel better my libraries are more usable. I realized that bigger is your dark passenger and the more you are a cool developer. S o at terravision we do love sf2 and it ’ s fabpot passenger.
  • OK the real title is zzz and blblabla will move your something to the next level
  • OK the real title is zzz and blblabla will move your something to the next level
  • We are developer our ecosystem create a lot of data.    [ok hey liuggio what problem do you have with data?] -------- data is difficult to measure, difficult to read, data is time consuming. ------- data is boring,-- data sucks * (sometimes) Logging ---------- If you think that logging is a file with a lot of unreadable information. Logging will have a Matrix effect with data. You don’t’need logging only when something bad happen How many of you use logging, How many of you thinks that logging is a file with too much iformation in it… Logging is a black hole in the developers career,
  • [ok hey liuggio what problem do you have with data?] data is difficult to measure, difficult to read, data is time consuming. ------- data is boring,-- data sucks * (sometimes)
  • what is data? different information from ----- Note riunione (5/15/13 08:26) ----- data gathers
  • what is data? different information from ----- Note riunione (5/15/13 08:26) ----- data gathers
  • I do we do, we used to ignore the most of data that we receive. But sometimes we track it, maybe passively but we keep track of some events for example logging
  • What do you do? 1. I ignore them 2. I ignore them but I have a clear coscience (because there ’ s a file for it) 3. I ignore them mmm but I ’ ll use regex when an halling customer will call (if you have a problem that you want to solve with the regex, you have 2 problem) I f you try to solve a problem with a regular expression, you have 2 problems. 4. I ignore the log, regex and the customer. S ometimes you need to investigate, A lways you need to be alerted when .. Y ou want to see graphes In the daylife is very important have PULL information like a big lcd monitor over there
  • Why should we use all that data? What is business value: all the things that produces money W e are coder, we love write code, but our code produce business, W e use unit testing in order to have maintainability of the code, this could be a business value we use function testing, because we want to add new features easily this could be a business value A nd we want that if angry customer will call you we have already the answer Y ou want a fast user experience Y ou want to know what is happening in your application --------- code is not a trophy functionality measure it in order to know face the reality measure on production
  • If you don’t understand this slide you are not stupid maybe because you never seen the ‘ hitchhiker's guide to the galaxy ’ movie. Hitchhikers
  • We use phpunit in order to easily add new feature and have code manintainability not because is cool ----- Note riunione (5/16/13 17:55) ----- each company has it's own business value.
  • All of this are some business value that you could achieve tracking data. Business Value is everything that produces € ( For example Unit testing produce business value (we do use unit testing in order to have maintainability of the code (easy to add a new feature) that is a business value.) Each company has its own definition of business Value, but we could define some needs that could increment your Business Value. (could because if you don ’ t care it ’ s ok) Store log and error data (investigate) Complex query on past events Easily Locate the bugs Know Servers status: healthy/resources Know Application status: Responding as it should? Know what is happening in my ecosystem Understand why and where the app. is slow Analyze the behaviour when it ‘ s running I want to watch data I want to be alerted Fast website I want to know how if after a certain event “ how much ” something else changed Interpolate data from different sources               (after the deploy, how many t-shirt were bought?) We added to many information here Investigate means that we need to collect and create relations among different logs and make queries Be alerted means that we want on different context an action Visualize means meseaure and draw from the data Noi creiamo cose  e dobbiamo interessarci del business value, il nostro codice produce business value add new features easily business value  -> all the things that produces €€ code is not a trophy functionality T here ’ s a name for everything … measure it in order to know face the reality measure on production ----- Note riunione (5/14/13 00:11) ----- satisfy ----- Note riunione (5/16/13 17:55) ----- this is business value from the developer side and data
  • In order to do this we need to add different actions ----- Note riunione (5/15/13 08:50) ----- perform actions
  • In order to do this we need to add different actions ----- Note riunione (5/16/13 11:20) ----- in here there's the talk title: from metrics and optimizing you obtain a business value
  • I’d like to talk with you about Metrics Measure: WE WANT TO FACE THE REALITY The terms metric and measure have some overlap. We use measure for more concrete or objective attributes and metric for more abstract, higher-level, or somewhat subjective attributes. For instance, lines of code (LOC) is a measure: it is objective and concrete The robustness is a metrics M easurment are the basis for the metrics Measures help us approximate less tangible metrics.
  • Metrics is a link of two worlds technical and business vision of IT.    [yes ok but what is?] ------ definition of metrics … sucks is so general and everybody has their own, ----- Metrics -- definition       metrics gives you qualitatively information (and measure quantity information) ----- Note riunione (5/15/13 09:15) ----- line of code examples measure, metrics
  • A quantitatively expressed reduction of uncertainty based on one or more observations. Think a measure as a quantitative number (image you could measure anything) This “ uncertainty reduction ” point of view is what is critical to busi- ness reduction can be worth millions. HOW TO MEASURE ANYTHING Hubbard measurement process of experimentally obtaining one or more quantity values that can reasonably be attributed to a quantity
  • When you don’t know something you could ask to them. What metrics is
  • Metrics number visual ops it’s a ninja territory?
  • Why metrics is so important? Is a join of two worlds
  • Metrics in order to produce Graphs → Decision decision because you could Collects and aggregates from different sources sources of metrics: webservers, application, cache server, averything! Better decision using numbers  -- @coda
  • How to do metrics? StatsD We want a great tool in order to add easily new metrics. From every envoirnment, every language every where.
  • Calculate the amout of php ram during it’s peak time Ready for the WTF effect? This is a “timer”
  • Each stat is in its own " bucket ” the KEY, They are not predefined anywhere. Buckets can be named anything that will translate to Graphite (periods make folders, etc) After the flush interval timeout (defined by config.flushInterval, default 10 seconds), stats are aggregated and sent to an upstream backend service.
  • A gauge simply indicates an arbitrary value at a point in time and is the most simple type in StatsD. It just takes any number and ships it to the backend.
  • Counters are simple. It adds a value to a bucket and stays in memory until the flush interval.
  • Timers collects numbers. They does not necessarily need to contain a value of time. You can collect bytes read, number of objects in some storage, or anything that is a number. Useful for: mean/upper/sum/lower/count Operation
  • Timers collects numbers. They does not necessarily need to contain a value of time. You can collect bytes read, number of objects in some storage, or anything that is a number. Useful for: mean/upper/sum/lower/count Operation
  • Here is a golden rule that you find in all talks about metrics I prefer metrics CAN … Because too much not usefull information could go to CONFUSION. == with Metrics you track everythings that moves  [on internet this is like  a cool definition] BUT
  • Se teniamo sempre alto il livello di razionalità questo ci farà dire:
  • È molto
  • Se teniamo sempre alto il livello di razionalità questo ci farà dire:
  • Better decision using numbers  -- @coda Good and Fast decisions help the business
  • Better decision using numbers  -- @coda Good and Fast decisions help the business
  • OK the real title is zzz and blblabla will move your something to the next level
  • Another topic that has a lot of usages is the cache What do you think if I say cache?
  • Another topic that has a lot of usages is the cache What do you think if I say cache?
  • Another topic that has a lot of usages is the cache What do you think if I say cache?
  • ----- Note riunione (5/11/13 21:43) ----- proxy
  • This is a great idea, We are web developer So we need to follow the web protocol Why I didn’t think about it before?
  • ----- Note riunione (5/14/13 00:11) ----- scrivere nei commenti fig!!
  • Another topic that has a lot of usages is the cache What do you think if I say cache?
  • OK the real title is zzz and blblabla will move your something to the next level
  • Y ou could do by yourself This code is problematic, and adds a lot of complexity that makes your unit tests' code even worse. https://github.com/Ocramius/ProxyManager/blob/master/docs/lazy-loading-value-holder.md https://github.com/Ocramius/symfony-docs/blob/931091d8e3aa0b9d4f1393d313388d5ee1a5d47d/components/dependency_injection/lazy_services.rst
  • flush multiples entities      $em->flush(array($entityA, $entityB); paginator      master-slave     Flush to Master Read from Slave join to interface (Target Entity Resolver: /**    @ORM\\ManyToOne(targetEntity="Tvision\\Bundle\\OrderBundle\\Entity\\StatusInterface") **/
  • flush multiples entities      $em->flush(array($entityA, $entityB); paginator      master-slave     Flush to Master Read from Slave join to interface (Target Entity Resolver: /**    @ORM\\ManyToOne(targetEntity="Tvision\\Bundle\\OrderBundle\\Entity\\StatusInterface") **/
  • ----- Note riunione (5/14/13 00:11) ----- levare flush
  • The deferred implicit policy is the default change tracking policy and the most convenient one.property-by-property comparison at commit time and also detects changes to entities or new entities that are referenced by other managed entities ( “ persistence by reachability ” ) Explicit only considers entities that have been explicitly marked for change detection through a call to EntityManager#persist(entity) or through a save cascade.
  • Starting from the daylife in order to undestand UOW ----- Note riunione (5/16/13 11:41) ----- there's a line that could'nt be there
  • T he unit of work class in doctrine ha 3206 lines, is complex UOW start with with init of EM or after a flush. The size of a Unit of Work mainly refers to the number of managed entities Is an Entity managed by UOW when is persisted or when is fetched Identity Map: In this case the Article is accessed from the entity manager twice, but modified in between. How costly a flush operation is, mainly depends on two factors: The size of the EntityManager ’ s current UnitOfWork. The configured change tracking policies You can get the size of a UnitOfWork as follows: <?php $uowSize = $em->getUnitOfWork()->size();
  • T he unit of work class in doctrine ha 3206 lines, is complex UOW start with with init of EM or after a flush. The size of a Unit of Work mainly refers to the number of managed entities Is an Entity managed by UOW when is persisted or when is fetched Identity Map: In this case the Article is accessed from the entity manager twice, but modified in between. How costly a flush operation is, mainly depends on two factors: The size of the EntityManager ’ s current UnitOfWork. The configured change tracking policies You can get the size of a UnitOfWork as follows: <?php $uowSize = $em->getUnitOfWork()->size();
  • T he unit of work class in doctrine ha 3206 lines, is complex UOW start with with init of EM or after a flush. The size of a Unit of Work mainly refers to the number of managed entities Is an Entity managed by UOW when is persisted or when is fetched Identity Map: In this case the Article is accessed from the entity manager twice, but modified in between. How costly a flush operation is, mainly depends on two factors: The size of the EntityManager ’ s current UnitOfWork. The configured change tracking policies You can get the size of a UnitOfWork as follows: <?php $uowSize = $em->getUnitOfWork()->size();
  • The deferred implicit policy is the default change tracking policy and the most convenient one.property-by-property comparison at commit time and also detects changes to entities or new entities that are referenced by other managed entities ( “ persistence by reachability ” ) Explicit only considers entities that have been explicitly marked for change detection through a call to EntityManager#persist(entity) or through a save cascade.
  • Deferred Implicit P ro : persist($entity) is not necessary Cons : bad performance with large UOW Deferred Explicit Pro : flush operation cheaper Cons : DB != UOW spread persists into the code Unit of work now has $entity.title=liuggio loves symfony2 Database has flushed without persist $entity.title=liuggio loves symfony2 Pro : $entityManager->persist($entity) is not necessary Cons : Slowness, Doctrine has to check property-by-property for all the object in the unit of work Explicit Doctrine2 tracking policy 23 April 2013 doctrine2 1 php 9 Which tracking policy are you using? With doctrine you can change the tracking policy for each entity, the tracking policy is the way in which Doctrine understands if a entity property must be ‘ saved ’ in the database. Doctrine definition: “ Change tracking is the process of determining what has changed in managed entities since the last time they were synchronized with the database. ” There are 3 tracking policies: Implicit, Explicit, Nofity. Implicit tracking (the default one) The implicit is the default one and the slowest, you don ’ t need to explicit call ‘ persist ’ eg. // $entity is not a new object, and has $entity->Title = 'liuggio loves tvision.github.io' $entity->setTitle('liuggio loves symfony2'); $entityManager->flush(); echo $entity->getTitle(); // output 'liuggio loves symfony2' Unit of work now has $entity.title=liuggio loves symfony2 Database has flushed without persist $entity.title=liuggio loves symfony2 Pro : $entityManager->persist($entity) is not necessary Cons : Slowness, Doctrine has to check property-by-property for all the object in the unit of work The explicit tracking (the suggested one) You have to explicit flag which object you want to persist, than Doctrine will check property-by-property in order to find with property has changed. Pro : Faster than implicit, must if you have a lot of entities. Cons : the object in the unit of work could be different from that one in the db, not easy to debug. The example
  • Temporarily mark entities as read only. If you have a very large UnitOfWork but know that a large set of entities has not changed, just mark them as read only with $entityManager->getUnitOfWork()->markReadOnly($entity).
  • T he unit of work class in doctrine ha 3206 lines, is complex UOW start with with init of EM or after a flush. The size of a Unit of Work mainly refers to the number of managed entities Is an Entity managed by UOW when is persisted or when is fetched Identity Map: In this case the Article is accessed from the entity manager twice, but modified in between. How costly a flush operation is, mainly depends on two factors: The size of the EntityManager ’ s current UnitOfWork. The configured change tracking policies You can get the size of a UnitOfWork as follows: <?php $uowSize = $em->getUnitOfWork()->size();
  • OK the real title is zzz and blblabla will move your something to the next level
  • I’d like to talk to you about metrics I do love metrics Mr smisth sentence:
  • È molto
  • OK the real title is zzz and blblabla will move your something to the next level
  • OK the real title is zzz and blblabla will move your something to the next level
  • OK the real title is zzz and blblabla will move your something to the next level
  • OK the real title is zzz and blblabla will move your something to the next level
  • OK the real title is zzz and blblabla will move your something to the next level
  • OK the real title is zzz and blblabla will move your something to the next level
  • OK the real title is zzz and blblabla will move your something to the next level
  • OK the real title is zzz and blblabla will move your something to the next level
  • OK the real title is zzz and blblabla will move your something to the next level

Transcript

  • 1. Zzz and bla^3will move yourto the next level.???
  • 2. I have notinvented anythingpleaseread carefullythe referencesnot yet 
  • 3. I’m @liuggio(aka Giulio De Donato)I LOVEand its dark passengerfabpot
  • 4. _ _ _ and _ _ _will move yourto the next level.???
  • 5. _ _ _ and _ _ _will move yourto the next level.Symfony2application
  • 6. proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/photos/455.jpg HTTP/1.1"200 986 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/nav/tab_left_middle.gifHTTP/1.1" 200 1020 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13(KHTML, like Gecko) Version/3.1 Safari/525.13" "-"dyn-530.optus.com.au - - [22/Apr/2009:18:52:51 +1200] "GET /images/photos/5332.jpg HTTP/1.1" 200244 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322)" "-"<?xml version="1.0"?><catalog><book id="bk101"><author>Gambardella, Matthew</author><title>XML Developers Guide</title><genre>Computer</genre><price>44.95</price><publish_date>2000-10-01</publish_date>dhcp-465.telstra.com.au - - [22/Apr/2009:18:52:51@charset "UTF-8";html,body,div,span,applet,object,iframe,h1,h2,h3,h4,+1200] "GET /gallery.php?section=entertainment HTTP/1.0" 200 58636 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en)AppleWebKit/525.13 (KHTML, like Gecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /adclick.php?c=LME2 HTTP/1.1"302 378 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/ad/RH4Y.jpeg HTTP/1.1"200 1031 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/favicon.ico HTTP/1.1" 2002824 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /adclick.php?c=GD52 HTTP/1.1"302 379 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeIn your companyevery minutedata is generateda lot ofdifferenttotally
  • 7. proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/photos/455.jpg HTTP/1.1"200 986 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/nav/tab_left_middle.gifHTTP/1.1" 200 1020 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13(KHTML, like Gecko) Version/3.1 Safari/525.13" "-"dyn-530.optus.com.au - - [22/Apr/2009:18:52:51 +1200] "GET /images/photos/5332.jpg HTTP/1.1" 200244 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322)" "-"dhcp-465.telstra.com.au - - [22/Apr/2009:18:52:51 +1200] "GET /gallery.php?section=entertainmentHTTP/1.0" 200 58636 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13(KHTML, like Gecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /adclick.php?c=LME2 HTTP/1.1"302 378 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/ad/RH4Y.jpeg HTTP/1.1"200 1031 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/favicon.ico HTTP/1.1" 2002824 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /adclick.php?c=GD52 HTTP/1.1"302 379 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /stories.php?date=yesterdayHTTP/1.1" 200 4231 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13(KHTML, like Gecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /latest.php HTTP/1.1" 200 991 "-""Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, like Gecko)Version/3.1 Safari/525.13" "-"proxy-622.dsl.clear.net.nz - - [22/Apr/2009:18:52:51 +1200] "GET /latest-news.rss HTTP/1.0" 200 7199data- Difficult to read- Difficulto to measure- Time consuming
  • 8. proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/photos/455.jpg HTTP/1.1"200 986 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/nav/tab_left_middle.gifHTTP/1.1" 200 1020 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13(KHTML, like Gecko) Version/3.1 Safari/525.13" "-"dyn-530.optus.com.au - - [22/Apr/2009:18:52:51 +1200] "GET /images/photos/5332.jpg HTTP/1.1" 200244 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322)" "-"dhcp-465.telstra.com.au - - [22/Apr/2009:18:52:51 +1200] "GET /gallery.php?section=entertainmentHTTP/1.0" 200 58636 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13(KHTML, like Gecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /adclick.php?c=LME2 HTTP/1.1"302 378 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/ad/RH4Y.jpeg HTTP/1.1"200 1031 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/favicon.ico HTTP/1.1" 2002824 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /adclick.php?c=GD52 HTTP/1.1"302 379 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /stories.php?date=yesterdayHTTP/1.1" 200 4231 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13(KHTML, like Gecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /latest.php HTTP/1.1" 200 991 "-""Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, like Gecko)Version/3.1 Safari/525.13" "-"proxy-622.dsl.clear.net.nz - - [22/Apr/2009:18:52:51 +1200] "GET /latest-news.rss HTTP/1.0" 200 7199dataDifference audences and destination-business                                                      -applications  -external services-error logs ( js error, and application error)- . . .sucks
  • 9. proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/photos/455.jpg HTTP/1.1"200 986 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/nav/tab_left_middle.gifHTTP/1.1" 200 1020 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13(KHTML, like Gecko) Version/3.1 Safari/525.13" "-"dyn-530.optus.com.au - - [22/Apr/2009:18:52:51 +1200] "GET /images/photos/5332.jpg HTTP/1.1" 200244 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322)" "-"dhcp-465.telstra.com.au - - [22/Apr/2009:18:52:51 +1200] "GET /gallery.php?section=entertainmentHTTP/1.0" 200 58636 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13(KHTML, like Gecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /adclick.php?c=LME2 HTTP/1.1"302 378 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/ad/RH4Y.jpeg HTTP/1.1"200 1031 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/favicon.ico HTTP/1.1" 2002824 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /adclick.php?c=GD52 HTTP/1.1"302 379 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, likeGecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /stories.php?date=yesterdayHTTP/1.1" 200 4231 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13(KHTML, like Gecko) Version/3.1 Safari/525.13" "-"proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /latest.php HTTP/1.1" 200 991 "-""Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, like Gecko)Version/3.1 Safari/525.13" "-"proxy-622.dsl.clear.net.nz - - [22/Apr/2009:18:52:51 +1200] "GET /latest-news.rss HTTP/1.0" 200 7199rocksdataDifference audences and destination-business                                                      -applications  -external services-error logs ( js error, and application error)- . . .
  • 10. What we dowith data?_ _ _ _ _ !ignoretrack
  • 11. We track data_ _ _ _ _ !ignorethen
  • 12. dataBUSINESSVALUEthen
  • 13. … still 42.andbussinessvalue?andbussinessvalue?4242hitchhikers guide to the galaxy
  • 14. Our code generates business value when it runs,not when we write code.@coda
  • 15. Store log and error dataQuery on past eventsLocate the bugKnow Servers/applicationstatus: healthy/resources/Responding as it should?Face the reality of myecosystemUnderstand why andwhere the app. is slowAnalyze the behaviourwhen it ‘s runningwatch databe alertedfast websiteNeedsdata…BUSINESSVALUE
  • 16. NeedsBUSINESSVALUEloggingdebuggingmonitoringprofilingmetrickingoptimizingActionsStore log and error dataQuery on past eventsLocate the bugKnow Servers/applicationstatus: healthy/resources/Responding as it should?Face the reality of myecosystemUnderstand why andwhere the app. is slowAnalyze the behaviourwhen it ‘s runningwatch databe alertedfast website…
  • 17. BUSINESSVALUEloggingdebuggingmonitoringprofilingmetrickingoptimizingActionsmonolog, Logstash,GraylogSentry / GraylogNagios, Cacti …external serviceXhProfstatsd-php-client,StatsD, Graphite
  • 18. Its not about the accuracy of estimating itsabout facing the reality of uncertainty.@jimhighsmith
  • 19. metricshttp://samate.nist.gov/index.php/Metrics_and_Measures.htmlhttp://nvlpubs.nist.gov/nistpubs/jres/104/6/j46gra.pdfThe IEEE Standard Glossary of Software Engineering Terminologyalso defines quality metric as :(1) A quantitative measure of the degree to whichan item possesses a given quality attribute.(2) A function whose inputs are software data and whose output is asingle numerical value that can be interpreted as the degree to whichthe software possesses a given quality attribute.* @param array $data* @return floatfunction calculateUglyness(array $data);Ninja territory
  • 20. measuringhttp://samate.nist.gov/index.php/Metrics_and_Measures.htmlHOW TO MEASURE ANYTHING - FINDING THE VALUE OF “INTANGIBLES” IN BUSINESSsecond edition Hubbardexperimentally obtainingone or more quantity valuesthat can reasonably beattributed to a quantity.reasonably => uncertainty reductionNinja territory
  • 21. What metricsmeans?DUNNOMerlin knows Yes Matrix Isaw the film
  • 22. metricsVisual tool tobetter understandreality
  • 23. metricsalign technical andbusiness vision ofIT.
  • 24. producesgraphsDECISIONSmetrics
  • 25. StatsD simple— Node.JS daemon. Listens for message over UDPsimple daemon for easy stats aggregation.Graphite rude <key> <numeric value> <timestamp>Graphite stores numeric time-series data.— Real-time graphing system— Components:— Carbon stores the data in Graphite’s specialized database.— The data can then be visualized through graphiteweb interface developed with django.— Whisper allows carbon to write multiple data points.metrics
  • 26. Symfony2 integration:there’s a bundle for it ‘liuggio/StatsDClientBundle’— Amount of visitor— Logged vs Anonymous user— Doctrine collector— Amount of PHP peak— Execution Time— Monolog— Easily add a new metric:$this->get(‘stats_d_client.factory)->increment(’upload.file);metrics
  • 27. Symfony2 integration PHP Peakmetricsthis is a‘Timer’
  • 28. Concepts— Bucket— Values— Flush Intervalmetricsmetrics— Counter— Gauge— Timer— Set— SampleUnderstand StatsD
  • 29. GAUGES are arbitrary values, which can be recorded.metrics$factory = $this->get(stats_d_client.factory);$data[] = $factory->gauge(cart.item.number, 30);$data[] = $factory->gauge(cart.item.number, 34);$data[] = $factory->gauge(cart.item.number, 38);$this->get(stats_d_client.service’)->send($data);cart.item.number|38Result is the last one (per flush time)
  • 30. COUNTS increment and decrement a value.metrics$factory = $this->get(stats_d_client.factory);$data[] = $factory->increment(user.visitor);$this->get(stats_d_client.service’)->send($data);The value stay in memory until the flush interval.
  • 31. TIMERS collects numbers.metrics$factory = $this->get(stats_d_client.factory);$data[] = $factory->timing(’php-peak, 500);$this->get(stats_d_client.service’)->send($data);Operations [mean, upper, sum, lower, count, ...]are already calculated 
  • 32. Collect from different sources# echo "deploy.web:1|c" | nc -w 1 -u graphite.com 8125
  • 33. trackeverythings thatmovescanmetrics
  • 34. you have moredata than you thinkand you need lessdata than you thinkMORELESS
  • 35. new metrics— Why do you want to know?— How much do you know now?— What is the value to additionalinformation
  • 36. The time you spend for metrics has a cost,and usually is more economicalthan many managers assume.maybe becausethey didn’tmeasure.bazingametrics
  • 37. producesDECISIONSfastmetricsgood
  • 38. good and fastDECISIONSBUSINESShelp the
  • 39. and _ _ _will move yourto the next level.metrics???
  • 40. OptimizingPerformance problems?Enlargeyourprofiler
  • 41. Great caching is like great sex.It hides all the real problems@vivekhaldar
  • 42. Wikipedia: “a Cache ( /’kæ∫/ KASH[1]) is acomponent that transparently stores dataso that future requests for that data can beserved faster.”the cache misshappens once for thesame response.given the same request the Cache miss happens once.given the same request the response is not ‘processed’ twice.A Cache Hit happens if the Cachecontains the answer.A Cache Miss happens if the Cachedoesn’t contain the answer.The perfect cache when
  • 43. CACHEIS A RESPONSEMATTER
  • 44. — Application caching: app/cache with opcode— Response caching and HTTP-CACHE— Doctrine2 Query and Result Caching— CDN and static files— OtherCACHE + SF2
  • 45. Terravison vendor folder is about 340 MBThe app/cache/prod folder is about 3000 files1. Annotation (Entity/Controller/...)2. Templatea. name => fileb. twig in php3. Assetic (definition)4. Url matcher, Url creator5. Translations array key=>value6. Doctrine Entity Proxy./app/cache
  • 46. Add your class to the services with the tag kernel.cache_warmerNote: the CacheWarmerInterface is not properly the same as the snippet.cache warmernamespace SymfonyComponentHttpKernelCacheWarmer;interface CacheWarmerInterface/*** Warms up the cache.* @param string $cacheDir The cache directory*/public function warmUp($cacheDir);public function isOptional()
  • 47. Symfony2 is a web framework built from scratch aroundthe http specification— a resource could be Fresh || Stale— http-caching only on safe method— validation || expirationHTTP
  • 48. HTTP cache
  • 49. HTTP headers$response->setPublic() //default is private$response->setMaxAge(600);// Same as above but only for shared caches$response->setSharedMaxAge(600);$response->setETag(md5($response->getContent()));$response->setLastModified($datetime);if ($response->isNotModified($this->getRequest()))...$response->setVary(‘Accept-Encoding’);symfony.com/doc/master/book/http_cache.html
  • 50. The leastexpensivequery is thequery younever run.
  • 51. DoctrineCommonCacheCache(There’s not tagging)doctrine cache// Fetches an entry from the cache.string function fetch($id);// Tests if an entry exists in the cache.bool function contains($id);// Put data into the cache.bool function save($id, $data, $lifeTime);// Deletes a cache entry.bool function delete($id);
  • 52. MetaData Cache Annotation (OPCODE)Query Cache DQL parsed into SQL(OPCODE/REDIS/MEMCACHED)Result Cache The query result(REDIS/MEMCACHED/OPCODE)doctrine caches
  • 53. doctrine cacheClass ProductRepository...$query = $this->createQueryBuilder(‘p’)->where(‘p.price > :price’)->setParameter(‘price’, ‘19.99’)->getQuery();$products = $query->getResult();$query->useResultCache(true,$lifetime,__METHOD__ . serialize($query->getParameters()));$query->useQueryCache();
  • 54. There are only two hard things in Computer Sciencecache invalidation and naming thingsPhil KarltonThere are only two hard things in Computer Sciencecache invalidation, naming things and off-by-1errors.anonymous
  • 55. InvalidationEspecially for http, but in general, you should not waste time toinvalidate a value, but you should invest resources to find the righttime of validity.Cache miss stormWhen a resource is no longer available (cache miss), and is requiredby many clients simultaneously (storm), a congestion happens.cache miss storm, cache stampede,dog-piling, APC Cache slamcache
  • 56. Pre-Caching/CachebackSometimes you need to have some data already available and youdon’t want that the first request wait.Uniqueness of the cache keyAttention to the conflicts in the key definition, always put a prefix ornamespace such as language or role.Http-Microcaching:Save all the responses for 1||2 secondslot of examples on internet for Nginxcache
  • 57. and _ _ _will move yourto the next level.???
  • 58. Performancetips - rule #1Mantainibilitywin overPerformance
  • 59. — The user should never wait    slow operation data in a queue (Redis/*MQ)    processed by a consumer (cli + supersivord):— Image Manipulation— File Creation (pdf)— Sending email as spool— I/O operation— Use Session with Redis/Memcache— Caution on what do you do in the loop && nested loopeasy …
  • 60. … tips— Must! OPCODE: APC using the apc.stat = 0— $php composer.phar dump-autoload –optimize— Twig.c php extension— Assets— Minify and combine all of your CSS and JS files— Run all (or just some) of your CSS or JS filesthrough some sort of compiler,such as LESS, SASS or CoffeeScript— Run image optimizations on your imageshttp://symfony.com/doc/current/book/performance.htmlhttps://speakerdeck.com/alexandresalome/symfony2-and-performances
  • 61. the DIC of sf2.3 is lazyA lazy loading value holder proxy is a virtual proxy that wraps andlazily initializes a "real" instance of the proxied class.<service id="foo" class="AcmeFoo" lazy="true" />When do I use a lazy value holder?-Your object takes a lot of time and memory to be initialized (with alldependencies)-Your object is not always used, and the instantiation overhead can beavoidedgithub.com/Ocramius/ProxyManager/blob/master/docs/lazy-loading-value-holder.mdgithub.com/Ocramius/symfony-docs/blob/master/components/dependency_injection/lazy_services.rst
  • 62. Cache Driver# config_prod.ymlservices:cache:class: DoctrineCommonCacheApcCache #MemcacheCache# config_dev.ymlservices:cache:class: DoctrineCommonCacheArrayCacheif ($fooString = $this->get(‘cache’)->fetch(‘foo’)) {$foo = unserialize($fooString);} else {// do something with $foo$this->get(‘cache’)->save(‘foo’, serialize($foo),$lifetime);}http://docs.doctrine-project.org/en/latest/reference/caching.html
  • 63. DOCTRINEObject relational mapper(ORM) for PHP that sits ontop of a powerful databaseabstraction layer (DBAL)
  • 64. doctrine >=2.4New features:https://speakerdeck.com/asm89/what-is-new-in-doctrineit ill be faster ?NO
  • 65. doctrine >= 2.3—2.4 flush multiples entities$em->flush(array($entityA, $entityB);—2.4 paginator—2.4 master slave—2.4 join to the interface—2.3 You can override metadata—2.3 Event listener in a separated class/*** @ORMManyToOne(targetEntity="Tvision...EntityStatusInterface")**/https://speakerdeck.com/asm89/what-is-new-in-doctrine
  • 66. Optimize Doctrinehttp://xkcd.com/1195/
  • 67. Optimize Doctrinehttps://doctrine-orm.readthedocs.org/en/latest/reference/unitofwork.htmlstep1:deeply understand UOWstep2:goto step1
  • 68. $article = new Article();$article->setHeadline(I love PHPday);$em->persist($article); // the entity is managed$em->flush(); // the entity is persisted...$article->setHeadline(I love JSday and PHPday);$em->flush(); // the Headline is updatedon the persistence layer$em->persist($article);
  • 69. $article = $em->find(Article, 1234);$article->setHeadline(’I love GRUSP);$article2 = $em->find(Article, 1234);echo $article2->getHeadline(); // I love GRUSPif ($article === $article2) ---> true!!
  • 70. Unit of workhttps://doctrine-orm.readthedocs.org/en/latest/reference/unitofwork.htmlMaintains a list of objects affected by a business transaction andcoordinates the writing out of changes and the resolution of concurrencyproblems.P of EAA page 184 Patterns of Enterprise Application Architectureby Martin FowlerUOW = keeps trackchanges of ObjectsTL;DR
  • 71. Identity MapThe flush has a cost— Number of managed Entities— Tracking policy— Type of entitieshttps://doctrine-orm.readthedocs.org/en/latest/reference/unitofwork.htmlcontains all the managed entitiesOn flush makes a property by property comparison for all theentities managed.
  • 72. tracking policyDoctrine definition:“is the process of determining what has changed in managed entitiessince the last time they were synchronized with the database.”— implicit tracking (default and simple)— explicit tracking (suggested for lot of entities managed)— notify tracking/*** @Entity* @ChangeTrackingPolicy("DEFERRED_EXPLICIT”)http://docs.doctrine-project.org/en/latest/reference/change-tracking-policies.htmlFlush optimization #1
  • 73. tracking policyDeferred ImplicitUOW = DBhttp://docs.doctrine-project.org/en/latest/reference/change-tracking-policies.htmlDeferred ExplicitUOW != DB— UOW = ’I love PUG Roma’— DB is not updated.Need the persist!Flush optimization #1// $article is already managed or loaded// we are changing the Headline$article->setHeadline(I love PUG Roma);$em->flush();echo $article->getHeadline();// is ‘I love PUG Roma’
  • 74. Read only entity/*** @Entity* @READONLY*/Class Status {$em->getUnitOfWork->markReadOnly($entity);Temporarily mark entities as read onlyFlush optimization #2
  • 75. Flush only on fewentities$em->flush($entity)Flush optimization #3
  • 76. Reducing theflush costThe flush has a cost— Number of managed Entities (flushing only some)— Tracking policy (explicit || implicit)— Type of entities (read-only || temp. read-only)Recap.
  • 77. reference$post = $postRepository->find($postId);$comment->setPost($post);...// $postId comes from somewhere$comment->setPost($em->getReference(‘AcmeBundleEntityPost’, $postId));.
  • 78. lazy associationsAssociation are LAZY by default// Order/*** @ManyToOne(targetEntity=”Cart”, cascade={“all”},fetch=”EAGER”)*/private $cart<?php$query = $em->createQuery(“SELECT o FROM Order o”);$query->setFetchMode(“Order”, “Cart”, “LAZY”);$query->execute();Change the fetch mode
  • 79. associationsWhat’s better than LAZY ?/** @ORMManyToMany(targetEntity=”Trips”,mappedBy=”rides”,fetch=”EXTRA_LAZY”)*/private $trips;$turin->getTrips()->contains($turinCaselle);$turin->getTrips()->count();$turin->getTrips()->slice($offset, $length);
  • 80. and _ _ _will move yourto the next level.Symfony2applicationOptimization
  • 81. Done everything?
  • 82. Did you reallyneed it?
  • 83. Rationality
  • 84. Knowing when optimization is prematuredefines the differences the master engineerand the apprenticeTheo SchlossnagleOptimizationWe’ve heardthis a milliontimes
  • 85. When you discover the real problem, you done the half.Problem are not all equals, and most are illusions.Pieter Hintjens
  • 86. Optimizationyou are doing it wrong:— Cache everything— Guessing / action without measurement
  • 87. and _ _ _will move yourto the next level.Symfony2applicationOptimization
  • 88. andwill move yourto the next level.Symfony2applicationOptimization
  • 89. andwill move yourto the next level.BUSINESS
  • 90. andwill move yourto the next level.metrics
  • 91. andwill move yourto the next level.BUSINESS
  • 92. andwill move yourto the next level.BUSINESS
  • 93. andwill move yourto the next level.metrics
  • 94. andwill move yourto the next level.metricsOptimization
  • 95. andwill move yourto the next level.metricsSymfony2application
  • 96. datathenBUSINESS VALUE
  • 97. Performance optimization couldMetrics shouldRationality must
  • 98. @liuggioplease read the references
  • 99. https://speakerdeck.com/asm89/what-is-new-in-doctrinehttp://docs.doctrine-project.org/http://nvlpubs.nist.gov/nistpubs/jres/104/6/j46gra.pdfhttp://ilia.ws/files/phpuk_bottlenecks.pdfhttp://www.slideshare.net/anton_chuvakin/log-mining-beyond-log-analysishttp://pivotallabs.com/139-metrics-metrics-everywhere/http://www.hubbardresearch.com (HOW TO MEASURE ANYTHING - FINDING THE VALUE OF “INTANGIBLES” )https://people.mozilla.com/~rmiller/heka-intro-2013-03/#/2http://tools.ietf.org/html/draft-ietf-httpbis-p6-cache-18https://developers.google.com/speed/docs/best-practices/cachinghttp://notmysock.org/blog/php/user-cache-timebomb.htmlhttp://sonata-project.org/blog/2012/5/15/assetic-package-configurationhttp://www.slideshare.net/jonkruger/advanced-objectorientedsolid-principleshttp://slides.seld.be/?file=2011-10-20+High+Performance+Websites+with+Symfony2.htmlhttp://stackoverflow.com/questions/8893081/how-to-cache-in-symfony-2http://www.slideshare.net/postwait/scalable-internet-architecturehttp://techportal.inviqa.com/2009/12/01/profiling-with-xhprof/http://www.mysqlperformanceblog.com/2010/09/10/cache-miss-stormhttp://www.slideshare.net/iamcal/scalable-web-architectures-common-patterns-ap...http://blog.servergrove.com/2012/04/18/how-to-create-a-cache-warmer-in-symfony2/http://www.slideshare.net/jwage/doctrine-intherealworldsf-live2011sanfranhttp://www.slideshare.net/quipo/scalable-architectures-taming-the-twitter-firehosehttps://github.com/liuggio/StatsDClientBundlehttp://www.slideshare.net/quipo/monitoring-at-scale-intuitive-dashboard-designhttp://www.slideshare.net/ihower/rails-performancehttp://www.infoq.com/presentations/Architecture-Scale-ZeroMQhttps://speakerdeck.com/keithpitt/keith-and-marios-guide-to-fast-websiteshttp://www.slideshare.net/anton_chuvakin/log-mining-beyond-log-analysishttp://marco-pivetta.com/proxy-pattern-in-php/#/9http://www.slideshare.net/guilhermeblanco/object-calisthenics-applied-to-phphttp://talks.php.net/show/dojo/https://github.com/symfony/symfony/pull/7818https://speakerdeck.com/roidrage/metrics-monitoring-logginghttp://codahale.com/codeconf-2011-04-09-metrics-metrics-everywhere.pdfhttp://pivotallabs.com/139-metrics-metrics-everywhere/http://blog.pkhamre.com/2012/07/24/understanding-statsd-and-graphite/References