Avoiding the Slog of Real-time
      Data Distribution

                             Eric Lunt
                            Glue 2012
Eric Lunt (@elunt)
CTO, BrightTag (@BrightTag)

Backchannel: #glueslog




                              2
Minimizing
 Avoiding the Slog of Real-time
       Data Distribution or at
                        ,
Least Understanding Why It’s
    a Slog to Begin With



                                  3
The Players
                           Brokered
   Site Owner     Vendor              Publisher
                           Vendors




   Site Visitor


                                                  4
What kind of data?


  Site Owner                          Vendor



                     ?
                     Analytics
                   Ad Networks
                  Social/Sharing
               Retargeting Networks


                                               5
What kind of data?


  Site Owner




                     6
How does the data get shared?

              I would like to use your
                  services please.


                                  Great! All we need to get
                                 started is some data about
                                      your site visitors.


             Okay, how do we do that?


                                  Just add these tags to your site.




                                                                      7
How does data get shared?

                     Tags = HTML Fragments

<script src="http://vendy    <img height="1" width="1"    <iframe width="1"
corp.com/tag.js"></script>   src="http://vendycorp.com/   height="1" frameborder="0"
<script>                     tag?id=1234">                scrolling="no"
  vendy.id=1234;                                          marginheight="0"
  fireTag();                                              marginwidth="0"
</script>                                                 topmargin="0"
                                                          leftmargin="0"
                                                          src="http://vendycorp.com/
                                                          tag?id=1234"></iframe>



        JavaScript                Image “Beacon”                   IFrame




                                                                                       8
Meanwhile, inside Shoemart…
             Oh, benevolent mighty IT
              group, we need to add
              these tags to our site.


                                             No.


                     Why not?


                                   It'll slow the site down.


             But this tag will increase
            our ROI CPM SEO B2B WTF
                       BBQ!


                                   My bonus is based on site
                                        performance.




                                                               9
Meanwhile, inside Shoemart…

             Here's an email from VP
              of marketing saying we
            need to add it. And here's
            the email from the vendor
              with the tag code. And
                 some spring rolls.



                              Okay, our next build is in
                             two weeks. We'll try to get
                                     it in there.




                                                           10
Meanwhile, inside Shoemart…




            Three weeks later…




                                 11
Meanwhile, inside Shoemart…

                          Okay, we pushed your tag.



               Victory!




                                                      12
Meanwhile, inside Shoemart…




             24 hours later…




                               13
Meanwhile, inside Shoemart…
          Wait, none of these
          reports make sense.
            What's wrong?
                                    We need to change the tag.


          You didn't implement
          the tag correctly. The            Okay, our next build is in
        segment id should use a            two weeks. We'll try to get
        pipe delimiter instead of                  it in there.
         colons, you're not URI
          encoding the product
        name, and the revenue
         should be an integer in
                pennies.




                                                                         14
What are the problems with tags?

• Can slow the site down




                                   15
What are the problems with tags?

           We've all heard the stat…


  Every 100ms increase in latency means
   losing a BILLION dollars in revenue!*



                *Not true in all cases


                                           16
What are the problems with tags?

• Can slow the site down
  – <script> tag blocks the rendering thread
  – Most vendor tags use the <script> tag
  – Many vendors have lousy JavaScript

                                  I HATE tags!




                                                 17
What can we do about slow tags?

 Understand client performance at an intimate level




                   READ THESE BOOKS!

                                                      18
What can we do about slow tags?

        Move the tags to the bottom of the page

 <html>                           <html>
 <head>                           <head>
 ...                 Bad          ...
 <script src=                     </head>
 "http://vendycorp.com/tag.js">   <body>
 </script>                        ...
 </head>                          <script src=
 <body>                           "http://vendycorp.com/tag.js">
 ...                              </script>
 </body>                          </body>

                                                   Better




                                                                   19
What can we do about slow tags?

              Make tags asynchronous or post-load
            Evolution of the Google Analytics markup:

<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js'
type='text/javascript'%3E%3C/script%3E"));
</script>
<script type="text/javascript">
try{
var pageTracker = _gat._getTracker("UA-xxxxxx-x");
pageTracker._trackPageview();
} catch(err) {}</script>



                                         Old Blocking Markup




                                                                                              20
What can we do about slow tags?

                Make tags asynchronous or post-load
              Evolution of the Google Analytics markup:
<script type="text/javascript">

 var _gaq = _gaq || [];
 _gaq.push(['_setAccount', 'UA-XXXXX-X']);
 _gaq.push(['_trackPageview']);

  (function() {
    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-
analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
  })();

</script>



                                         New Async Markup



                                                                                                   21
What can we do about slow tags?

          Make tags asynchronous or post-load

 <script src=
 "http://vendycorp.com/tag.js">
 </script>
                                                        Better

                  <script>
                  (function() {
                    var s = document.createElement('script');
                    s.type = 'text/javascript';
                    s.async = true;
                    s.src = 'http://vendycorp.com/tag.js';
                    var me = document.getElementsByTagName('script')[0];
                    me.parentNode.insertBefore(s, me);
                  })();
                  </script>



                                                                           22
What can we do about slow tags?

       Make tags asynchronous or post-load
          Or use a JS loader like LABjs:

    <script>
    $LAB.script('http://vendycorp.com/tag.js');
    </script>




                                                  23
SIDEBAR
What can we do about slow tags?
                 Don't do this:


           Make tags asynchronous or post-load
  <script type="text/javascript">
  var prot = (("https:" == document.location.protocol) ? "https:" : "http:");
  new Image().src = prot+'//vendycorp.com/img.cgi?id=123';
  </script>   Or use a JS loader like LABjs:

       <script>                   Do this:
       $LAB.script('http://vendycorp.com/tag.js');
       </script>
  <script type="text/javascript">
  new Image().src = '//vendycorp.com/img.cgi?id=123';
  </script>




                                                                                24
What can we do about slow tags?

      Make tags asynchronous or post-load
            What are the problems?

            document.write()
           Listening for DOM events




                                            25
What are the problems with tags?

• Can slow the site down
• Incomplete data collection
• Data integrity




                                   26
What are the problems with tags?

• Data integrity
                                                          Report For Shoemark
                                                          SKU          Sales
                                                          123456       $82034.32
                                                          589483        90459.34




      Site Owner                                 Vendor


                   sku=123456
                   price=339.25
                   qty=1


                                      sku=123456
                                      price=339.25
                                      qty=1




                       Site Visitor
                                                                                   27
What are the problems with tags?

• Data integrity
                                                          Report For Shoemark
                                                          SKU          Sales
                                                          123456       $82034.32
                                                          589483        90459.34




      Site Owner                                 Vendor


                   sku=123456
                   price=339.25
                   qty=1


                                      sku=123456
                                      price=3392.50
                                      qty=10




                       Site Visitor
                                                                                   28
What are the problems with tags?

• Can slow the site down
• Incomplete data collection
• Data integrity




                                   29
Can we cut the browser out?

• Asynchronous server-side call, so no slowdown
• 100% data collection
• No chance for data manipulation



                       Site Owner   sku=123456
                                                   Vendor
                                    price=339.25
                                    qty=1




    Site Visitor


                                                            30
Can we cut the browser out?


                  So what's the problem?




                         Site Owner   sku=123456
                                                     Vendor
                                      price=339.25
                                      qty=1




   Site Visitor


                                                              31
Can we cut the browser out?

• Server-side integrations are much more difficult

• COOKIES!

                        Site Owner   sku=123456
                                                    Vendor
                                     price=339.25
                                     qty=1




    Site Visitor


                                                             32
Cookies are the real challenge
Site Owner                                                                     Publisher
                                         Vendor




                                                     Cookie:
         <script                                     name=Pat
                                                                             <script
         src="vendycorp.com">
                                                                src="vendycorp.com">



                           Set-Cookie:            "Hi Pat!"
                           name=Pat




                                         Later…
Site Visitor
                                                                                       33
Cookies are the real challenge

• No way to assert site visitor identity to the vendor
• Actually, there is a way…




                         Site Owner   sku=123456     Vendor
                                      price=339.25
                                      qty=1
                                      name=???




    Site Visitor


                                                              34
Cookie Sync

1. Establish the first-party id




                                  Site Owner


                    Set-Cookie:
                    id=shoe321



     Site Visitor


                                               35
Cookie Sync

1. Establish the first-party id
2. Check to see if we know the vendor id for visitor




                   Cookie:      Site Owner
                   id=shoe321
                                             Vendycorp id for id=shoe321?
                                             Nope




    Site Visitor


                                                                            36
Cookie Sync

1. Establish the first-party id
2. Check to see if we know the vendor id for visitor
3. Initiate a cookie sync to map identities
    302
    Location:
    http://vendycorp.com/sync?redir=http://shoemark.com/sync%3Fvendycorpid%3D%[[id]]



                                          Site Owner




     Site Visitor


                                                                                       37
Cookie Sync

1. Establish the first-party id
2. Check to see if we know the vendor id for visitor
3. Initiate a cookie sync to map identities



                   Cookie:
                                                                  Vendor
                   name=Pat




                       302
                       Location:
                       http://shoemark.com/sync?vendycorpid=Pat
    Site Visitor


                                                                           38
Cookie Sync

1. Establish the first-party id
2. Check to see if we know the vendor id for visitor
3. Initiate a cookie sync to map identities


                     Cookie:          Site Owner
                     id=shoe321
                                                   Vendycorp id for id=shoe321 is
                   http://shoemark.                "Pat"!
                   com/sync?
                   vendycorpid=Pat




    Site Visitor


                                                                                    39
Cookie Sync

1.   Establish the first-party id
2.   Check to see if we know the vendor id for visitor
3.   Initiate a cookie sync to map identities
4.   Now we can pass along the vendor id server-side

                     Cookie:      Site Owner   sku=123456     Vendor
                     id=shoe321                price=339.25
                                               qty=1
                                               name=Pat




      Site Visitor


                                                                       40
Cookie Sync

1.   Establish the first-party id
2.   Check to see if we know the vendor id for visitor
3.   Initiate a cookie sync to map identities
4.   Now we can pass along the vendor id server-side
5.   Repeat for each vendor!
                     Cookie:      Site Owner   sku=123456     Vendor
                     id=shoe321                price=339.25
                                               qty=1
                                               name=Pat




      Site Visitor


                                                                       41
Mo' cookies, mo' problems
Site Owner                                                                                Publisher
                                              Vendor




                                                             Cookie:
                                Cookie:                      name=Pat
         <script                name=Pat                     wants=boots
                                                                                        <script
         src="vendycorp.com">
                                                                           src="vendycorp.com">


                                Set-Cookie:        "Hi Pat! Here's an
                                wants=boots
                                                   ad for some boots!"




                                              Later…
Site Visitor
                                                                                                  42
So what can we do?

• Just accept it
   – Live with the problems
   – Pretend Safari doesn't exist




                                    43
No third-party cookies
Site Owner                                                                                 Publisher
                                              Vendor




                                                              Cookie:
                                Cookie:                       name=Pat
         <script                name=Pat                      wants=boots
                                                                                         <script
         src="vendycorp.com">
                                                                            src="vendycorp.com">


                                Set-Cookie:        "Hi stranger!"
                                wants=boots




Site Visitor
                                                                                                   44
So what can we do?

• Just accept it
   – Live with the problems
   – Pretend Safari doesn't exist
   – Ignore the EU cookie directive




                                      45
EU Cookies




             46
So what can we do?

• Just accept it
   – Live with the problems
   – Pretend Safari doesn't exist
   – Ignore the EU cookie directive
• Forge new paths
   – RTB (real-time bidding) is a fore-runner
   – Update user information outside the context of a HTTP
     transaction
   – Beyond the browser


                                                             47
So what can we do?

I actually don't have the answers, but wanted you to be
aware of all the moving parts, so the next time you see
some JavaScript that looks brain-dead and kicks off a
series of six 302 redirects to different domains that
piggyback in forty new tags, you have some context.

You don't have to like it, though.




                                                          48
Thank You!

Eric Lunt (@elunt)
CTO, BrightTag (@BrightTag)




                              49

Avoiding the Slog of Real-time Data Distribution

  • 1.
    Avoiding the Slogof Real-time Data Distribution Eric Lunt Glue 2012
  • 2.
    Eric Lunt (@elunt) CTO,BrightTag (@BrightTag) Backchannel: #glueslog 2
  • 3.
    Minimizing Avoiding theSlog of Real-time Data Distribution or at , Least Understanding Why It’s a Slog to Begin With 3
  • 4.
    The Players Brokered Site Owner Vendor Publisher Vendors Site Visitor 4
  • 5.
    What kind ofdata? Site Owner Vendor ? Analytics Ad Networks Social/Sharing Retargeting Networks 5
  • 6.
    What kind ofdata? Site Owner 6
  • 7.
    How does thedata get shared? I would like to use your services please. Great! All we need to get started is some data about your site visitors. Okay, how do we do that? Just add these tags to your site. 7
  • 8.
    How does dataget shared? Tags = HTML Fragments <script src="http://vendy <img height="1" width="1" <iframe width="1" corp.com/tag.js"></script> src="http://vendycorp.com/ height="1" frameborder="0" <script> tag?id=1234"> scrolling="no" vendy.id=1234; marginheight="0" fireTag(); marginwidth="0" </script> topmargin="0" leftmargin="0" src="http://vendycorp.com/ tag?id=1234"></iframe> JavaScript Image “Beacon” IFrame 8
  • 9.
    Meanwhile, inside Shoemart… Oh, benevolent mighty IT group, we need to add these tags to our site. No. Why not? It'll slow the site down. But this tag will increase our ROI CPM SEO B2B WTF BBQ! My bonus is based on site performance. 9
  • 10.
    Meanwhile, inside Shoemart… Here's an email from VP of marketing saying we need to add it. And here's the email from the vendor with the tag code. And some spring rolls. Okay, our next build is in two weeks. We'll try to get it in there. 10
  • 11.
    Meanwhile, inside Shoemart… Three weeks later… 11
  • 12.
    Meanwhile, inside Shoemart… Okay, we pushed your tag. Victory! 12
  • 13.
    Meanwhile, inside Shoemart… 24 hours later… 13
  • 14.
    Meanwhile, inside Shoemart… Wait, none of these reports make sense. What's wrong? We need to change the tag. You didn't implement the tag correctly. The Okay, our next build is in segment id should use a two weeks. We'll try to get pipe delimiter instead of it in there. colons, you're not URI encoding the product name, and the revenue should be an integer in pennies. 14
  • 15.
    What are theproblems with tags? • Can slow the site down 15
  • 16.
    What are theproblems with tags? We've all heard the stat… Every 100ms increase in latency means losing a BILLION dollars in revenue!* *Not true in all cases 16
  • 17.
    What are theproblems with tags? • Can slow the site down – <script> tag blocks the rendering thread – Most vendor tags use the <script> tag – Many vendors have lousy JavaScript I HATE tags! 17
  • 18.
    What can wedo about slow tags? Understand client performance at an intimate level READ THESE BOOKS! 18
  • 19.
    What can wedo about slow tags? Move the tags to the bottom of the page <html> <html> <head> <head> ... Bad ... <script src= </head> "http://vendycorp.com/tag.js"> <body> </script> ... </head> <script src= <body> "http://vendycorp.com/tag.js"> ... </script> </body> </body> Better 19
  • 20.
    What can wedo about slow tags? Make tags asynchronous or post-load Evolution of the Google Analytics markup: <script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> try{ var pageTracker = _gat._getTracker("UA-xxxxxx-x"); pageTracker._trackPageview(); } catch(err) {}</script> Old Blocking Markup 20
  • 21.
    What can wedo about slow tags? Make tags asynchronous or post-load Evolution of the Google Analytics markup: <script type="text/javascript"> var _gaq = _gaq || []; _gaq.push(['_setAccount', 'UA-XXXXX-X']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google- analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); </script> New Async Markup 21
  • 22.
    What can wedo about slow tags? Make tags asynchronous or post-load <script src= "http://vendycorp.com/tag.js"> </script> Better <script> (function() { var s = document.createElement('script'); s.type = 'text/javascript'; s.async = true; s.src = 'http://vendycorp.com/tag.js'; var me = document.getElementsByTagName('script')[0]; me.parentNode.insertBefore(s, me); })(); </script> 22
  • 23.
    What can wedo about slow tags? Make tags asynchronous or post-load Or use a JS loader like LABjs: <script> $LAB.script('http://vendycorp.com/tag.js'); </script> 23
  • 24.
    SIDEBAR What can wedo about slow tags? Don't do this: Make tags asynchronous or post-load <script type="text/javascript"> var prot = (("https:" == document.location.protocol) ? "https:" : "http:"); new Image().src = prot+'//vendycorp.com/img.cgi?id=123'; </script> Or use a JS loader like LABjs: <script> Do this: $LAB.script('http://vendycorp.com/tag.js'); </script> <script type="text/javascript"> new Image().src = '//vendycorp.com/img.cgi?id=123'; </script> 24
  • 25.
    What can wedo about slow tags? Make tags asynchronous or post-load What are the problems? document.write() Listening for DOM events 25
  • 26.
    What are theproblems with tags? • Can slow the site down • Incomplete data collection • Data integrity 26
  • 27.
    What are theproblems with tags? • Data integrity Report For Shoemark SKU Sales 123456 $82034.32 589483 90459.34 Site Owner Vendor sku=123456 price=339.25 qty=1 sku=123456 price=339.25 qty=1 Site Visitor 27
  • 28.
    What are theproblems with tags? • Data integrity Report For Shoemark SKU Sales 123456 $82034.32 589483 90459.34 Site Owner Vendor sku=123456 price=339.25 qty=1 sku=123456 price=3392.50 qty=10 Site Visitor 28
  • 29.
    What are theproblems with tags? • Can slow the site down • Incomplete data collection • Data integrity 29
  • 30.
    Can we cutthe browser out? • Asynchronous server-side call, so no slowdown • 100% data collection • No chance for data manipulation Site Owner sku=123456 Vendor price=339.25 qty=1 Site Visitor 30
  • 31.
    Can we cutthe browser out? So what's the problem? Site Owner sku=123456 Vendor price=339.25 qty=1 Site Visitor 31
  • 32.
    Can we cutthe browser out? • Server-side integrations are much more difficult • COOKIES! Site Owner sku=123456 Vendor price=339.25 qty=1 Site Visitor 32
  • 33.
    Cookies are thereal challenge Site Owner Publisher Vendor Cookie: <script name=Pat <script src="vendycorp.com"> src="vendycorp.com"> Set-Cookie: "Hi Pat!" name=Pat Later… Site Visitor 33
  • 34.
    Cookies are thereal challenge • No way to assert site visitor identity to the vendor • Actually, there is a way… Site Owner sku=123456 Vendor price=339.25 qty=1 name=??? Site Visitor 34
  • 35.
    Cookie Sync 1. Establishthe first-party id Site Owner Set-Cookie: id=shoe321 Site Visitor 35
  • 36.
    Cookie Sync 1. Establishthe first-party id 2. Check to see if we know the vendor id for visitor Cookie: Site Owner id=shoe321 Vendycorp id for id=shoe321? Nope Site Visitor 36
  • 37.
    Cookie Sync 1. Establishthe first-party id 2. Check to see if we know the vendor id for visitor 3. Initiate a cookie sync to map identities 302 Location: http://vendycorp.com/sync?redir=http://shoemark.com/sync%3Fvendycorpid%3D%[[id]] Site Owner Site Visitor 37
  • 38.
    Cookie Sync 1. Establishthe first-party id 2. Check to see if we know the vendor id for visitor 3. Initiate a cookie sync to map identities Cookie: Vendor name=Pat 302 Location: http://shoemark.com/sync?vendycorpid=Pat Site Visitor 38
  • 39.
    Cookie Sync 1. Establishthe first-party id 2. Check to see if we know the vendor id for visitor 3. Initiate a cookie sync to map identities Cookie: Site Owner id=shoe321 Vendycorp id for id=shoe321 is http://shoemark. "Pat"! com/sync? vendycorpid=Pat Site Visitor 39
  • 40.
    Cookie Sync 1. Establish the first-party id 2. Check to see if we know the vendor id for visitor 3. Initiate a cookie sync to map identities 4. Now we can pass along the vendor id server-side Cookie: Site Owner sku=123456 Vendor id=shoe321 price=339.25 qty=1 name=Pat Site Visitor 40
  • 41.
    Cookie Sync 1. Establish the first-party id 2. Check to see if we know the vendor id for visitor 3. Initiate a cookie sync to map identities 4. Now we can pass along the vendor id server-side 5. Repeat for each vendor! Cookie: Site Owner sku=123456 Vendor id=shoe321 price=339.25 qty=1 name=Pat Site Visitor 41
  • 42.
    Mo' cookies, mo'problems Site Owner Publisher Vendor Cookie: Cookie: name=Pat <script name=Pat wants=boots <script src="vendycorp.com"> src="vendycorp.com"> Set-Cookie: "Hi Pat! Here's an wants=boots ad for some boots!" Later… Site Visitor 42
  • 43.
    So what canwe do? • Just accept it – Live with the problems – Pretend Safari doesn't exist 43
  • 44.
    No third-party cookies SiteOwner Publisher Vendor Cookie: Cookie: name=Pat <script name=Pat wants=boots <script src="vendycorp.com"> src="vendycorp.com"> Set-Cookie: "Hi stranger!" wants=boots Site Visitor 44
  • 45.
    So what canwe do? • Just accept it – Live with the problems – Pretend Safari doesn't exist – Ignore the EU cookie directive 45
  • 46.
  • 47.
    So what canwe do? • Just accept it – Live with the problems – Pretend Safari doesn't exist – Ignore the EU cookie directive • Forge new paths – RTB (real-time bidding) is a fore-runner – Update user information outside the context of a HTTP transaction – Beyond the browser 47
  • 48.
    So what canwe do? I actually don't have the answers, but wanted you to be aware of all the moving parts, so the next time you see some JavaScript that looks brain-dead and kicks off a series of six 302 redirects to different domains that piggyback in forty new tags, you have some context. You don't have to like it, though. 48
  • 49.
    Thank You! Eric Lunt(@elunt) CTO, BrightTag (@BrightTag) 49

Editor's Notes

  • #3 We are in Chicago, 2 years old, we are essentially middleware for data distribution. My background is in building large distributed scalable low-latency systems, not digital marketing.
  • #6 List the kinds of services, build into the big diagram
  • #7 List the kinds of services, build into the big diagram
  • #8 List the kinds of services, build into the big diagram
  • #16 This is kind of a general statement. What is really important is to understand the relationship between the site markup and the browser rendering thread.
  • #17 Okay, maybe not in all cases, but there are several studies that show abandonment rate is directly related to latency, and that does actually tie to revenue.
  • #18 IT hates this situation because it represents a part of the site that they don&apos;t control.
  • #19 Steve Souders is the man. Talks in detail about what blocks the browser rendering thread and what doesn&apos;t.
  • #20 Go as low as you can so as much of the page will render before you make the third-party calls.Oftentimes, vendors will suggest you go high in the page to minimize data loss, though.
  • #24 There are all sorts of things that we haven&apos;t gotten into here, such as dependencies between scripts. LABjs helps with that, so you can wait for a script to load before executing functions in that script file.
  • #25 Talk about how an http reference on an https page is bad. Use protocol relative URLs.
  • #26 For those of you keeping track, we&apos;re now talking about the problems to the solution to the problem.