Case in point:Discovery and recommendation features: when was the last time YOU “might be interested in this product”? How accurate are the typical recommendation systems for you?Just how relevant is the typical ad or marketing campaign? When was the last time you went into Amazon and got a coupon for a truly relevant occasion or product? How tired are you of flashing banners?
Public data source: http://bizinformation.ca/www.sears.com#visitors
Massive amounts of data:* Imagine modeling every person on the planet (say, 6 billion). Now say each person is connected to just 100 others (a conservative estimate)Image source: http://networkweaver.blogspot.com/2010/03/overlapping-boards.html
Time distribution image source: http://blog.nielsen.com/nielsenwire/global/social-norms-twitter-users-follow-the-797-rule-in-the-u-k/Twitter following distribution image source: http://www.personalizemedia.com/twitter-long-tail-broadcastization-pre-twitter-reputation/
Inevitably, someone will ask: what are the problems you encountered?Barrier of entry:Ops: setting up a GigaSpaces cluster is not a hassle-free affair. Lots of work went into a robust, efficient bootstrapping procedure and we had to content with quite a few unexpected snags. I believe things are a lot better with the current version than they were a while ago. Furthermore, the overall cost of setting up and deploying GigaSpaces is significantly less than the total overhead for using specific products to tackle our various needs (compared to a traditional system, the cost of setting up e.g. MySQL+RHCS+client configuration; more likely we’d have had to use some sort of 3rd party graph storage, clustering and persistence solution)Devs: working against GigaSpacesis considerably harder than vanilla, commonplace RDBMS. To counter the barrier of entry we modeled our organization so that a core team of developers handle graph storage and data analysis, with most other teams either integrating with this subsystem or handling their own requirements with regular Hibernate/NHibernate over MySQL.Hard to handle migration paths, zero-time deployment and schema evolution. Features in 8.0 should help remedy the situation (cue Nati Shalom)
Transcript of "Introducing social networking into an e commerce platform - (delver) sears holdings IL"
Introducing Social Networking Into an e-commerce Platform<br />Tomer Gabel |Sears Holdings IL<br />03.02.2011<br />
Social Commerce: An Introduction<br />The last few years have seen tremendous growth in social networks<br />Some estimates place Facebook above Google<br />Even if not, we’re talking millions of daily unique visitors<br />So the obvious question is… where’s the money?<br />2<br />
Social Commerce: Business Case<br />What’s wrong with traditional e-commerce?<br />Discovery/recommendation features are extremely hard to get right<br />Overly broad market targeting means lost sales and disgruntled, ad-weary customers<br />The trust model is inherently broken<br />Impossible to gauge truth and accuracy in customer reviews<br />“Wisdom of the masses” does not always apply<br />Not fun!<br />Shopping is a social experience (going to the mall, holiday shopping sprees)<br />This does not translate to existing e-commerce sites!<br />4<br />
Social Commerce: Business Case<br />“Social commerce” aims to address these deficiencies<br />Correlating interests and products is more accurate and significantly easier when based on social context<br />Social circles are inherently constructed on shared interests and perspectives<br />A customer’s social network is much smaller in scope than generating a global, statistical recommendation model<br />More accurate personalized data exposes new opportunities<br />Personalized discovery allows more opportunity to tap the long tail<br />Social interaction makes it easy to identify domain experts<br />A single opinion provided by a friend, family member or acquaintance is more trustworthy than dozens of unrelated product reviews/ratings<br />5<br />
Social Commerce: Business Case<br />Most crucially, social commerce is all about user engagement and collaboration:<br />Should I buy an iPhone, Blackberry or Android phone?<br />Which wedding dress looks best? <br />Which video games are suitable for a preschooler?<br />6<br />Ask your friends!<br />
Social Commerce: The Axiom<br />Social features increase user engagement<br />Increased conversion<br />Profit!<br />7<br />
Enter: Delver<br />The Delver team has two products on the market<br />Two sides of the same coin, really:<br />sears.com is a traditional e-commerce website with a social twist<br />delver.com is a traditional social website with an e-commerce twist<br />9<br />
The Technical Challenge<br />sears.com is a fully blown commercial retail site<br />Over 1 million page-views daily<br />Over 270,000 visitors daily<br />Traffic can easily spike up to ten times in the holiday season!<br />10<br />
The Technical Challenge<br />Processing social networks is not an easy proposition<br />Massive amounts of branching data<br />No data locality<br />Very few assumptions can be made about the data<br />Let’s address each of these in turn<br />11<br />Source: NetworkWeaver<br />
The Technical Challenge<br />Massive amounts of branching data:<br />Imagine every Facebook user (500 million)<br />Imagine each person is only connected to 100 others (conservative estimate)<br />How is user X connected with Y?<br />X has 100 friends<br />Each of them has 100 friends<br />10,001 nodes visited!<br />101 reads from the underlying storage system!<br />12<br />X<br />Y<br />
The Technical Challenge<br />No data locality:<br />Any object may be connected to any other object in no particular order<br />How to split the data?<br />Some research is being done in the area (SPAR)<br />13<br />
The Technical Challenge<br />No easy assumptions:<br />No “typical user”<br />Not enough data to draw archetypes<br />Significant, unavoidable long tail<br />Difficult to pre-tune data structures<br />14<br />
The Technical Challenge<br />The crux of the problem:<br />High branch factor necessitates many loads to serve even a simple request<br />No data locality + high branch factor means very high random I/O<br />Traditional storage models (RDBMS, flat files etc.) are a poor fit<br />Serious research into graph storage, social network composition etc. only dates back a few years<br />No best practices or “accepted truths” to build on<br />15<br />
Use Case for GigaSpaces<br />To solve the graph storage and traversal problem, we arrived at the following requirements:<br />Completely in-memory storage<br />No data locality means caching is inefficient<br />Massive amounts of random I/O cannot scale vertically, and hardware (basically, spindle count) cost quickly becomes prohibitive<br />If data access is sufficiently fast, data can be randomly partitioned<br />Horizontal scaling with a well-known scale-up strategy<br />Add more memory or more nodes to handle data growth<br />Add more CPUs or additional nodes to handle load growth<br />16<br />
Use Case for GigaSpaces<br />Additional requirements include:<br />Map/Reduce execution framework<br />Graph traversal and data analysis requirements lend well to the map/reduce paradigm<br />Code execution on the data nodes<br />Because of the massive amounts of data involved, the network interface will be quickly saturated by retrievals<br />Memory retrieval is at least two orders of magnitude faster than network throughput (DDR2-800 on a dual channel memory controller has a theoretical throughput maximum of 102.4Gb/s)<br />17<br />
Use Case for GigaSpaces<br />As an operations tech I had a few things to add to the list, namely…<br />Nonfunctional requirements:<br />Built-in fault tolerance and high availability<br />Zero-configuration (or as close to it as it gets) setup; in particular, component discovery and assignment must be automated<br />Well-documented deployment, configuration and tuning process<br />Monitoring API<br />Administrative client for diagnosis, trouble resolution and manual intervention<br />18<br />
Use Case for GigaSpaces<br />GigaSpaces features map well to our requirements<br />Data grid<br />Compute grid<br />High availability<br />Horizontal data and load scaling<br />Management API<br />Very few viable alternatives:<br />Hadoop, neo4j are disk-based<br />Terracotta is overly simplistic and has no execution framework<br />Oracle Coherence is expensive and has a limited feature set<br />19<br />
Delver Architecture<br />We ended up with a hybrid platform:<br />GigaSpaces for graph storage, traversal and analysis<br />MySQL for traditional, “simple” data as well as a backing store for GigaSpaces<br />.NET-based front-end, Java-based back-end<br />We had to factor our organization accordingly<br />Data access team provides abstracted interfaces on top of GigaSpaces and MySQL<br />Back-end “heavy lifting” services (e.g. recommendation engine) work directly against GigaSpaces<br />Most other components either use the abstracted DAL or are simple enough to work directly against MySQL using (N)Hibernate<br />20<br />
Key Benefits<br />Significantly reduced integration costs<br />GigaSpaces does a lot of what we need out of the box<br />An alternative solution would require integrating several products, incurring significant integration and development overhead<br />Broad feature set<br />Social commerce is an emerging, dynamic market requiring rapid experimentation and adaptation<br />The large feature set allows us to introduce new features into the system at a furious pace<br />While primarily intended for graph storage, we also use GigaSpaces as a message queue, distributed lock server and distributed scheduler<br />22<br />
Now is a good time for…<br />Questions?COMMENTS?<br />23<br />
Endgame<br />Experience our work!<br />Visit Delver at http://www.delver.com/in?invite=friends-and-family<br />Visit Sears Social at http://catalog.sears.com<br />Read about our work at http://blog.delver.com<br />Have anything to discuss?<br />Contact me at email@example.com<br />Visit my blog at http://www.tomergabel.com<br />Follow me on Twitter at http://www.twitter.com/tomerg<br />Thank you for your time!<br />24<br />
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.