neilellis@cazcade.com
THANK YOU FOR BEING     my guinea pigs today
I’D LOVE YOUR FEEDBACK TOO
SERENDIPITYHow do I find what I am not looking for?
Serendipity: the occurrence and development of events by chance in a happy or beneficial way
HOW SERENDIPITY HELPS• Many new inventions occur because related information crosses conventional boundaries, leaving it’s...
Serendipity has lead to an incredible amount of discoveries.
A PRACTICAL EXERCISE  Please fill in the forms I handed out.
WHO NEEDS SERENDIPITY?•   B2B Sites - encourages businesses to find ways of collaborating they may    never have thought of...
THE THREE STEP PLAN
STEP 1: REMOVE ISOLATION
Science Books                 Art Books                   Cookery BooksSEMANTICALLY ISOLATED
User              Blog Posts  DocumentsCONTEXTUALLY ISOLATED
DocumentsCONTENT CONNECTED
UsersDocuments            Blog Posts                Documents                         Blog Posts  SOCIALLY CONNECTED
Documents Users           Blog PostsHIGHLY CONNECTED
GET CONNECTED•   Contextually isolated systems only show us information regarding a closed set of    data and activities.•...
OUR STORAGE SYSTEMSAFFECT HOW CONNECTED  WE MAKE THE WORLD
FILE BASED STORAGE SEES THEWORLD AS A SET OF NESTED COLLECTIONS OF ISOLATED        INFORMATION
LIKE FILING CABINETS
OR A WAREHOUSE
RELATIONAL DATABASES AS HIGHLY  ORGANISED COLLECTIONS OF        INFORMATION       WHICH INTERSECT
LIKE ENROLMENT
LIKE BANKING
OR AN OCD LARDER
AND GRAPH DATABASES ASDISORGANISED BUT HIGHLY INTERCONNECTED DATA
LIKE .....
The Human Brain
Ideas
People
The Internet
Data
AND POSSIBLY EVERYTHING!
RDMS VS GRAPH• Highly      connected systems can be modelled relatively easily on an RDMS, but adding new relationships cr...
STEP 2: USE MULTIPLE HOPS
User  User          DocumentsRECOMMEND A FRIEND
User             Documents YOU MIGHT ALSO LIKE
RDMS VS GRAPH• Multiple        hop queries are horrific under an RDBMS in both performance pitfalls and legibility of queri...
STEP 3: WEIGHT AND FILTER
WEIGHT & FILTER• Proximitystill matters, information should be closely  connected if not semantically or contextually rela...
RDMS VS GRAPH• RDMS cannot categorise relationships independently of the content for example ‘like’, ‘owns’, ‘has viewed’....
EXAMPLES
TEFLON FRYING PANS:SERENDIPITY IN ACTION
Marc Grégoire   Mme. GrégoireINVENTED BY MARC GREGOIRE  AT THE BEHEST OF HIS WIFE
Marc Grégoire    PTFEMARC USED PTFE ON HIS TACKLE
Mme. GrégoireHIS WIFE WANTED PANS THAT DIDN’T STICK
PTFESEMANTICALLY ISOLATED
Marc Grégoire   Mme. Grégoire    PTFECONTEXTUALLY ISOLATED
Marc Grégoire   Mme. Grégoire  PTFESOCIALLY CONNECTED
Marc Grégoire   Mme. GrégoirePTFEMULTIPLE HOPS
Mme. GrégoirePTFE       SERENDIPITY
Marc Grégoire   Mme. Grégoire     PTFEHIGHLY CONNECTED SYSTEM
RE-TWEET
RE-TWEETS• Re-tweets allow rapid dissemination of information beyond a limited social group, they cross semantic and conte...
HAVE YOU FILLED IN YOUR       FORMS?
WHAT SERENDIPITY ISN’T!• Random; random   combinations of information are just noise. putting teflon on a dolphin’s nose wo...
THREE STEPS TO SERENDIPITY• Remove     Isolation. Relationships are low cost and can be added to data at any point, so cre...
CODING SERENDIPITY       How can we add serendipity into our systems?• Information   must be able to travel freely between...
HOW NEO4J HELPS• Relationships              are created trivially at low cost at any time with no regards to semantic boun...
TAKE AWAY• Create   more relationships.• Let   information cross contextual and semantic boundaries.• Make    sure relevan...
@neilellisneilellis@cazcade.com
AUTOMATIC WEIGHT&FILTER• Sum    the ‘weight’ of each relationship traversed to the node.• Find   a random number between 0...
MANUAL WEIGHT&FILTER• Re-Tweeting        or forwarding.• Tell   a friend.• Like.• etc.
OTHER EXAMPLES•   Research papers are a semantically arranged collection of information and    therefore create semantic i...
Serendipity-neo4j
Serendipity-neo4j
Serendipity-neo4j
Upcoming SlideShare
Loading in...5
×

Serendipity-neo4j

1,614

Published on

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,614
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • We’ll come back to these forms a little later.\n
  • \n
  • So how can we encourage serendipity?\n
  • \n
  • Semantically related information such as science books, art books and cookery books are unlikely to refer to each other, keeping the information isolated by it’s semantics. When these boundaries are crossed we get some of the inventions we saw earlier.\n
  • Contextually isolated information is separated by the context the information was created in; i.e. it belongs to a single user, a single team, company, project. Anything that links information together into a closed network. When scientists, companies, teams and people communicate their work or interests great things also happen.\n
  • The internet broke away from these two information ghettos by joining documents together on the internet, so our information could be connected.\n
  • We’ve now moved forward into the socially connected era where our systems now encourage the spread of information by users, we share, recommend and forward.\n
  • \n
  • But we can go a stage further, highly connected systems need to not just connect information but people and information in arbitrary combinations - further more we need to allow this information to travel in real time across these links. \n
  • History shows that when we allow information to flow fast and freely in society we see revolutions in science and spirituality. As our collective understanding increases so does the welfare of the individual and society. So it is with information systems, by increasing the flow of information we increase the value to all those using it.\n
  • Whenever information doesn’t flow, ignorance takes over and clearly we all suffer for that.\n
  • So recommendation number one, increase connectivity.\n
  • But our storage systems affect how connected we make the world\n
  • File based systems basically encourages us to dump stuff together, but don’t encourages us to think how it interconnects. So we end up seeing the world as ....\n
  • \n
  • \n
  • Relational databases help us to organise and connect related information in a highly organised formal manner, like ....\n
  • \n
  • \n
  • \n
  • Whereas graph databases or more like ....\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • We also need data to escape it’s ghettos and a way we can do this is to potential allow information to travel arbitrary degrees of separation, for example like emails or tweets. Not just manually like viral marketing, but also automatically - in status updates, suggested content etc.\n
  • We see this already in recommend a friend....\n
  • Or related documents, but the key here is to allow multiple hops across all boundaries, semantic and contextual.\n
  • Multiple hop queries are horrific under an RDBMS in both performance pitfalls and legibility of queries. This is the main reason RDMS systems rarely help the spread of information by automatic means and rely on users passing on information instead.\n\n
  • But we don’t want just any old information, we still need to filter according to relevancy.\n
  • But the key I believe when automating relevancy is not to use relevancy as a fixed one off judgement on whether something is visible or not, rather to use it as an indicator of the likelihood the information will be visible.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • In a semantically isolated example, books would be written about how teflon helps in fishing. Meanwhile frying pans are only of interest to the catering industry and would not have references to fishing equipment.\n
  • In a contextually isolated system Marc would have been busy using teflon for his fishing equipment and never mentioned it to his wife.\n
  • Luckily they talked to each other.\n
  • Now in this system which is not at this point highly connected information was able to travel multiple hops as Marc discussed his fishing equipment and his wife saw the potential application \n
  • \n
  • Now we have a highly connected system that has crossed social and semantic boundaries, how long did it take before we had teflon baking trays, cake tins etc. Once a semantic boundary has been broken the process accelerates and the speed at which other boundaries are broken increases.\n
  • Re-tweets traverse a graph with ‘n’ degrees of separation I can be looking at how to increase the viral nature of my new startup. When I notice a tweet about the use of landing pages - which leads me to write a viral landing page. Such a collaboration is serendipitous, it is unintentional but beneficial and rewarding.\n\n
  • Re-tweets allow rapid dissemination of information beyond a limited social group. Because of the 5 degrees of separation on Twitter, a single tweet can reach the entire 200 million user base within minutes. As shown by Osama Bin Laden’s death.\n
  • Please can you swap forms with one other person .... now the information on those forms is closely related to you because most of the people in the room have similarity in the backgrounds. However it’s outside of your pre-defined social group and the common semantical links between people here. For your homework I’d like you to watch that movie, listen to that music and take a look at that technology!\n
  • \n
  • -- Weight and Filter -> Whether they recommend, make favourite lists or send as a message. Maintain the source of the information for future automatic recommendations. Keep it connected.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Serendipity-neo4j

    1. 1. neilellis@cazcade.com
    2. 2. THANK YOU FOR BEING my guinea pigs today
    3. 3. I’D LOVE YOUR FEEDBACK TOO
    4. 4. SERENDIPITYHow do I find what I am not looking for?
    5. 5. Serendipity: the occurrence and development of events by chance in a happy or beneficial way
    6. 6. HOW SERENDIPITY HELPS• Many new inventions occur because related information crosses conventional boundaries, leaving it’s ghetto.• Ourlives are made richer by discovering ideas and experiences outside our comfort zones and habitual patterns.• Serendipity accelerates information discovery by making new and unexpected connections.
    7. 7. Serendipity has lead to an incredible amount of discoveries.
    8. 8. A PRACTICAL EXERCISE Please fill in the forms I handed out.
    9. 9. WHO NEEDS SERENDIPITY?• B2B Sites - encourages businesses to find ways of collaborating they may never have thought of.• Social sites - let people discover new friends and new interests.• Collaborative software - find projects that could work together in unexpected ways.• Document management - find documents that help you look at your work in a different way?• Contact management - find new people who you could do business with that might not be in a narrowly defined field.
    10. 10. THE THREE STEP PLAN
    11. 11. STEP 1: REMOVE ISOLATION
    12. 12. Science Books Art Books Cookery BooksSEMANTICALLY ISOLATED
    13. 13. User Blog Posts DocumentsCONTEXTUALLY ISOLATED
    14. 14. DocumentsCONTENT CONNECTED
    15. 15. UsersDocuments Blog Posts Documents Blog Posts SOCIALLY CONNECTED
    16. 16. Documents Users Blog PostsHIGHLY CONNECTED
    17. 17. GET CONNECTED• Contextually isolated systems only show us information regarding a closed set of data and activities.• Semantically isolated systems only show us information which is similar to other information.• Content connected systems show us data that relates to each other which can crosses weakening contextual and semantic boundaries.• Socially connected systems show us information regarding our friends and their activities, weakening contextual and semantic boundaries.• Highly connected systems show us information with n-degrees of separation and multiple paths across contextual and semantic boundaries.
    18. 18. OUR STORAGE SYSTEMSAFFECT HOW CONNECTED WE MAKE THE WORLD
    19. 19. FILE BASED STORAGE SEES THEWORLD AS A SET OF NESTED COLLECTIONS OF ISOLATED INFORMATION
    20. 20. LIKE FILING CABINETS
    21. 21. OR A WAREHOUSE
    22. 22. RELATIONAL DATABASES AS HIGHLY ORGANISED COLLECTIONS OF INFORMATION WHICH INTERSECT
    23. 23. LIKE ENROLMENT
    24. 24. LIKE BANKING
    25. 25. OR AN OCD LARDER
    26. 26. AND GRAPH DATABASES ASDISORGANISED BUT HIGHLY INTERCONNECTED DATA
    27. 27. LIKE .....
    28. 28. The Human Brain
    29. 29. Ideas
    30. 30. People
    31. 31. The Internet
    32. 32. Data
    33. 33. AND POSSIBLY EVERYTHING!
    34. 34. RDMS VS GRAPH• Highly connected systems can be modelled relatively easily on an RDMS, but adding new relationships creates complexity and must be planned in advance.• Queryingis easier for semantically and contextually isolated models on an RDMS.• Querying is extremely messy (indeed!) for highly connected models.
    35. 35. STEP 2: USE MULTIPLE HOPS
    36. 36. User User DocumentsRECOMMEND A FRIEND
    37. 37. User Documents YOU MIGHT ALSO LIKE
    38. 38. RDMS VS GRAPH• Multiple hop queries are horrific under an RDBMS in both performance pitfalls and legibility of queries.• Graph databases love multiple hop logic and one can say thrive upon it. It’s much easier to find out related items through arbitrary degrees of separation and semantic barriers.
    39. 39. STEP 3: WEIGHT AND FILTER
    40. 40. WEIGHT & FILTER• Proximitystill matters, information should be closely connected if not semantically or contextually related.• Relevancy should relate to frequency.• Filtering can be done manually by users choosing what to recommend or pass on.• If possible use customer feedback to adjust weighting.
    41. 41. RDMS VS GRAPH• RDMS cannot categorise relationships independently of the content for example ‘like’, ‘owns’, ‘has viewed’.• RDMS cannot add meta-data to the relationship to help ranking of the relevancy.• Graph databases can do both these and can quickly calculate the cost of traversing to an item of content.
    42. 42. EXAMPLES
    43. 43. TEFLON FRYING PANS:SERENDIPITY IN ACTION
    44. 44. Marc Grégoire Mme. GrégoireINVENTED BY MARC GREGOIRE AT THE BEHEST OF HIS WIFE
    45. 45. Marc Grégoire PTFEMARC USED PTFE ON HIS TACKLE
    46. 46. Mme. GrégoireHIS WIFE WANTED PANS THAT DIDN’T STICK
    47. 47. PTFESEMANTICALLY ISOLATED
    48. 48. Marc Grégoire Mme. Grégoire PTFECONTEXTUALLY ISOLATED
    49. 49. Marc Grégoire Mme. Grégoire PTFESOCIALLY CONNECTED
    50. 50. Marc Grégoire Mme. GrégoirePTFEMULTIPLE HOPS
    51. 51. Mme. GrégoirePTFE SERENDIPITY
    52. 52. Marc Grégoire Mme. Grégoire PTFEHIGHLY CONNECTED SYSTEM
    53. 53. RE-TWEET
    54. 54. RE-TWEETS• Re-tweets allow rapid dissemination of information beyond a limited social group, they cross semantic and contextual boundaries.• Re-tweets can be (and are often) re-tweeted, allowing multiple hops.• Other Twitter users act as the filters, and we further weight by reputation.
    55. 55. HAVE YOU FILLED IN YOUR FORMS?
    56. 56. WHAT SERENDIPITY ISN’T!• Random; random combinations of information are just noise. putting teflon on a dolphin’s nose would not be a useful contribution to society. Don’t confuse unexpected with random!• Accidental; serendipitycomes from an attentive, and often intuitive mind receiving diverse information.• Luck; serendipity is a cognitive process that creates new connections between previously unrelated concepts and realises the value in them.
    57. 57. THREE STEPS TO SERENDIPITY• Remove Isolation. Relationships are low cost and can be added to data at any point, so create them and create as many as possible ignoring contextual or semantic boundaries.• UseMultiple Hops. Cross semantic and contextual boundaries when providing relevancy.• Weight and Filter. The value of the information found should relate to the route traversed. Allow users to manually pass on information to others.
    58. 58. CODING SERENDIPITY How can we add serendipity into our systems?• Information must be able to travel freely between users.• Information should be able to travel multiple levels of indirection with ease.• Information should have the maximum number of inter- connections across semantic boundaries.• Information relationships should be categorised and potentially contain meta-data required for weighting.
    59. 59. HOW NEO4J HELPS• Relationships are created trivially at low cost at any time with no regards to semantic boundaries.• Connected information over many hops can be retrieved quickly using Node#traverse or the Traversal framework.• Relationships can have both types and properties making weight and filter calculations easy.
    60. 60. TAKE AWAY• Create more relationships.• Let information cross contextual and semantic boundaries.• Make sure relevancy is probabilistic, not deterministic.• Serendipity is not accidental, random or lucky!• Themore heterogeneous and connected your data becomes, the more you should consider Neo4j.
    61. 61. @neilellisneilellis@cazcade.com
    62. 62. AUTOMATIC WEIGHT&FILTER• Sum the ‘weight’ of each relationship traversed to the node.• Find a random number between 0 and that weight.• Order the discovered nodes by this random value.• Choose the nodes with the nth lowest values.• Byusing random numbers we increase serendipity without sacrificing relevance.
    63. 63. MANUAL WEIGHT&FILTER• Re-Tweeting or forwarding.• Tell a friend.• Like.• etc.
    64. 64. OTHER EXAMPLES• Research papers are a semantically arranged collection of information and therefore create semantic isolated areas of information.• A lending library is another semantically isolated collection of information.• A project management website creates a contextually isolated set of information.• The internet is a highly connected disorganised information storage system - which leads to a fair amount of serendipity. How many interesting things have you ‘stumbled upon’ on the internet, but it still has a tendency to have semantic or contextual silos. There’s still a lot of room for improvement.

    ×