• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
p2p-proxy.ppt
 

p2p-proxy.ppt

on

  • 5,213 views

 

Statistics

Views

Total Views
5,213
Views on SlideShare
5,213
Embed Views
0

Actions

Likes
0
Downloads
23
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    p2p-proxy.ppt p2p-proxy.ppt Presentation Transcript

    • Searching very large bodies of data using a transparent peer-to-peer proxy Mike Taylor and Marc Cromme, Index Data [email_address] [email_address] Albertosaurus sacophagus skull modified from Carr 1999. (Not relevant to the talk, but pretty.)
    • Overview
      • Where we're headed in the next half-hour:
      • The problem
      • Standardised semantically rich search-and-retrieve protocols
          • ANSI/NISO Z39.50
          • SRU
      • Transparent protocol proxies
      • Fan-out proxies, singly and in combination
      • Peer-to-peer proxies
      • Operation of the peer-to-peer proxy network
      • Using the peer-to-peer proxy in Alvis
      • Conclusions
      Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • The Problem
      • The key advantage of the Internet is distribution
      • – That's why there is so much information out there.
      • The key problem of the Internet is aggregation
      • – That's why it's so darned hard to find anything!
      • How can we get at all that tasty data?
      • Monolithic systems can only get us so far.
      • Even Google – with its huge index – is limited by its inability to
      • probe into the “deep web”. It is limited to dumb screen-scraping.
      • We propose a solution made up of many autonomous nodes.
      • We will approach this in several steps.
      Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • Step 1: standardised search-and-retrieve protocols Z39.50 client Z39.50 Library of Congress Z39.50 server Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • British Library Z39.50 server Library of Congress Z39.50 server Step 1: standardised search-and-retrieve protocols Z39.50 client Z39.50 Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • Step 1: standardised search-and-retrieve protocols Z39.50 client Z39.50 Library of Congress Z39.50 server British Library Z39.50 server Local catalogue Z39.50 server Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • Step 1: standardised search-and-retrieve protocols Library of Congress Z39.50 server Metasearching Z39.50 client Z39.50 British Library Z39.50 server Local catalogue Z39.50 server Z39.50 Z39.50 This is possible because of the semantic alignment of the servers. Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • So can Z39.50 save the world? Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • No. Then the serpent saith unto Adam, “Lo, why doth thine information service not use XML?” And Adam saith, “Verily, Z39.50 worketh just fine.” But the serpent, who was subtle of tongue, saith unto him, “But XML is more fashionable .” And, behold, Adam was deceived, and did fall. – The Book of Standards, ch. 3, v. 4-6. Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • Welcome to the 21 st Century Library of Congress Z39.50 server Metasearching Z39.50 client Z39.50 British Library Z39.50 server Local catalogue Z39.50 server Z39.50 Z39.50 Everything must be XML Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • Welcome to the 21 st Century Library of Congress Z39.50 server Metasearching Z39.50 client Z39.50 British Library Z39.50 server Local catalogue Z39.50 server Z39.50 Z39.50 Resistance is useless! Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • XML-based search-and-retrieve protocols The binary Z39.50 protocol is superseded by SRU. (Search/Retrieve by Url). This is a NISO-registered standard for expressing queries using rich URLs, to obtain XML responses that contain records matching the query. h ttp://sru.miketaylor.org.uk/sru.pl? version=1.1& operation=searchRetrieve& query=dinosaur& startRecord=1& maximumRecords=1& recordSchema=dc Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • An SRU response (single DC record) <?xml version=&quot;1.0&quot;?> <zs:searchRetrieveResponse xmlns:zs='http://www.loc.gov/zing/srw/'> <zs:version>1.1</zs:version> <zs:numberOfRecords>29</zs:numberOfRecords> <zs:records> <zs:record> <zs:recordSchema>info:srw/schema/1/dc-v1.1</zs:recordSchema> <zs:recordPacking>xml</zs:recordPacking> <zs:recordPosition>1</zs:recordPosition> <zs:recordData> <srw_dc:dc xmlns:srw_dc=&quot;info:srw/schema/1/dc-schema&quot; xmlns=&quot;http://purl.org/dc/elements/1.1/&quot;> <title>Fossils</title> <creator>Lappi, Megan.</creator> <type>text</type> <publisher>New York, NY: Weigl Publishers</publisher> <date>2005</date> <language>en</language> <description>Studying fossils -- Fossil facts -- Gone forever -- A fossil is born -- From bone to stone -- Insects in amber -- Dinosaur footprints</description> <identifier>http://www.loc.gov/catdir/toc/ecip0415/2004004136.html</identifier> <identifier>URN:ISBN:1590362136</identifier> </srw_dc:dc> </zs:recordData> </zs:record> </zs:records> </zs:searchRetrieveResponse> Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • An SRU response (single DC record) <?xml version=&quot;1.0&quot;?> <zs:searchRetrieveResponse xmlns:zs='http://www.loc.gov/zing/srw/'> <zs:version>1.1</zs:version> <zs:numberOfRecords>29</zs:numberOfRecords> <zs:records> <zs:record> <zs:recordSchema>info:srw/schema/1/dc-v1.1</zs:recordSchema> <zs:recordPacking>xml</zs:recordPacking> <zs:recordPosition>1</zs:recordPosition> <zs:recordData> <srw_dc:dc xmlns:srw_dc=&quot;info:srw/schema/1/dc-schema&quot; xmlns=&quot;http://purl.org/dc/elements/1.1/&quot;> <title>Fossils</title> <creator>Lappi, Megan.</creator> <type>text</type> <publisher>New York, NY: Weigl Publishers</publisher> <date>2005</date> <language>en</language> <description>Studying fossils -- Fossil facts -- Gone forever -- A fossil is born -- From bone to stone -- Insects in amber -- Dinosaur footprints</description> <identifier>http://www.loc.gov/catdir/toc/ecip0415/2004004136.html</identifier> <identifier>URN:ISBN:1590362136</identifier> </srw_dc:dc> </zs:recordData> </zs:record> </zs:records> </zs:searchRetrieveResponse> Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • An SRU response (single DC record) <?xml version=&quot;1.0&quot;?> <zs:searchRetrieveResponse xmlns:zs='http://www.loc.gov/zing/srw/'> <zs:version>1.1</zs:version> <zs:numberOfRecords>29</zs:numberOfRecords> <zs:records> <zs:record> <zs:recordSchema>info:srw/schema/1/dc-v1.1</zs:recordSchema> <zs:recordPacking>xml</zs:recordPacking> <zs:recordPosition>1</zs:recordPosition> <zs:recordData> <srw_dc:dc xmlns:srw_dc=&quot;info:srw/schema/1/dc-schema&quot; xmlns=&quot;http://purl.org/dc/elements/1.1/&quot;> <title>Fossils</title> <creator>Lappi, Megan.</creator> <type>text</type> <publisher>New York, NY: Weigl Publishers</publisher> <date>2005</date> <language>en</language> <description>Studying fossils -- Fossil facts -- Gone forever -- A fossil is born -- From bone to stone -- Insects in amber -- Dinosaur footprints</description> <identifier>http://www.loc.gov/catdir/toc/ecip0415/2004004136.html</identifier> <identifier>URN:ISBN:1590362136</identifier> </srw_dc:dc> </zs:recordData> </zs:record> </zs:records> </zs:searchRetrieveResponse> Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • So we can go back to doing what we did before Library of Congress Z39.50 server Metasearching Z39.50 client Z39.50 British Library Z39.50 server Local catalogue Z39.50 server Z39.50 Z39.50 Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • So we can go back to doing what we did before Library of Congress Z39.50 server Metasearching Z39.50 client Z39.50 British Library Z39.50 server Local catalogue Z39.50 server Z39.50 Z39.50 Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • So we can go back to doing what we did before Library of Congress SRU server Metasearching SRU client SRU British Library SRU server Local catalogue SRU server SRU SRU SRU gives us the same semantic alignment as Z39.50. Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • SRU's query language: CQL CQL (Common Query Language) is used by SRU. It may also be used in other contexts (including Z39.50). Its syntax is easy to learn, but very expressive. dinosaur title=dinosaur title=(dinosaur or pterosaur) and author=martill dc.title=*saur and dc.author=martill title exact &quot;the complete dinosaur&quot; and date < 2000 name=/phonetic &quot;smith&quot; fish prox/distance<3/unit=sentence frog Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • 2. Transparent protocol proxies
      • Just a Squid acts as a proxy for the dumb HTTP protocol, so we can
      • have proxies for semantically rich search-and-retrieve protocols.
      • YAZ Proxy is one such – http://indexdata.com/yazproxy
      • Because the protocol is rich, the proxy can do more than Squid:
        • Performance improvements:
          • Cache and re-use intialised sessions
          • Cache and re-use search results
          • Cache and re-use fetched records
        • Server protection:
          • Query sanitisation (for broken servers ... you know who you are)
          • Client throttling, based on request frequency or bandwidth
        • Protocol-level and application-level logging.
      Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • 2. Transparent protocol proxies Library of Congress SRU server SRU proxy SRU client SRU SRU Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • 3. Fan-out proxies – free metasearching for simple clients Library of Congress SRU server Metasearching SRU proxy SRU British Library SRU server Local catalogue SRU server SRU SRU SRU client SRU The client knows nothing about what is happening to its innocent requests. All the metasearching intelligence goes here. Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • 3. Cascading fan-out proxies Server 1 Proxy 1 Proxy 2 Proxy 3 Client Server 5 Server 6 Server 3 Server 2 Server 4 Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • Hey, go nuts Server Proxy Proxy Proxy Client Server Proxy Server Proxy Server Server Server Proxy Server Proxy Proxy Proxy Server Server Server Server Server Server Server Server Server etc., etc., etc. ... Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • Why this doesn't actually work
      • Scaling problems!
      • Every proxy must be administrated:
          • Information about searched resources kept up to date
          • Proxies must be kept running
      • – a single failure knocks out a whole subtree
      • Load on servers
          • Every server is visited by every query
          • What happens when a proxy calls another proxy higher up the tree?
      • Loop detection is difficult in protocols such as Z39.50 and SRU.
      Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • 4. The peer-to-peer proxy Another appropriate SRU server SRU Some appropriate SRU server Really cool SRU server SRU SRU SRU client SRU Big cloud of peers, acting as a proxy Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • What's going on here?
      • Life is simple at the edges of the cloud:
      • SRU clients connect to peers that act as SRU servers
      • SRU servers respond to requests from peers that act as SRU clients
      • This means that off-the-shelf SRU clients and servers can be used.
      • Web-based SRU clients can be redeployed
      • Servers such as the Library of Congress catalogue are available
      • You can use our free Z39.50/SRU-enabled XML database, Zebra
      • http://indexdata.com/zebra
      • Although the cloud has its own structure, it is opaque to the clients and
      • servers at the edge.
      Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • What's going on here?
      • Life is a little more complex within the cloud:
      • Peers must communicate between themselves using a P2P protocol
      • Peers associated with a server must also make SRU requests
      • Peers associated with a client must also handle SRU requests
      • Each peer may act for a client, a server, both, or neither:
      • “ Client” peers are the entry-points into the P2P cloud
      • “ Server” peers actually get the job of searching done
      • “ Servent” peers behave as both clients and servers
      • Some peers may participate in the network for routing purposes only.
      • Clearly the “servent” peer is the general case that all the other specialise.
      • This only needs to be written once.
      Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • Why this rocks
      • The separation between the edges and the cloud is very important.
      • SRU clients and servers are easy to write
      • There will be lots of them out there:
          • Clients providing many different user interfaces
          • Servers providing access to many different collections
      • “ Servent” peers are difficult to write
      • But that's OK, because only one such peer need ever be built
      • Many instances make up a single peer-to-peer proxy cloud
      • The cloud can be used by many clients, and can use many servers
      • Client and server writers don't have to think about the hard stuff.
      Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • 5. Operation of the peer-to-peer proxy network
      • What goes on inside the mysterious cloud?
      • There is a dedicated peer-to-peer protocol used to:
      • Introduce a new peer to the network
      • Welcome a new peer with an initial list of neighbours
      • Pass queries in a chain between peers
      • Return search results back along the chain
      • We won't cover the protocol in detail here, but:
      • Many P2P protocols have “Ping” and “Pong” messages for introductions
      • In ours, we have different messages:
          • New peers cry “ Cathy! ” when they nuzzle up the network
          • Existing peers respond with a cry of “ Heathcliff! ”
      Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • Everybody needs good neighbours
      • Key principle: the network contains NO global information.
      • (So there is no single point of failure.)
      • Each peer knows only about a few “nearby” peers – its “neighbours”
      • Old neighbours are dropped from the pool if they don't prove useful
      • New neighbours are discovered in search responses:
          • Peer A forwards a query to its neighbour Peer B
          • Peer B can't answer it, so it forwards it to its neighbour, Peer X
          • Peer X responds with useful information
          • Peer A accepts this response, and remembers Peer X for next time.
      • The “usefulness” of peers may be judged relative to specific subject areas
      • rather than with a single absolute score.
      Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • SRU client SRU Everybody needs good neighbours Peer A Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • SRU client SRU Everybody needs good neighbours Peer B Peer A Can you help me with this? Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • SRU client SRU Everybody needs good neighbours Can you help me with this? No, but I'll ask my friend. Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com> Peer B Peer X Peer A
    • SRU Dinosaur Data SRU server SRU client SRU Everybody needs good neighbours Can you help me with this? No, but I'll ask my friend. Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com> Peer B Peer X Peer A
    • SRU Dinosaur Data SRU server SRU client SRU Everybody needs good neighbours Can you help me with this? No, but I'll ask my friend. Here you go, pal. Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com> Peer B Peer X Peer A
    • SRU Dinosaur Data SRU server SRU client SRU Everybody needs good neighbours Peer B Peer X Peer A Can you help me with this? No, but I'll ask my friend. Here you go, pal. (I'll remember that.) Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • Why this rocks, part II
      • The peer-to-peer network has the following desirable properties:
      • joinable – easy for new peers to join the network
      • adaptive – the system evolves to improve through time
      • autonomous – each peer can have its own strategies and policies
      • robust – can cope seamlessly with holes appearing in the network
      • efficient – generates minimal network traffic
      • tunable – has parameters that we can play with
      • “ ecologically diverse” – different peers may be tuned differently
      • Most of these properties are related to the key issue:
      • NO GLOBAL KNOWLEDGE!
      Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • Life history of a query
      • Queries can't be allowed to wander around the network forever.
      • Each query begins with a certain lifespan.
      • Each peer that accepts a query decrements its lifespan by one.
      • The remaining lifespan travels with the query to other peers.
      • If a query is passed to multiple peers, the lifespan is divided between them:
      • It might be divided equally between two relevant neighbours
      • It might be split between many neighbours
      • It might be allocated to a single promising neighbour
      • When a query's lifespan is expired, the peer may not propagate it further.
      Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • Ecological diversity
      • Different peers will have different strategies for propagating queries.
      • Some will tend to fan out in a broad but shallow pattern
      • Some will produce tend to pass almost all lifespan to a single neighbour.
      • Some will behave differently depending on the query
      • We hope that diversity of peer strategies will help make the network robust.
      • A query may carry with it a hint about how it likes its lifetime to be spent.
      • “ Tracer bullet” queries have a long, thin trajectory, then fan out.
      • This is a useful way to periodically probe remote parts of the network,
      • in order to discover new and relevant neighbours.
      Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • 6. Using the peer-to-peer proxy in Alvis Alvis is an ongoing European collaborative project to build what the proposal document calls a “Superpeer semantic search engine”. Named after the dwarf Alvis (“all-wise”) from Norse mythology, who answered Thor's questions all night ... (And then turned to stone when the sun rose.) The Alvis “superpeers” are what we just call “peers” in this presentation. This is because Alvis also has another whole layer of peers. These implement a distributed hash table (DHT) of individual keys to find a suitable entry-point to the superpeer network. Testing will show us how much this optimisation buys us. Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • 7. Conclusions
      • Standardised search-and-retrieve protocols facilitate interoperability.
      • A well-defined protocol can be proxied.
      • Proxies may transparently perform many different services.
      • They may perform metasearching (“fan-out proxy”).
      • Metasearching proxies may be cascaded.
      • In practice, such cascades are hard to maintain, and scale poorly.
      • Instead, an entire cloud of peers may function as a proxy.
      • Queries are routed through the cloud to reach the most appropriate servers.
      • Neither clients nor servers need know anything at all about the proxy.
      • Tunable parameters allow us to tweak performance.
      • The European project Alvis is built on such a peer-to-peer proxy.
      • We want to see this kind of network running “in the wild” with many nodes.
      Transparent peer-to-peer proxy Mike Taylor, Index Data <mike@indexdata.com>
    • Thanks for listening! Mike Taylor and Marc Cromme, Index Data [email_address] [email_address] Albertosaurus sacophagus skull modified from Carr 1999. (We should all take the time to look at more dinosaurs.)