Future of semantic apps
•
2 likes•221 views
Presentation from SemTech 2009 Conference
![Have Your [Software] Agent Message My
[Software] Agent and We’ll Do [Virtual] Lunch
Tony Sarris
Engineering Director, Unisys Corporation
SemTech 2009
Ideas for Future Semantic Applications Combo Session
June 18, 2009](https://image.slidesharecdn.com/futureofsemanticapps-130415191217-phpapp01/85/Future-of-semantic-apps-1-320.jpg)
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to Future of semantic apps
Similar to Future of semantic apps (12)
Recently uploaded
Recently uploaded (20)
Future of semantic apps
- 1. Have Your [Software] Agent Message My [Software] Agent and We’ll Do [Virtual] Lunch Tony Sarris Engineering Director, Unisys Corporation SemTech 2009 Ideas for Future Semantic Applications Combo Session June 18, 2009
- 2. The 90s Vision of Intelligent Software Agents • Rich sets of context-aware agents with deep domain knowledge available as ‘services’ • Easy and inexpensive access to agents anywhere, anytime, with instantaneous results • Agents handle singular or complex tasks based on user- specified requests with associated criteria and goal(s) • General-purpose agents contract as needed with specific- purpose agents to accomplish end-user goal • Integration among distributed agents is automatic and transparent, preserving and adapting context across domains and tasks • Agents seamlessly woven into human-centric social fabric Source: gimpsavvy.com (public domain) © 2009 Unisys Corporation. All rights reserved. Page 2
- 3. 90s Vision: Missing Some ‘Implementation Details’ • Truly mobile computing devices • Ubiquitous high-speed networks • Large repositories of domain-specific, searchable content • Reference ontologies or domain ontologies • Ability to relate data/events to other data/events, apart from text string matching or other syntactical operations • Ability to contextually situate data/events, and to move from one context to another • Intelligent agents Source: gimpsavvy.com (Public Domain) © 2009 Unisys Corporation. All rights reserved. Page 3
- 4. Are We There Yet?...Well, Not Quite Some Gaps Filled But Gaps Still Remain • Mobile computing devices • Ubiquitous, reasonably-priced (‘smart’ phones, ultralight network access PCs/netbooks) • Reference/domain ontologies • Widely-available high-speed • Semantic metadata tags networks (3G, WiFi, WiMax) • Large repositories of domain- • Context and situational specific, searchable content awareness (the web, as we know it), plus • Collaborative, distributed social networks agents • Metadata tags & conceptual – General-purpose and models specialized agents working together to get the job done • Limited, single-purpose rule- • Understanding of social based or agent-based implications of technology applications © 2009 Unisys Corporation. All rights reserved. Page 4
- 5. Imagine If You Will This Scenario (1) Your lonely hotel room, maybe even more work? Business trip to Dublin Source: pdpphoto.org (Creative Commons) A nearby pub with good local food, beer and conversation? Source: pdpphoto.org (Creative Commons) Source: Trig’s photos on Flickr (Creative Commons) © 2009 Unisys Corporation. All rights reserved. Page 5
- 6. Imagine If You Will This Scenario (2) Traditional Irish Food? Check Really? RUaFoodie.com? Doublecheck AllAboutIrelandsPast.org? Source: Or Hiltch photos on Flickr (Creative Commons) Welcome to Blarney’s Best Local Brew? Check Source: pdpphoto.org (Creative Commons) brewophile.com? Source: pdpphoto.org (Creative Commons) Dublin’s best locally-brewed stout & traditional Irish stew Close to centre-city hotels © 2009 Unisys Corporation. All rights reserved. Page 6
- 7. The Plot Thickens You Are Here! Free for Dinner at 19:30? Source: pdpphoto.org (Creative Commons) Anyone within 1K? Dinner: 19.00-22.00 <Status=Open> Source: lpinseel photos on Flickr (Creative Commons) Source: nikkorsnapper photos on Flickr (Creative Commons) } Check profiles on librarything.com? Interested in Discussing James Joyce’s Ulysses? © 2009 Unisys Corporation. All rights reserved. Page 7
- 8. …And Thickens First Match: Great, Except He’s from Final Match: Irish Woman New York! (Non-Dubliner), Non- Drinker, Keen Interest in Joyce Second Match: Local, But Likes His Drink a Bit Too Much Source: Cobra Libre SA photos on Flickr (Creative Commons) Source: FredArmitage photos on Flickr (Creative Commons) © 2009 Unisys Corporation. All rights reserved. Page 8
- 9. Doing It the Olde Fashioned Way Ask Concierge or ‘Man on the Street’ Hang Out in Hotel and Look for Familiar Face Source: pdpphoto.org (Creative Commons) Hang Out in Hotel and Meet Someone New Source: GEL photos on Flickr (Creative Commons) Source: Ulterior Epicure photos on Flickr (Creative Commons) © 2009 Unisys Corporation. All rights reserved. Page 9
- 10. Help From Better Semantics & Intelligent Agents? “Let’s Get Together Again Stay In Sometime” Go Out (8.7/10.0) So, Trusty Agent, What Are My Options? Drink Eat Talk Friend Local SME © 2009 Unisys Corporation. All rights reserved. Page 10
- 11. General Social Issues and Implications • Limited vs. extensive data sets • Virtual vs. physical interaction • Integrating perceptual/sensory experience with factual knowledge • Expectations based on [constrained] context – Pre-web • Job interview • Blind-date © 2009 Unisys Corporation. All rights reserved. Page 11
- 12. Special Safety-Related Issues & Implications • Would you normally chat up a stranger in- person? • Would you meet up with a ‘stranger’ based on a social network or other web connection? • Would you trust an intelligent agent to assemble a profile sufficient to assess safety risks, or to make risk assessments based on FOAF relationships or other criteria? Source: Phillip Roth photos on Flickr (Creative Commons) – How much agent triage, based on what data, before you’d walk down a dark street at night, in a strange town, with a stranger? © 2009 Unisys Corporation. All rights reserved. Page 12
- 13. Visionaries Needed to Drive Intelligent Agents • Richer, deeper [vertical] domain ontologies, coupled with ‘commonsense’ [horizontal] ontologies • Agents that use ontologies and situational data to establish, integrate and/or navigate contexts • Semantic ‘fusion’, similar to sensor fusion • Research into relationships of semantic, agent-based technologies, social networks and other web technologies, with social sciences • Application of that research to ensure that the semantic web – Web 3.0 – truly enriches our everyday lives © 2009 Unisys Corporation. All rights reserved. Page 13
Editor's Notes
- I’m Tony Sarris and during the first half of this session I’d like to provide three things: a brief retrospective on early work on semantic, agent-based technologies; a look at the current state, and a look ahead at what still needs to be done. I’d like to raise some issues around the impacts – good and bad – that such technologies may have on us from a sociological standpoint. In the second half of this session, my colleague, Peter Sweeney, will discuss and demonstrate a technology that fills some of the current gaps in fusing information together in meaningful ways for use by both humans and intelligent agents.
- Some years ago, back in the 90s, the notion of intelligent distributed software agents coordinating on behalf of their human charges was all the rage. In fact, I had a tag line in my e-mail signature block that read “Have Your [Software] Agent Message My [Software] Agent and We’ll Do [Virtual] Lunch.” As part of the R&D community in the field of artificial intelligence, specifically working on ontology, we envisioned a time in the not-too-distant future when automated agents representing our needs and interests would shop for us, book trips and appointments, do research, manage our finances and help us run that complex, dynamic enterprise known as our lives. We envisioned that there would be a rich set of such agents available as ‘services’. They’d be context aware and have deep knowledge about the domain they were operating in. They’d be easy and inexpensive to access, would be available anytime, anywhere and would provide instantaneous results. There would be different kinds of agents employed -- some handling fairly simple and singular tasks and others handling more complex tasks -- and determining the appropriate agents would be based on the criteria and goals of user requests. More general purpose, ‘horizontal’ type agents would contract with more vertical, specialized agents as needed to accomplish the end user goal. That sort of integration among distributed agents would take place automatically, and hopefully largely be transparent to end users. Context would be preserved and adapted – again automatically – across different domains and sets of associated tasks required to accomplish some goal. And lastly, those agents would be woven seamlessly into our human-centric social world. In other words, the agents would fit into our world rather than us having to fit into theirs.
- That was a pretty good vision, don’t you think? But the problem was: at the time the vision was missing some pretty important implementation details – so it came out looking more like this. That’s because unfortunately : We didn’t have truly mobile computing devices and ubiquitous high-speed networks Cell phones were still clunky and service was expensive, laptops were more like desktops and web access was still anchored to a dial-up phone line. Few large repositories of domain-specific, searchable content Things were being connected into the internet – but many were in legacy formats making them difficult to access. And search was nothing like it is today. There were effectively few or no reference ontologies or domain ontologies with which to make sense of that content There was little or no ability to relate data/events to other data/events, apart from text string matching or other syntactical operations and no ability to contextually situate data/events, and to move from one context to another And because we didn’t have many of the fundamental building blocks, we clearly weren’t in a position to really build intelligent agents. AI research was perhaps at its zenith, but AI technologies, including agent-oriented applications, were almost non-existent.
- Now times have changed quite a lot since then. I’ve been impressed sitting in the sessions this week to see just how far we’ve come in a number of areas. So you can reasonably ask the question: are we there yet? Well, we’re certainly much closer than we were, but we’re still not quite there. While some gaps have been filled, a lot of gaps still remain. We’ve got agent-amenable mobile devices and widely-available high-speed networks. But the networks still aren’t ubiquitous and prices for the level of access required to support these sorts of technologies are still relatively expensive. We do have large bodies of content – some might say the problem right now is too much data, but not enough information – and while search has improved greatly, getting the right information is still difficult. Some people argue it’s actually been getting worse lately. We’ve got decent metadata tagging and some conceptual models, but we’re still in the infancy stage of ontologies. Web applications still have very limited context and situational awareness. There are some pretty good single purpose rule-based or agent-based applications, but we certainly are not at the point where we have collaborative, distributed agents working seamlessly together to accomplish some goal. And for all of the technology we do have, we have very little understanding of the social implications of its use. I guess you can distill all that down to the fact that there has been considerable progress – only cellular service is still expensive and AI still doesn’t exist.
- But at least the fundamental pre-requisites exist to enable the sorts of things we envisioned being performed by intelligent agents, plus a number of other more socially-centered things most of us never considered at the time. So where could we, and where should we, go with agent-based technologies? Let’s start by considering a scenario that I’ve heard in various forms over the past several months. It is centered on the intersection of smart mobile devices and social networks. Let’s say you’re visiting Dublin, Ireland on a business trip. Maybe you had a business dinner arranged for the first night, but the next evening you’re free, you don’t really have the time or energy to go on some grandiose exploratory journey through the city, or maybe you’ve already been there a couple times before and you’ve already seen the highlights. Since you’re kind of tired, you’re tempted to just grab a quick bite, go back to your hotel room, listen to some favorites from your playlist on your phone or some mobile music device you’ve brought along or surf the web from your smart phone or portable PC -- whether to do some more work, or for news or entertainment. As a former road warrior myself I can identify with this scenario, as I’m sure many of you can. But then it occurs to you, “Wait a minute, I’m in Dublin! Isn’t there some place not too far from the hotel here in the city centre where I could get some traditional Irish pub food, a locally-brewed stout, and maybe even be around some other people?” because hey, it can get lonely on the road and you could really use some human companionship. No, not *that* kind of companionship. We don’t need more technology dedicated to that purpose, because there seems to be plenty of that already on the internet.
- Here’s where you can use technology to first find a suitable pub restaurant near your hotel in the city centre. You want one that has traditional Irish food (at least according to RUaFoodie.com’s definition of traditional Irish food and a quick historical check you did on AllAboutIrelandsPast.org). So you discover that Irish stew is one example of a traditional Irish food. But in addition to traditional Irish food you want the place to serve a locally-brewed stout that is well-regarded, so you check your favorite beer-related social networking site (brewophile.com) to find a stout maybe rated at least 4 stars out of 5 by people who have established very good on-line pedigrees for that sort of thing. So you end up with a hit on Blarney’s, serving traditional Irish stew and Dublin’s best locally-brewed stout, and located in the centre-city. That’s all pretty basic technology these days.
- Next comes something that’s really starting to be talked about – whether it is totally realized yet or not. You are here! You want to search for someone who, per their mobile device, has a geospatial position that is within 1 K of your hotel, and who per one or more linked social web site(s) or public calendars indicates they are available to meet for dinner within half an hour of your desired dinner time. Now let’s also add in the attribute of wanting, or at least being willing and able, to discuss James Joyce’s Ulysses (based on their profile on a site such as LibraryThing.com). And you’d prefer it be somebody local who can give you an Irish perspective on Joyce and his work.
- You get a hit. He’s registered on LibraryThing and is noted as a fan of James Joyce, including Ulysses. But wait! From other data it looks like he’s born, raised and currently lives in New York, and apparently just happens to be in Dublin at the same time you are. So much for the local perspective! Alternatively, there is a local guy who meets all the criteria, but he has a reputation for posting tirades in various web forums when drunk – which apparently is almost nightly. You don’t mind a little slurred speech here and there, and a little passion in the discussion is fine, but you’d prefer not to have fisticuffs with an angry drunk. There’s also an Irish woman – not a Dubliner, but Irish nonetheless – someone in the education field who is in Dublin for a conference at Trinity College. From Library Thing, she appears to have a keen interest in James Joyce, including Ulysses – but on another site, a dating match site, she notes that she’s a non-smoking, non-drinking divorcee. Still, maybe she won’t mind if you have a drink, so you decide to send her an invitation to meet for dinner, making clear for both parties that the nature of the dinner is strictly platonic. As it turns out when you meet her, she is a professor of European literature and is somewhat of an expert on Joyce and Ulysses. She expected that you had wanted to discuss Joyce’s work from a more critical, literary standpoint. But you muddle through dinner okay and move on. How much searching would it really take to reveal all this? And wouldn’t it be great to have an intelligent agent that quickly analyses multiple data sources about these people in some meaningful context? Or do you simply chat or text briefly first, asking the sorts of questions that might better determine the extent of overlap of your shared interest in the proposed get-together, as described?
- None of this would even have been an issue a few years ago, because using technology this way and at this level wouldn’t even have been possible. To find your desired restaurant or pub, one that’s likely to be populated with the type of person you might have the desired conversation with, you would probably have asked for a recommendation from the hotel concierge or someone on the street who has a perceptibly Irish accent and seems like a local. They’d perhaps recommend a perfectly nice seafood restaurant just down the block. Or perhaps you would have simply hung out for awhile in the hotel bar. Maybe in the bar you would have seen the woman who sat next to you on the plane, with whom you had great conversation for several hours on the way across the Atlantic. And you would have ended up going to dinner with her at some little Indian place in an off-beat neighborhood that she loves to eat at when he’s in Dublin, and you would have continued the enjoyable conversation from the plane – comparing the trials and tribulations of your teens with hers and discussing some more how growing up in the turbulent 60s and 70s continues to influence your lives. Or maybe you would have sat at the bar next to some guy you normally wouldn’t necessarily strike up a conversation with and yet he starts a conversation with you. It turns out you have a shared love for trout fishing and you spend a wonderful evening exchanging fish stories – including over dinner at a little out of way local place you’d never find on the web, a place with great fresh local ruby red trout. You wouldn’t achieve your goal of discussing Joyce with an Irishman – something technology might help you to achieve – but in each of the scenarios above, you’d still have an enjoyable time, and one you likely couldn’t have foreseen.
- For me, that all begs the question: Will we lose the beauty of randomness when we use technology to overtly – and perhaps overly – target an explicit goal? Or could intelligent agents actually expand opportunities for a broader range of interactions? For example, let’s say you had exchanged contact information with the woman on the plane and then noted in your personal preferences that given an opportunity to talk with her again (say if you were in the same city over the same time period, or booking seats on a plane again sometime), you would enjoy doing so (on a 8.7 out of 10.0 scale), and an intelligent agent made use of that information. Applications like TripIt offer very rudimentary starts down that path. Or what if an intelligent agent presented a varied menu of the evening’s options, rank-ordered based on some set of criteria including a low-key personality, likes beer (but no reputation for drunkenness), is interested in literature (but as a lay person), and maybe even check to see if they have good financial credentials with on-line payment clearinghouses (so you’re less likely to get stuck footing the entire dinner bill), etc.?
- One thing to naturally consider at this point is under what conditions people are willing to strike up a conversation with a stranger, or positively react to a conversation that a stranger attempts to start with them? A lot has to do with basic personality, whether you are an introvert or an extrovert. Some of it has to do with attractiveness, not just physical looks per se, but a whole set of body language-based signals, coupled perhaps with verbal queues, that each person sends and that get interpreted by other people. People quickly assess other people whether they actually meet them or are simply in some shared physical space with them. Are they a threat, friendly, weird, interesting, attractive/sexy, or whatever. Numerous studies indicate that the time it takes a person to form a basic assessment of someone else can be as rapid as a few seconds. Even with pictures or live video feeds on your mobile device of choice, it’ll be awhile before we replicate the experience of really meeting and interacting with someone face-to-face, or even being in the same general physical space with them. Similar sorts of things may take place in a virtual space, but the interaction is much different in its nature and is based on a much more limited set of inputs, perhaps even just words on a tiny screen. From my perspective, virtual worlds like Second Life are still a poor substitute for the real-world. Your mileage, of course, may vary – but there’s a complex equation behind social interactions that virtual agents probably won’t master anytime soon. Let’s go back to our previous scenario. How do you even begin a face-to-face conversation with someone if all you know is that the other person has a desire to discuss James Joyce’s Ulysses? Is it more difficult than having a text-based interaction with that same person via some electronic device? How does it change things when you in fact know a lot about the person (from data collected), but yet you’ve still never actually met them before? It’s that whole virtual versus physical world thing coming into play, as I talked about a minute ago . Even if you know a lot of data about someone, as assembled from the web by you or your intelligent agent, when you first meet them in-person, it is a unique, and perhaps uniquely human, experience to integrate your perceptual or sensory experience of the person face-to-face with your factual knowledge of the person. From the pre-web world, maybe the best correlate to this sort of social experience is having reviewed a prospective employee’s detailed resume, and then meeting them for a face-to-face interview. Only in that context, the data set is typically quite narrowly constrained and the expectation of the nature of the interaction is also fairly well understood from common social conventions and the previous experiences of the individuals. Or perhaps it is more similar to a ‘blind-date’? I’m curious what sorts of social conventions, if any, will arise going forward based on these new forms of technology-aided social interaction.
- What about safety issues? That’s a reasonable concern, particularly if you’re traveling alone at night, or for that matter for anyone who wouldn’t normally meet a ‘stranger’ for a drink or dinner, but who might be willing to now based on a social network type of connection. How and when does safety get factored into this mix, or does it? This is clearly an already widely-discussed issue in the area of virtual, internet-based interaction moving to the realm of real-world, physical interaction. Internet predators are an almost constant media topic. If the number of meet-ups increases exponentially in volume and speed because of new technologies, will safety, fraud and other forms of exploitation become even bigger issues? Can an intelligent agent actually help lower risk through producing and analyzing composite, on-line profiles of potential meet-up partners on the fly, including factoring in weightings based on navigating social networks (e.g., so-called ‘friend of a friend’ (FOAF) relationships)? Scenarios that involve real-time ride-sharing may be good for global warming, but could be hazardous for personal safety if they aren’t done safely. Or what about a localized Twitter-like message sent at 11:00 PM: “Is there anyone near Mission and 5th right now, walking down towards 10th who would like to walk with me?” Is that a boon to safety or a risky proposition? How much triage by you or your intelligent agent, and based on what kind of data, would it take before you’d feel safe being accompanied by a complete stranger for a walk down the street at night? Admittedly, that’s an extreme example, compared to suffering through a mistake in a hotel room booking or bad dinner company, but it’s a good illustration of the opportunities and challenges for intelligent agents and real-world human interaction.
- We’re at the point now where we have the vision – in fact it’s fundamentally the same one from the 90s I think – but what we need now are the visionaries to bring it to realization. There is still work to be done. The deeper the processing and analysis that software agents are expected to perform on data on our behalf, the more the need for richer domain ontologies coupled with horizontal ‘commonsense’ ontologies. Simple conjunctions or intersections of metadata tags aren’t the same thing as meaningful interpretation of data in situational contexts that span multiple domains. Context is critical. What’s needed is semantic fusion, similar to what’s done in the field of sensor fusion. As the linkages between the virtual world and the real-world increase and begin to meld, the relevancy and value of data for the types of activities covered by personal intelligent agents will be established by both the speed and accuracy of that data – just like internet searches today – but with a much more demanding component of situational context and data integration, and perhaps with more profound consequences if we don’t get it right. These new forms of social interaction, as enabled by the intersection of mobile devices, the web, social networks, and the use of intelligent agent technology, could prove very beneficial to us as humans living in a world that is at once more global, but at the same time more fractured. We can perhaps transition technology from a set of tools that today enables a lot of virtual interaction to a set of tools that increases and enhances real-world interaction of a deeper social nature. Technology is making the opportunities for real-world interactions more abundant, but those opportunities are constrained (sometimes reasonably so) by limitations of technology and of our social conventions. For these interactions to be useful, meaningful, convenient and safe, there needs to be advances in the enabling technologies, as well as exploration of new social structures and patterns. Together that will help ensure that the semantic web – Web 3.0 – truly enriches our everyday lives. Let’s start by having your agent message my agent, and we’ll do lunch. But before lunch, I’d like to introduce my colleague, Peter Sweeney, from Primal Fusion.