FOCUS K3D Research Road Map


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

FOCUS K3D Research Road Map

  1. 1. FOCUS K3D D1.4.1 Deliverable D1.4.1 of Task T1.4 Road map for Future Research Circulation: PU1 Partners: ALL Authors: Chiara Catalano (IMATI), Michela Spagnuolo (IMATI), Michela Mortara (IMATI), Bianca Falcidieno (IMATI), Andre Stork (FRAUNHOFER), Marianne Koch (FRAUNHOFER), Pierre Alliez (INRIA), Frederic Cazals (INRIA), Mariette Yvenec (INRIA), Wolfgang Huerst (UU), Remco Veltkamp (UU), Marios Pitikakis (CERETETH), Caecilia Charbonnier (MIRALab), Lazhari Assassi (MIRALab), Jinman Kim (MIRALab), Nadia Magnenat-Thalmann (MIRALab), Patrick Salamin (EPFL), Daniel Thalmann (EPFL), Tor Dokken (SINTEF), Ewald Quak (SINTEF) Version: 02 Date: Thursday, 01 April 2010 1 Please indicate the dissemination level using one of the following codes: PU=Public PP=Restricted to other programme participants (including the Commission Services). RE= Restricted to a group specified by the Consortium (including the Commission Services). CO= Confidential, only for members of the Consortium (including the Commission Services). 01/04/2010 1
  2. 2. FOCUS K3D D1.4.1 Copyright © Copyright 2010 The FOCUS K3D Consortium consisting of: CNR-IMATI-GE C.N.R. – Istituto di Matematica Applicata e Tecnologie Informatiche Dept. of Genova, Italy CERETETH Informatics and Telematics Institut – Center for Research and Technology Hellas, Greece EPFL Ecole Polytechnique Fédérale de Lausanne, Switzerland FRAUNHOFER Fraunhofer Institut für Graphische Datenverarbeitung, Germany INRIA Institut National de Recherche en Informatique et Automatique, France MIRALab Université de Genève, Switzerland SINTEF Stiftelsen SINTEF, Norway UU Utrecht University, the Netherlands This document may not be copied, reproduced, or modified in whole or in part for any purpose without permission from the FOCUS K3D Consortium. An acknowledgement of the authors of the document should be clearly referenced. All rights reserved. This document may change without notice. Project Coordinator: Dr. Bianca Falcidieno Organisation: IMATI Phone: + 39 010 6475667 Fax: + 39 010 6475660 E-mail: Project web site: 01/04/2010 2
  3. 3. FOCUS K3D D1.4.1 Document History Vers. Issue Date Stage Content and changes 01 2nd February 2010 80% First version submitted st 02 31 March 2010 100% Final submitted version 03 04 Executive Summary This document contains the deliverable D1.4.1 of the FOCUS K3D Coordination Action. Deliverable D1.4.1 is the final technical report of Task 1.4 entitled “Road map for Future Research”. This deliverable defines the guidelines envisaged by the FOCUS K3D Consortium for future research, consolidating input from all the different kinds of AWG activities (i.e. ad hoc meetings, questionnaires, workshops, dissemination events) carried out during the two years of the project. The document also includes the outcomes of the discussion with AWG members about the challenges of the road map, as presented during the final FOCUS K3D conference. 01/04/2010 3
  4. 4. FOCUS K3D D1.4.1 Table of Contents Chapter 0 : Preface................................................................................................. 5 Chapter 1 : Visionary scenarios .............................................................................. 7 1.1 Visionary scenario in BioTech, health & medicine .................................................. 7 1.2 Visionary scenario in Robotics ............................................................................ 8 1.3 Visionary scenario in Business & education, home & leisure.................................... 9 1.4 Discussion ...................................................................................................... 9 Chapter 2 : Definition of the field ......................................................................... 12 2.1 3D media representation: from geometry to semantics........................................ 12 2.2 Knowledge sources in 3D applications ............................................................... 14 2.3 Discussion .................................................................................................... 16 Chapter 3 : 3D in application domains .................................................................. 17 3.1 Medicine ....................................................................................................... 17 3.2 Bioinformatics ............................................................................................... 20 3.3 CAD/CAE and Virtual Product Modelling ............................................................. 23 3.4 Gaming and Simulation................................................................................... 26 3.5 Cultural Heritage and Archaeology.................................................................... 28 3.6 Discussion and conclusion ............................................................................... 30 Chapter 4 : The FOCUS K3D research road map .................................................... 32 4.1 Derive symbolic representations....................................................................... 34 4.1.1 Time line and dependence diagram ......................................................... 43 4.2 Goal-oriented 3D model synthesising ................................................................ 44 4.2.1 Time line and dependence diagram ......................................................... 48 4.3 Documenting the life cycle of 3D objects ........................................................... 49 4.3.1 Time line and dependence diagram ......................................................... 55 4.4 Semantic interaction and visualisation............................................................... 56 4.4.1 Time line and dependence diagram ......................................................... 62 4.5 Standards ..................................................................................................... 63 4.5.1 Time line and dependence diagram ......................................................... 64 4.6 Trends in Semantic Web research..................................................................... 65 4.7 Final discussion ............................................................................................. 68 4.7.1 Medicine scenario: liver segmentation ..................................................... 69 4.7.2 Bioinformatics scenario: from models to annotation................................... 70 4.7.3 Gaming and Simulation scenario: vessel design workflow ........................... 70 4.7.4 CAD/CAE and Virtual Product Modelling scenario: semantics based virtual 3D product modelling............................................................................................. 72 4.7.5 Archaeology and Cultural Heritage scenario: large-scale repository of 3D digital artefacts ......................................................................................................... 73 4.7.6 Robust geometry processing: practical benefits across the AWGs ................ 75 4.7.7 Conclusion........................................................................................... 77 References ........................................................................................................... 81 01/04/2010 4
  5. 5. FOCUS K3D D1.4.1 Chapter 0 : Preface 3D media are digital representations of either physically existing objects or virtual objects that can be processed by computer applications. They may be defined either directly in the virtual world with a modelling system, or acquired by scanning the surfaces of a real physical object. The production and processing of digital 3D content was and still is a traditional field of expertise of Computer Graphics, but only recently 3D entered the multimedia world: only in the last decade, indeed, has Computer Graphics reached a mature stage where fundamental problems related to the modelling, visualisation and streaming of static and dynamic 3D shapes are well understood and solved. Considering that nowadays most PCs connected to the Internet are equipped with high-performance 3D graphics hardware, it seems clear that in the near future 3D data will represent a huge amount of the data stored and transmitted using Internet technologies. Therefore, 3D media introduce a new kind of content in the established multimedia scenario, with the term multimedia characterised by the possible multiplicity of content, by its availability in digital form and its accessibility via electronic media. Text, audio, animations, videos, and graphics are typical forms of content combined in multimedia, and their consumption can be either linear (e.g. sound) or non-linear (e.g. hypermedia), usually allowing for some degree of interactivity. At the same time, research on semantic multimedia, defined by the deep integration of semantic web techniques with multimedia analysis tools, has shown how to use and share content of multiple forms, endowed with some kind of intelligence, accessible in digital form and in distributed or networked environments. The success of semantic multimedia largely depends on the extent to which we will be able to use them in systems that provide efficient and effective search capabilities, analysis mechanisms, and intuitive re-use and creation facilities, at the level of content, semantics and context (Golshani06). Going one step further, we could easily envisage semantic multimedia systems of a higher level of complexity, in which the generic architectural framework underpinning semantic multimedia systems could be extended to knowledge and data intensive applications, which have been historically developed in sectors that were quite far from multimedia and knowledge technologies. Throughout this document, we will illustrate this idea focusing on prospective applications of 3D content, as a rapidly emerging new form of media in the semantic multimedia panorama and an extremely challenging application context for semantic multimedia. 3D media, indeed, encompass all forms of digital content concerning 3D objects used and managed in networked environments, and not only fancy-looking graphics used in entertainment applications. 3D media are endowed with a high knowledge value carried either by the expertise needed to design them or by the information content itself. Currently, research on multimedia and semantic multimedia is largely devoted to pixel-based content, which is at most two-dimensional (e.g. images), possibly with the addition of time and audio (e.g., animations or videos), while 3D media are defined by vector-based representations. Due to its distinctive properties, 3D media make it necessary to develop ad hoc solutions for content analysis, content-based and context-based retrieval, modelling and presentation, simply because most 2D methods do not generalise directly to 3D (SpFa2009). The AIM@SHAPE Network of Excellence ( was the first EC project addressing the issue of adding semantics to plain geometrical shapes. The prototype infrastructure created by AIM@SHAPE and the ontologies developed by it demonstrated the potential gain for the Computer Graphics community of having detailed metadata attached to 3D models. Based on the experience of AIM@SHAPE, the FOCUS K3D project ( tackled the more complex problem of raising the interest of users in a number of application domains for semantics-driven processing of 3D data. 3D graphics are key media in many 01/04/2010 5
  6. 6. FOCUS K3D D1.4.1 sectors, among them the ones we selected for the application working groups in FOCUS K3D: Medicine and Bioinformatics, Gaming and Simulation, CAD/CAE and Virtual Product Modelling, Archaeology and Cultural Heritage. In these areas, representing a complex shape in the various stages of its complete life-cycle is known to be highly non-trivial, due to the sheer mass of information involved and the complexity of the knowledge that a shape can reveal as the result of a modelling process. The research road map presented in this deliverable is an attempt to synthesise the vision of the FOCUS K3D project on these themes, after extensive discussions with a variety of users in the application domains. The perspective of the road map is oriented towards challenges that exist more in the content production and sharing phase, rather than in the networking phase. Thinking about life fifteen years ago, you will discover how it is different from now and how some technologies we regularly use today were unimaginable at that time, mobile phones and internet above all. Therefore, we decided to start the document with three visionary scenarios, which imagine different aspects of life in 2040. What emerged from the stories in the end is that semantic 3D content will strongly permeate everyday life, even if not directly perceived. A discussion of where, how and which kind of 3D data will be involved follows to make the link between geometry and semantics explicit. In the second chapter, we take a step backward and briefly define our view of semantic 3D media, as originated from AIM@SHAPE, while, in the third chapter, we describe their current status in the application working groups selected in FOCUS K3D. The state-of-the-art reports produced by FOCUS K3D are an important source of information for the interested readers (see deliverables D2.1.1, D2.2.1, D2.3.1, D2.4.1), as they give an idea of the level of acquaintance with 3D semantics in these fields, with expected and natural differences between more established fields, such as CAD/CAE and Product Design, when compared to fields such as the Gaming and Simulation domains, where the potential of a concurrent use of semantics and digital 3D data is really high but relatively less perceived and reflected by current practices. Finally, chapter 4 provides the real road map, in which we have identified high-level goals that somehow represent the long-term challenges that the Computer Graphics community should face as targets for new and disruptive research, with a strong need to breach the borders of a single discipline, calling for a truly multi-disciplinary effort. Within each of these high-level challenges, we have also identified a number of mid-term challenges that, without being exhaustive of course, could be required building blocks for further important research advances. After the experience of FOCUS K3D and as reflected by the research road map, the path towards really effective semantic 3D content is still complex, but doable. As the research road map tries to convey, we believe that the potential of semantic multimedia technologies could be fully exploited in 3D and knowledge intensive application areas, where the processes deal with contents of multiple form and type, the processing workflows are guided by knowledge and semantics, and the working environment is usually distributed. Computer Graphics is ready to answer these challenges, but it needs a closer connection with Knowledge Management, Machine Learning, and Cognitive Modelling. 01/04/2010 6
  7. 7. FOCUS K3D D1.4.1 Chapter 1 : Visionary scenarios To illustrate how pervasive semantic 3D technology will be in about 30 years, we present in the following three visionary scenarios and discuss which aspects related to 3D and semantics are involved and should be tackled in the future. The three scenarios cover some areas where 3D media are a horizontal technology enabling the megatrends of the future, which are, as many experts agree (Fut09): • BioTech, health, medicine; • Robotics; • Business & education, home & leisure. 1.1 Visionary scenario in BioTech, health & medicine The scenario takes place in 2040. The current challenge for Dr Edwardes is to improve the quality of life of his patient Bill Thomson who is suffering from a serious arthritic problem in one of his knees. Bill not only suffers from unbearable pain, but also from the serious discomfort of having limited flexion. To begin with and check whether the pathology is amenable to drug treatment, Dr Edwardes decides to have Bill's genome fully sequenced using the latest high throughput sequencing technique. The output of the procedure being the collection of Bill's genes, Dr Edwardes aligns portions of this genome against those of other patients suffering from related disorders, so as to precisely identify the genetic determinants of the disease and check for possible treatments. The matching portions of the genome are visualised through a holographic projector. Interestingly for Dr Edwardes – but not for Bill – three genes appear as mutated, respectively HOOK-01, ESS-21, and RGid-07, each of them being involved in a different form of arthritis. While the first two mutations are amenable to drug treatment, no drug is known for the third one. Therefore, Dr Edwardes carries out a homology search in the Advanced Protein Data Bank to see whether the structure of proteins whose sequence is similar to that of the protein coded by RGid-07 is known, and indeed finds one such structure. Screening a library of candidate drugs through a virtual panel against this protein, Dr Edwardes identifies two of them with good binding affinity. These molecules are candidate drugs, and he hands the investigation to a colleague of his at the National Institute of Arthritis, so as to check whether the protein coded by RGid-07 can be patched by one of these drugs. Meanwhile, he and Bill decide to go for a surgical treatment. The latest advances in computer-aided medicine enable Dr Edwardes to tackle the challenge through function-driven modelling and simulation, with precise estimates of the risks so as to optimise both the decisions and the treatments. The novel methodology consists of driving the whole medical process by modelling, simulating and monitoring it with a constant eye on the end goal, here restoring the function (knee motion) while minimizing the pain. The first crucial decision is to move from non-operative to operative treatment: on the one hand, pain medication controls pain but does not change the underlying arthritic problem; on the other hand, it is possible to go ahead with prosthesis surgery albeit with some non-negligible risks. In the present case the results of a simulation on patient-specific data led Bill to decide to go ahead with surgery in order to place an implant. Different from previous approaches the shape of the implant is not chosen among a predefined set but instead optimised through the simulation so as to best restore the original motion amplitude and flexion of the knee. The implant is manufactured by a 3D printer, which can make physical 3D shapes from a variety of biologically compatible materials, while the actual surgical operation is robot-assisted. Another novelty is that the body is treated as a global system in order to best engineer the implant together with the treatment that comes with it in order to avoid infections and other complications such as late mechanical dysfunction of the implant. The present system mixes knowledge and geometric modeling of the implant. This requires simulating and monitoring 01/04/2010 7
  8. 8. FOCUS K3D D1.4.1 Bill’s organism at different scales ranging from the proteins to the organs through the cells, tissues and ligaments. After surgery Bill has to live with a machine, which monitors his body until the risk of complication is considered low enough to return to a normal life. One year later Bill asks Dr Edwardes whether or not he would be able to pursue an athletic activity, like when he was 25. To answer this question Dr Edwardes runs a new series of acquisitions and simulations (not reimbursed by social security!), which provide precise quantitative risk estimates. While looking at the simulation Bill has the strange feeling of being engineered as an industrial product by a computational engineer. This is, however, nothing compared to the fun of running once more the Boston marathon. 1.2 Visionary scenario in Robotics It is Monday morning and Cynthia is starting her new working week. She is a product development engineer at Robotical Inc., a company focusing on the development of the next generation of humanoids. Cynthia is specialised in behaviour modelling and simulation with a special focus on swarm behaviour. At the beginning of her career she developed a 3D sensor system and the embedded software for the interpretation of hand gestures of human beings for robot-human-interaction. Her system was able to build a 3D model of the situation and to interpret the gestures in a semantically described context. After breakfast Cynthia gets into her car. The car is the new AUTONObile 2040 model. Cynthia just tells the car the destination of her ride. The autonomous automobile checks the traffic situation, plans the trip and starts to move. All of a sudden a child runs across the street and the AUTONObile brakes instantly. Its semantic reasoning systems have come to the decision to brake after analysing the 3D scene, which is constantly generated and updated by various 3D sensors (global positioning system, lasers, radars, and cameras). The decision to brake is taken after considering the distance between the car and the child and rejecting the possibility to drive around the child because of opposing traffic. After the situation is cleared, the car continues. Having ‘delivered’ Cynthia to her office, the car searches for a parking space and waits for the next order. During the day, Cynthia is trying to optimise the shape and behaviour of the new humanoid she is developing with her colleagues at Robotical Inc. She is using the newest version of CAHIA (Computer-Aided Humanoid Interactive Application), which provides function-based constraints on free-space deformation techniques. With CAHIA all kinds of behaviour models can be embedded and simulation models are generated on the fly and evaluated in real-time. Not only one humanoid can be optimised but also strategies for jointly solving problems can be used. The humanoid is to work as a fire fighter to replace humans in this dangerous job. To achieve this, group behaviour in 3D environments needs to be trained and simulated. While waiting for Cynthia, her intelligent house communicates with AUTONObile and asks to bring some beverages from the supermarket. The car uses the drive-through option of the supermarket and a service robot is putting the beverages into its trunk. In the meantime the car’s traffic system receives the information that weather conditions will change rapidly within the next hours and snowy roads are expected. So on the way back to Cynthia’s office it drives to the next garage to have the tires changed. After her working day, Cynthia calls the car to fetch her at the entrance of her office and they start their way home. During the drive, the car is informed about an accident that happened on the road ahead. Thus, it is starting to re-plan the trip. This is done in online- communication with the other vehicles participating in traffic to avoid jams due to too many cars planning the same route. In addition to the street map, also the 3D profile of the landscape is taken into account to optimise energy consumption. AUTONObile returns Cynthia safely to her home. Cynthia is very happy with her personal mobility assistant. Also her children appreciate their personal chauffeur and Cynthia is unworried since she knows that the grandparents will supervise the safe and comfortable drive she already programmed to allow only for certain routes, e.g. to their friends and the football stadium. AUTONObile recognises the people authorised to drive using 3D face recognition together with voice analysis, a system Cynthia developed herself after finishing the gesture 01/04/2010 8
  9. 9. FOCUS K3D D1.4.1 recognition system. 1.3 Visionary scenario in Business & education, home & leisure When Semantha gets into her next sleeping phase, the mattress adjusts automatically, and the room lighting changes slightly, to anticipate her waking up soon. After she gets up, she prepares for going to work, and leaves a message to her daughter Alice, that will be synthesised by her private avatar with her own voice. Semantha is a 3D modeller at the gaming company 'Iterative Life'. When she approaches the company building, her gait and face are recognised, and entrance is permitted. She is currently working on the design and modelling of a collection of characters for a new game: "Parallel Universe Loophole Paradise". The characters in the game, the PULPese, have feet like wheels, arms like fins, and ears like wings. In her 'Evolving Personalised Information Construct' (Epico, marketed by GoogleZone, the joint venture of Google and Amazone) Semantha sees that she has modelled the feet last week. According to the schedule, she now must design the accompanying gadgets for chiropody. She invokes the encyclopaedic dictionary to learn about that, and the semantic network helps her to link these concepts to the maintenance tools for wheels. In the collection of instruments she wants to select those instruments that somewhat fit to the feet and can be used as a start for modelling new ones. Then, she activates her “FindMeShape”, the shape matcher tool able to select the appropriate models for the redesign process. She is interrupted by her iWant, her personal digital life assistant, a seventh sense to her. She has to collect Alice from school. On her way, she consults City 3.0 to find a restaurant near school and near the transportation dock, so that they do not lose time on their way to the afternoon event. One of her friends that are currently also signed in to City 3.0 tells her of a good place to eat. They decide to spend the afternoon visiting the Clonehenge area, a replica of the Stonehenge site that disappeared under water after the sudden ice meltdown catastrophe in 2025. When Alice puts on her glasses, she sees the enhanced environment consisting of projected 3D media and the real world. Through the looking glass, Alice sees a wonderland with a lot of cultural information on this piece of heritage, and she gets really affected by experiencing the past. When they finally get back home, Semantha continues her work to meet the planning schedule, while Alice is doing homework with her iCat. 1.4 Discussion In the three stories above, we tried to envisage how technology will influence and support different aspects of our future life. Although it may not be obvious, 3D multimedia play a key role in the megatrends identified. In all the three scenarios futuristic technology makes strong use of 3D data: since our perception of the world is 3D, the advanced tools of the future should be able to support it as efficiently and effectively as (and together with) any other kind of multimedia. The interaction between real and virtual world will be tighter: virtual avatars will replace humans, while intelligent environments will become the human-computer interface. As a consequence, one of the main research trends refers to the description and understanding of the surrounding world, which have to rely on efficient methodologies for modelling and processing 3D data. Another prominent aspect is the personalisation of the digital content and technology according to the context and the user. Therefore, the formalisation of the data cannot be static, but it should vary dynamically according to the purpose, the context and the user. In particular, the BioTech, health & medicine scenario implies a function-centric modelling and simulation of the human body, which relates different scales ranging from proteins to organs. In such a context where the human body is simulated as a complex system, semantic plays a key role to model the knowledge that links the various scales and the border between medicine and computational engineering becomes fuzzier. The future of computer-assisted medicine will certainly require thinking in terms of goal-oriented modelling and simulation 01/04/2010 9
  10. 10. FOCUS K3D D1.4.1 (with the goal being a knee that functions well in terms of motion amplitude, and with no pain), where some parts of the data come from knowledge about the generic human anatomy and other parts come from patient-specific acquisition and semantic annotation both before and after surgery. Furthermore, as medicine is not an exact science and developed countries care increasingly about legal issues and quality assurance, precise semantic description and estimates of the risks of any physical medical procedure will probably be of increasing importance. While generic knowledge about the human body model will certainly be of crucial importance for statistical risk estimation, a patient-specific estimate will require semantics- based analysis of massive data sets coming both from past simulations and real experiments carried out on other patients, which show some similarity. In addition, the adoption of theragnostics is investigated more and more, which is a treatment strategy that combines therapeutics with diagnostics. It associates both a diagnostic (semantically rich) test that identifies patients most likely to be helped or harmed by a new medication, and a targeted drug therapy based on the test results. Bioinformatics, genomics, proteomics, and functional genomics are molecular biology tools essential for the progress of molecular theragnostics. Generic human anatomy, as well as patient specific data, will be modelled and integrated on 3D replicas of the physical parts and used to attach information and run simulation models that will be tailored for each patient. Finally, and instead of thinking of computer-aided medicine only when the disease or pain has already occurred, the future of computer-aided medicine will most probably consist of increasingly preventive medicine, where modelling and simulation will again be put to use for monitoring, predicting and avoiding dysfunction as much as possible. In addition, robots will have an increased role in assisting surgery with high precision. Not only in medical applications the number of robots will be huge in the future, as technology and market forecasts indicate. Not only their population will increase in terms of sheer numbers but they will also drive some key technology developments and will become ever more intelligent. Marvin J. Cetron, President of Forecasting International and a member of the world Future Society board, forecasts (Fut09) that: • until 2020 we will have self-diagnostic and self-repairing robots in self-monitoring infrastructures for almost any job in homes and hospitals; • until 2030 the robot population will surpass human population in the developed world, and • in 2040 robots will completely replace humans in the workforce. Robots can be regarded as intelligent machines, and semantic 3D models are key to their development and for their successful use. As pointed out in the robotics scenario, to orient themselves in the real world and to communicate with human beings and other robots, they first and foremost need to build up an understanding about the scene they are surrounded by. It is not sufficient to create just a point cloud or a static (or dynamic) distance field – e.g. to avoid collisions –, the robot has to create a ‘mental’ model from the objects it is surrounded with. It needs to understand the meaning of those objects, it needs to know the actions they may perform, it needs to understand their dynamics if they are moving, in order to be able to plan its own actions. These are issues related to understanding the 3D world, valid not only for robots but for all kinds of interaction between virtual entities and humans. The outdoor navigation domain is a great challenge for three-dimensional perception. One key to successful navigation in real world environments is the robot’s ability to reason about its environment in three dimensions, to handle unknown space in an effective manner and to detect and navigate in environments with obstacles of varying shapes and sizes. We believe that the problem should be addressed by a tighter coupling of geometry processing methods with cognitive science, so that the properties characterising the shapes can be better aligned with features that are used by humans to “understand and classify” shapes. Also machine learning or similar statistical 01/04/2010 10
  11. 11. FOCUS K3D D1.4.1 methods are expected to play a key role to cope with the complexity and variability of 3D object shapes, and as tools supporting the automatic segmentation of massive datasets. To be able to do so, the robot needs semantic and geometric 3D descriptions of reference objects (learnt/trained knowledge about objects potentially being present in the scene and their functionality). Such models have to be ‘implanted’ into the robot possibly along with self- learning mechanisms, i.e. they have to exist when the robot ‘comes to life’, which takes us to the development phase for robots. There is already a growing tendency to introduce high-level semantic information into robotic systems. This tendency is visible in many areas, including mapping and localisation, robotic vision, human-robot interaction, and the increasing use of ontologies in robotics. Today, robots are systems consisting of mechanics, electronics, and software. Some behave in a pre-programmed way, some react to their surroundings. In any case the development process of robots also requires semantic 3D models (not only the use phase of a robot). Robots can be built more rapidly if functional-oriented design and reuse of functional components are intrinsically supported by virtual product development tools along with advanced simulation techniques. Ontological information is has increasingly being used in distributed systems in order to allow for automatic reconfiguration in the areas of flexible automation and of ubiquitous robotics. There is a need to use ontological information to improve the inter-operability of robotic components developed for different systems. Also in the Business & education, home & leisure scenario, we see that a number of new technologies will be present, which allow for an integrated approach to search for and find functionalities, are built on the massive semantic annotation of 3D media, exploit the re-use of personalised information, depend on the connectedness between real and virtual worlds, and utilise location-aware ambient intelligence and smart objects, which altogether make the production and management workflows much more effective, and the leisure and care tasks much more affective and satisfying than in the old days before 2010. Although some of the above technologies are currently being developed, there is still a long road to go. Through professional production, user content generation, and digitization and preservation of cultural heritage, large amounts of digital media such as images, video, music, three-dimensional objects and environments are becoming available. Tools for creation, editing, searching and interaction have been developed by now. New techniques are being developed to build games, model 3D worlds, and create virtual prototypes of new products or architectural plans. The next generation of applications and environments will require more natural interaction and intelligence to become more dynamic, affective, and satisfying. Tools are necessary to create virtual persons that are endowed with speech and social and emotional intelligence. Toolboxes and interfaces must be created to add ambient intelligence to working and living spaces. These will lead to newly created products and services for personalised information finding, connecting real and virtual worlds, experiencing past culture, and adopting its lifestyle. Functionality and elements that are currently missing are rich semantic annotations of the world and environments, massive semantic annotation of 3D objects, localisation in 3D environments, and tangible 3D visual interfaces. In the next chapter, our view of 3D semantic multimedia will be introduced and the current status of the field will be briefly described; chapter 3 will then give an overview of some application fields. 01/04/2010 11
  12. 12. FOCUS K3D D1.4.1 Chapter 2 : Definition of the field In this chapter, we will introduce our view of semantic 3D media, taking into account the perspective of researchers in the field of Computer Graphics as well as the perspective of the users in the reference application domains. Researchers in Computer Graphics are indeed key players in the process of formalising the semantics through knowledge technologies as they are the experts able to develop the appropriate tools and schemes for documenting and sharing 3D media representations, in close cooperation with experts in different domains. In the following, Section 2.1 will discuss the evolution of 3D modelling paradigms, from the traditional geometry-oriented to the emerging semantics-driven approaches. In Section 2.2, we will outline the sources of knowledge that we believe need a formalisation and a more effective management much more integrated with the geometric modelling pipeline of the object’s shape. A discussion in Section 2.3 concludes this chapter. 2.1 3D media representation: from geometry to semantics Knowledge technologies can be effectively used in complex 3D application fields if the underlying 3D modelling approach is able to support and encapsulate the different levels of abstraction needed for describing an object’s form, function and meaning in a suitable way. The EC Network of Excellence project AIM@SHAPE proposed an evolution of the traditional geometric modelling paradigm towards a semantics-based modelling paradigm, which is consistent with the different description levels used for traditional 2D media and reflects an organisation of the information content that ontologies dealing with 3D application domains could nicely exploit. The use of computers has revolutionised the approach to shape modelling, opening new frontiers in research and application fields: Computer Aided Design, Computer Graphics and Computer Vision, whose main goal is to discover basic models for generating and representing shapes. In the beginning, this effort gave rise to research in geometric modelling, which sought to define the abstract properties, which completely describe the geometry of an object, and the tools to handle this embedding into a symbolic structure. The visual aspect of shapes has deeply influenced the development of techniques for digital shape modelling, which have mainly focused on the definition of mathematical frameworks for approximating the outer form of objects using a variety of representation schemes. The same geometry can be represented by different approximations (e.g. meshes, parametric surfaces, unstructured point clouds, to cite a few), each representation being chosen according to its advantages and drawbacks with respect to application purposes. For example, complex triangulated meshes are badly suited to interactive animation; simpler representations such as point set surfaces can be sufficient for special classes of applications (e.g. ray-tracing techniques only require ray-surface interrogations). The conversion between distinct representations is still a delicate issue in most systems but there are tools to derive triangular meshes from the majority of representation schemes. The selected representation model will eventually be mapped into an appropriate data structure, which is computer- understandable and devised according to optimisation and efficiency criteria. It is important to point out that for the development of semantic 3D media the conceptualisation of the geometry is not an issue: geometric modelling is, by itself, the answer of the Computer Graphics community to the need of defining and expressing in formal terms concepts and relationships concerning the geometric representation of 3D media. Shape models and related data structures encapsulate all the geometric and topological information needed to store and manipulate 3D content in digital form. While the technological advances in terms of hardware and software have made available plenty of tools for using and interacting with the geometry of shapes, the interaction with the semantic content of digital shapes is still far from being satisfactory: we still miss effective and robust tools to manage non-spatial information, i.e. the information not related to the shape of 01/04/2010 12
  13. 13. FOCUS K3D D1.4.1 the object (e.g. ownership, material, price) and the information related to the high-level conceptualisation of the object, either expressed as features (e.g. size, volume, compactness, presence of holes) or as synthetic concepts (e.g. what it is, what it is used for). Stated differently, the semantic gap - the lack of coincidence between the information that one can extract from the visual data and the interpretation that the same data have for a user in a given situation (Smeulders00) –is still far from being filled. To tackle this problem we may follow two complementary paths: on the one hand, it is possible to integrate the missing information through the definition of concept-based metadata (either by following a formal conceptualisation encoded in an ontology or by allowing a free tagging of the resources); on the other hand, it is possible to exploit state-of-the-art descriptors and analysis tools to extract content-based information from spatial information. Both of the mentioned approaches help to narrow the semantic gap: the design of ad-hoc ontologies and the development of tools for a versatile annotation of 3D objects are part of the plan, but yet another important element is crucial for the description process: the role of the user. In fact, it is not sufficient to provide a generic interpretation of the data, the interpretation should take the context into account. The awareness of the context should enable a dynamic and user-oriented description in which only the most relevant content-based descriptors are activated and combined. The integration of concept-based and content-based approaches and their embedding in a context dynamically defined by the user could be an efficient solution, whose evaluation was one of the main goals of FOCUS K3D through the Application Working Groups. This integration can be achieved if the 3D content is organised in a way that takes into account and supports reasoning at different levels of abstraction and that goes beyond the limits of the pure geometry. The conceptualisation of the knowledge domain has to adhere to a suitable organisation of the geometric data about the shapes. In AIM@SHAPE, shapes were defined as characterised by a geometry (i.e. the spatial extent of the object), describable by structures (e.g. form features or part-whole decomposition), having attributes (e.g. colours, textures, or names, attached to an object, its parts and/or its features), having a semantics (e.g. meaning, purpose, functionality), and possibly having interaction with time (e.g. history, shape morphing, animation) (Falcidieno et al. 2004). Leaving the traditional modelling paradigm, a simple one-level geometric model has to be replaced by a multi-layer view, where both the geometry and the semantics contribute to the representation of the shape. At the same time the structure is seen as an important bridge towards the semantics, as it supports a natural manner to annotate the geometry with semantic information. The key idea is that, while the geometry of a shape is unique, its structure can be defined in many possible ways, each contributing to the creation of a specific structural model, or view, of the shape. These views of the shape make it easier to interpret the geometry according to different semantic contexts and attach appropriate content to the relevant parts of the shape. To realise the shift towards this new modelling paradigm, it is also necessary to develop new and advanced tools for supporting semantics-based analysis, synthesis and annotation of shape models. They correspond to image analysis and segmentation for 2D media: features of a 3D model are equivalent to regions-of-interests in images. There is, however, a different input and a slightly different target, as image analysis is trying to understand what kind of objects are present in the scene captured by the image, while in 3D segmentation the object is known and it has to be decomposed into meaningful components that might be used to manipulate and modify the shape at later stages. From a high-level perspective, the main differences concern the different nature of the content: descriptors used for 2D images are concerned with colour, textures, and properties that capture geometric details of the shapes segmented in the image. While one-dimensional boundaries of 2D shapes have a direct parameterisation (e.g. arc length), the boundary of arbitrary 3D objects cannot be parameterised in a natural manner, especially when the shape exhibits a complex topology, e.g. many through-holes or handles. Most notably, feature extraction for image retrieval is intrinsically affected by the so-called sensory gap: “The sensory gap is the gap between the 01/04/2010 13
  14. 14. FOCUS K3D D1.4.1 object in the world and the information in a (computational) description derived from a recording of that scene” (Smeulders00). This gap makes the description of objects an ill-posed problem and casts an intrinsic uncertainty on the descriptions due to the presence of information, which is only accidental in the image or due to occlusion and/or perspective distortion. However, the boundary of 3D models is represented in vector form and therefore does not need to be segmented from a background. Hence, while the understanding of the content of a 3D vector graphics remains an arduous problem, the initial conditions are different and allow for more effective and reliable analysis results and offer more potential for interactivity since they can be observed and manipulated from different viewpoints. Finally, at the semantic level, which is the most abstract level, there is the association of specific semantics to structured and/or geometric models through annotation of shapes, or shape parts, according to the concepts formalised by a specific domain ontology. In the Figure below, a possible shape analysis pipeline is shown: in (a) the digital model of a chair obtained by acquisition is shown, in (b) the shape represented in structural form can be analysed in the domain of knowledge related to Computer Animation and the regions of possible grasping are annotated accordingly (see (c)). The aim of structural models is, therefore, to provide the user with a rich geometry organisation, which supports the process of semantic annotation. Consequently, a semantic model is the representation of a shape embedded into a specific context, and the multi-layer architecture emphasises the separation between the various levels of representations, depending on the knowledge embedded as well as on their mutual relationships. (a) (b) (c) From geometry to semantics The multi-layer view of 3D media resembles the different levels of description used for other types of media, but there is a kind of conceptual shift when dealing with 3D media: here, we have the complete description of the object and we want to be able to describe its main parts, or features, usually in terms of low-level characteristics (e.g. curvature, ridges or ravines). These features define segmentations of the shape itself, which is independent of a specific domain of application but that carry a geometric or morphological meaning (e.g. protrusions, depressions, and through holes). 2.2 Knowledge sources in 3D applications Effective and efficient information management, knowledge sharing and integration have become an essential part of more and more professional tasks and workflows in Product Modelling, one of the first fields where semantics came into play. However, it is clear that the same applies also to other contexts, from the personal environment to other applied sectors. There is a variety of information related to the shape itself, to the way it has been acquired or modelled, to the style in which it is represented, processed, and even visualised, and many more aspects to consider. The description of a shape is intrinsically not unique and varies according to both the application and user contexts. Therefore, the abstraction levels used to process or reason 01/04/2010 14
  15. 15. FOCUS K3D D1.4.1 about 3D media should correspond to the mental models used to answer questions such as “what does it look like?”, “what is its function?”, thus making it possible to model, manipulate and compare the various 3D media in a semantics-oriented framework. Therefore, the ingredients needed to implement a 3D semantic application should definitely include a conceptualisation of the shape itself, in terms of geometry, structure and semantics, and of the knowledge pertaining to the application domain. In order to fulfil the requirements of complex 3D applications, we need tools and methods to formalise and manage knowledge related to the media content and to the application domain, at least at the following levels [SpFa2007]: • knowledge related to the geometry of 3D media: while the descriptions of digital 3D media can vary according to the contexts, the geometry of the object remains the same and it is captured by a set of geometric and topological data that define the digital shape; • knowledge related to the application domain in which 3D media are manipulated: the application domain casts its rules on the way the 3D shape should be represented, processed, and interpreted. A big role is played by the knowledge of the domain experts, which is used to manipulate the digital model: for example, the correct manner to compute a finite element mesh of a 3D object represented by free-form surfaces is subject also to informal rules that should be captured in a knowledge formalisation framework; • knowledge related to the meaning of the object represented by 3D media: 3D media may represent objects that belong to a category of shapes, either in broad unrestricted domains (e.g. chair, table in home furniture) or narrow specific domains (e.g. T-slots, pockets in mechanical engineering). The shape categories can also be defined or described by domain-specific features, which are the key entities to describe the media content, and these are obviously dependent on the domain. The first bullet point of the list is concerned with knowledge, which has geometry as its background domain. There are a variety of different representations for the geometry of 3D media that cannot be simply reduced to the classical Virtual Reality Modelling Language (VRML) descriptions and its variations, as currently supported by MPEG-4. Here, the view of 3D media is more concerned with the visualisation, streaming and interaction aspects than with requirements imposed by applications. 3D geometry, as used in applications, has to do with a much richer variety of methods and models, and users might have to deal with different representation schemes for the same product within the same modelling pipeline. In this sense, describing the content of 3D media in terms of geometric data is much more complex for 3D than for 2D media. There are many attributes and properties of 3D models that scientists and professionals use to exchange, process and share content, and all these have to be classified and formalised. The second bullet point refers to the knowledge pertaining to the specific application domain, but it has to be linked to the geometric content of the 3D media. Therefore, if we want to devise semantic 3D media systems with some reasoning capabilities, we have to formalise also expert knowledge owned by the professionals of the field. Finally, the third bullet point has to do with the knowledge related to the existence of categories of shapes; as such, it is related both to generic and specific domains. Usually in 3D applications, it is neither necessary nor feasible to formalise the rules that precisely define these categories in terms of geometric properties of the shape, besides very simple cases. However, due to the potential impact of methods for search and retrieval of 3D media, there is a growing interest in methods that can be used to derive feature vectors or more structured descriptors that could be used to automatically classify 3D media. 01/04/2010 15
  16. 16. FOCUS K3D D1.4.1 2.3 Discussion 3D media content is growing both in volume and importance in more and more applications and contexts. In this chapter, we have introduced the issues related to handling this form of content from the point of view of semantic media systems, focusing on the level at which we should be able to capture knowledge pertaining to 3D media. 3D applications are characterised by several knowledge-intensive tasks that are not limited to browsing and retrieving properly annotated material, but deal with the manipulation, analysis, modification and creation of new 3D content out of the existing one. While this is true also for traditional 2D media applications, the tools and expertise needed to manipulate and analyse 3D vector-based media are still the prerogative of a rather specialised community of professionals and researchers in the Computer Graphics field. In this sense, the Computer Graphics community could make a significant contribution: on the one hand, it could provide modelling and processing tools able to handle and exploit the semantics of the shape, and interaction and visualisation tools, which allow users to interact with the shape in an intelligent way; on the other hand, it could contribute to comprehensive schemes for documenting and sharing 3D media and related processing tools, to be linked and further specialised by experts in different domains. Semantics-aware classification of available tools for processing the geometry and manipulating 3D shapes will be an important building block for the composition and creation of new easy-to-use tools for 3D media, whose use and selection can be made available also to professional users of 3D who are not experts in Computer Graphics. The use of 3D is indeed spreading out of the traditional communities of professional users and will soon reach a general and inexperienced audience. The role of experts in 3D modelling for the development of semantic 3D media is twofold: on the one side, the identification of key properties for the description of 3D media and processing tools, and on the other side, the contribution to the development of advanced tools for the interpretation, analysis and retrieval of 3D content. It is evident that if we want to be able to reason about shapes at the geometric, structural and semantic level, then we have to be also able to annotate, retrieve and compare 3D content at each of the three layers (SpFa2009). In the next chapter the current status of semantic 3D media in the four Application Working Groups explored during the FOCUS K3D project will be described, while in chapter 4 the grand challenges identified along the way will be stated together with a list of open issues necessary to reach these goals. 01/04/2010 16
  17. 17. FOCUS K3D D1.4.1 Chapter 3 : 3D in application domains FOCUS K3D has identified four representative application areas that are both consolidated (Medicine and Bioinformatics and CAD/CAE and Virtual Product Modelling) and emerging (Gaming and Simulation and Archaeology and Cultural Heritage) in the massive use of 3D digital resources coupled with knowledge. The selection of these different application areas was motivated by two reasons. First, all these fields face a huge amount of available 3D data; second, the use of 3D data is not only related to visual aspects and rendering processes, but also involves an adequate representation of domain knowledge to exploit and preserve both the expertise used for their creation and the information content carried. Among the first actions of FOCUS K3D, a state-of-the-art report (STAR) was produced for each of the four application areas to report on the differences and similarities in the approaches and desiderata of users in these representative areas (see deliverables D2.1.1, D2.2.1, D2.3.1, D.2.4.1). These four STARs were meant to identify the use of current methodologies for handling 3D content and coding knowledge related to 3D digital content in the different contexts, and also to identify possible gaps. Each STAR therefore reflects a snapshot of the current situation in the four subfields that all massively use 3D data, but which are distinct in their characteristics, requirements, status, and condition. In this chapter, we provide a summary of the STARs discussed from the perspective of the research road map, aiming at describing each of the five application areas (Medicine and Bioinformatics are presented separately for the sake of clarity) and discussing the current situations and limitations as perceived by the users in the different domains. 3.1 Medicine Computer-aided medicine is a rapidly growing field and recent advances in medical imaging yield 3D shape data that are more and more reliable, accurate and massive. Different medical applications such as diagnosis, radiotherapy planning, image-guided surgery, prosthesis fitting or legal medicine now heavily rely on the analysis and processing of 3D information (e.g., data measuring the spatial extent of organs, data about the electrical activity of the brain), requiring different criteria for the specific 3D content they use. 3D content is sometimes used for shape measurements, as basis for diagnostic information, or to establish statistical reference data. Therapy planning, image- guided surgery and assessments of post-operative results are other typical examples of use of 3D content. The starting point of a typical computer-assisted diagnosis session is often a set of image data acquired from CT, MRI, MicroCT or other medical imaging devices. The current goal is to generate a digital 3D model suited to visual inspection and analysis, but the goal could be much more ambitious, such as simulation, electromagnetic wave propagation and temperature distribution for hyperthermia, soft tissue deformation for cranial-maxillofacial surgery, biomechanics for orthopaedic surgery, computational fluid dynamics for rhino-surgery, electrical potentials for electro-cardiology, and many more. In between, we find not only a sequence of automatic image processing algorithms but also a true reconstruction pipeline, which involves data registration, segmentation, 3D geometry generation and processing. While it is now commonplace to store medical images in digital form, 3D content used in medical applications is far from easy to obtain: there is a labour-intensive pipeline to get from the raw images generated by the medical imaging devices to the digital reconstructed anatomical models suitable to medical applications, such as therapy planning or surgical simulation. The current assessment reveals that the pipeline is far from being automatic and requires a 01/04/2010 17
  18. 18. FOCUS K3D D1.4.1 lot of knowledge-based interaction, in particular for the segmentation of the images. Even after decades of research, a major challenge is to recover the inherent 3D models of anatomical structures from 3D medical images, as accurately and efficiently as possible. Surprisingly, the most common clinical practice for this task is an interactive correction of an automatic segmentation algorithm by a medical expert, which incorporates some knowledge throughout the pipeline. One important thread of research to facilitate this process is therefore devoted to the elaboration of semi-automatic tools such as magic wands, intelligent scissors or deformable models. As the resolution of the images always increases, it is however commonly admitted that such a labour-intensive interactive approach is not a long term approach. This motivates another thread of research devoted to the elaboration of fully automatic methods, which incorporate the knowledge through statistical or model-based approaches. One required feature is to let a door open for the expert to adjust the segmentation as a post-process in the few cases where the automated process fails. Similar issues arise for geometry modelling and processing, where one of the goals is to preserve the multi-domain labels during surface extraction and mesh generation. The practitioners resort to creating 3D models of organs or of human beings for simulation purposes. Those models are frequently input to finite element software, for example to simulate problems related to the hip joint from patient-specific data. This is a typical example of goal-oriented modeling, where the geometry of the organ to be modelled needs to fulfil the specific requirements imposed by the technical solution required to solve the objective, or function to be simulated, which might differ from those of the simple visual inspection: for finite element analysis, for instance, a fine volumetric mesh representing the boundary and interior of the bone is required, while for the simple visualisation of the bone shape a coarser triangulation of its boundary surface would suffice. Another example is the simulation of the human articulations for replacement by prosthesis. The different steps are bone resection, prosthesis alignment, and checking of motion amplitudes. In terms of knowledge, the parameters of interest are contact pressure within the cartilages, forces and muscle biomechanics. The connections to physiology are related to material properties as well as to bone motion and geometry. The current state of the art assesses that such a data process is considerably simplified when these parameters are stored into an ontology, and even more so if parts of the models of the geometry could be annotated with functional information or comments by the medical doctors. 3D data and knowledge are again tightly coupled in the simulation of laparoscopic surgery for training and practising. The goal is to simulate the planned surgery with a robot acting on a 3D model, so as to rehearse the gestures and to check the feasibility of the complete process. The knowledge involved is related to the anatomy, the forces applicable to the organs and the physical properties of a tumour. Virtual anatomy is perceived as a central problem concerned with the elaboration of 3D geometry models of human anatomy. 3D models must be augmented with knowledge about the function of the organs, as the primary end goal in computer-aided medicine is to obtain a human body, which functions well after surgery or treatment. The function itself cannot be decoupled from the topological and geometrical relationships between the organs. Virtual anatomy is important in medicine with respect to patient-specific computer-assisted therapy planning, as well as for industrial applications, such as virtual crash tests where human models are involved. It is common to distinguish between individual anatomy and generic anatomic atlases, which demonstrate anatomical structures and their relationships. Individual anatomies are of special interest when pathological situations need specific attention with respect to a medical treatment. Generic anatomy models are carefully designed using modelling software, whereas patient-specific anatomy models come as output of a scanning device, associated with point set based or image-based reconstruction algorithms. Another noticeable trend, in simulating and monitoring surgery and therapy is the replacement of generic anatomic atlases by patient-specific digital anatomical models. This trend results in the generation of more and more massive 3D content, thus requiring the help of knowledge technologies for classification, storage and retrieval of those data. 01/04/2010 18
  19. 19. FOCUS K3D D1.4.1 Monitoring therapy or following disease evolution requires storing and retrieving various data for each patient over long time scales. A typical workflow example is provided by a clinician in a department of neuro-rehabilitation, which routinely performs transcranial magnetic simulations (TMS). The data are initially acquired by MRI. The MRI images are then segmented and a so-called 3D anatomical project is created for each patient. Such project, which corresponds to a geometry reconstruction process, includes surface reconstruction and mesh generation for the head as well as for the cortical surfaces. This anatomical project is stored and reused for repeated TMS sessions performed for the same patient. Overall the evolution toward patient-specific modelling and simulation is recognised as an advance. However, the fact that either little or no reuse of data and processes is currently performed across patients is recognised as being very detrimental to the productivity. Intuitively, the more patients processed the more productive (and hence assisted) the expert would like to be. The experts and clinicians are conscious that the goal here is not only to reuse the data, but also to reuse the processes themselves through documenting the whole process. This challenge is where knowledge-based technologies must play a role. Knowledge technologies are increasingly required to manage structural, functional and topological information extracted from the medical data. More specifically, semantics is required at every step of the modeling and processing pipeline, which ranges from raw data to goal-oriented 3D numerical models through the crucial and yet current bottleneck step of data segmentation. Similarly, visualisation is always at the core of many medical applications such as diagnosis, surgical simulations, image-guided surgery and training. In the context of knowledge-based computer-aided medicine visualisation should be not only concerned with the plain rendering of the geometry, but also with semantics. While our current assessment is that geometry and semantics are most often decoupled when it comes to interaction and visualisation, some recent research experiments show that it would be very beneficial to combine both. For instance, the process of simulating a complex orthopaedic surgery is considerably simplified when both geometry and knowledge are stored into an ontology, as the practitioner can get an interactive, faithful simulation including both geometry and semantic information. Such simplification translates into a reduced surgery planning duration, which is critical for the practitioners. Having knowledge such as organ dependency and material properties combined with geometry is already a significant step toward efficient and reliable surgery planning. What is missing in the current knowledge representation is a detailed description of the link between the geometry (and topology) of an organ or tissue and its function. For example, the shape of a joint (e.g., a spherical cap) partially explains its function in terms of motion. The geometry and topology of an organ also often explains its function, such as connection to a bone or to another organ, etc. The simulation could thus be improved by integrating into its model the inherent functional constraints of the musculoskeletal system. Standards already play a prominent role in what is currently the most important use of knowledge technology in medical applications: description of anatomical features, pathologies and protocols. Examples are the Unified Medical Language System (UMLS), the Current Procedural Terminology (CPT), and the Foundational Model of Anatomy (FMA). These standards serve different purposes such as training and education, communication among practitioners, statistical analysis for health insurance, preventive medicine and legal issues. Current standards are primarily required to ensure the continuity of medical imaging data and the sharing of those data among care and research centres. The development of standards will most likely continue fostering the mutual enhancement of different imaging technologies. Standards for 3D medical content are however currently lacking, although they are likely to have in the near future a fundamental role in the field of computer-aided medicine where patient-specific anatomical modeling is of increasing importance. Similarly, the development of standardised procedures (including geometry and knowledge) for goal-oriented modelling and processing will be of increasing importance both for the sake of quality and for legal issues, when knowledge-intensive modeling and processing techniques will be routinely used. 01/04/2010 19
  20. 20. FOCUS K3D D1.4.1 3.2 Bioinformatics While computer science is concerned with the development of (generic) algorithms, which in general may be applied in a variety of settings, biology is more of a system-centric activity, since in general a panel of methods are used to investigate a particular system (molecule, cell, tissue, organ, organism). In biology, knowledge technologies are especially suited to structure the knowledge relative to a particular piece of this multi-scale puzzle, as well as to integrate data across the different scales. In this broad context, knowledge technologies in general and ontologies in particular are characterised by two main features. First, their development is under the guidance of biologists, since a precise understanding of the systems under investigation is necessary in order to select relevant annotations. Second, the knowledge technologies used cover the full range, from simple databases all the way to complex ontologies, as evidenced by the numerous tools which are made available from the web portal of the Protein Databank in Europe (PDBe). If one narrows down the application scope of knowledge technologies to applications dealing with 3D shapes, the natural setting in biology is that of structural bioinformatics. With structural biology being the sub-domain of biophysics and biology concerned with the investigation of the connection between the structure of macromolecules and their function, structural bioinformatics is concerned with the development of methods and algorithms meant to complement experiments in this perspective. Structural biology poses a number of difficult problems, among them the acquisition of reliable physical signals, the reconstruction of macromolecular models from these signals, the description of these models so as to design parameters best describing their biophysical properties, and finally, the utilisation of these models for prediction. In particular, predicting the 3D shape of a protein (the folding problem) and predicting the interaction of two or several partners (the docking problem) are clearly two of the main challenges in structural biology. When working on these challenges, since the shape of molecules is so important, all scientists resort to 3D content. This 3D content consists of a mix between geometry and biochemistry. The geometric side corresponds to molecular representations and conformations, either represented as a collection of balls, using Cartesian coordinates, or using internal coordinates (bond angles, valence and dihedral angles). On the knowledge side, one specifies the types of atoms and bonds, as well as a number of biochemical annotations (positional fluctuations for experimentally resolved structures, known partners of a protein, affinity constants, diseases the protein is involved with, etc.). The complexity of the concepts dealt with can again be seen from the various tools, which are accessible from the web portal to the European PDB at From a modelling perspective, the topics of interest depend on the application scenario. Experts in reconstruction from experimental measurements wish to output a model coherent with the physical measurements. Scientists designing drugs are especially interested in pockets on protein surfaces, since these accommodate drugs. Similarly, the affinity and specificity of protein–protein interactions are tightly coupled to geometric and biophysical properties of the molecules. Biologists wish to understand how the structure of molecules accounts for their function. This requires making progress on the two aforementioned mechanisms, namely folding and docking, which involves manipulating a mix of quantitative and symbolic information. We substantiate this claim by examining four situations, which exhibit a wide range of difficulties: • Characterising the packing properties of atoms. Native forms of proteins, that is biologically active forms, exhibit very specific patterns in terms of spatial properties, in particular regarding the packing properties of atoms. Packing properties are obviously numeric properties (for example atomic volumes). On the other hand, annotations of proteins are key: well-packed proteins are called native like, while proteins with loose packing are called decoys. In between these two extremes, a wide variety of situations occur, and some of them are particularly important for the description of diseases (e.g., 01/04/2010 20
  21. 21. FOCUS K3D D1.4.1 misfolding in amyeloid diseases); • Computing the depressions and/or protrusions of molecules. When docking two molecules in a blind fashion, i.e., without any a-priori knowledge, knowing where the binding sites is a must. Depressions/pockets and protrusions/knobs on molecular surfaces are good hints. In general, a pocket is defined in geometric terms (e.g., a concave region of sufficient volume, or the volume corresponding to the stable manifold of a maximum of the distance function to the molecular surface, etc.), but such quantitative models are not easily handled, and annotations such as flat pocket, a pocket with two crevices, etc. would help the design process. • Developing proper notions of curvature, in conjunction with solving models and binding modes. Curvature inherently refers to differential geometry, and there are indeed several strategies to quantify the curvature of a molecular surface. Aside from these quantitative statements, the qualitative description of regions of molecular surfaces in qualitative terms such as flat, convex, concave, saddle-like, etc. would help biochemists. • Partitioning a molecule into regions, which are likely to be flexible and rigid. Such regions are of utmost interest for flexible docking algorithms. Rigidity characterisation typically resorts to normal modes – the eigenvalues and eigenvectors of the internal energy of the molecular system about a minimum. These quantities are used to define molecular motion involved in deformations coupled to docking. Practically, annotations such as hinge motion, shear motion, etc. are more easily handled than the underlying geometric descriptions. Documenting the life cycle of 3D objects manipulated in bioinformatics naturally depends on this context. We shall discuss three examples illustrating the complexity of the documentation problem. Consider first a biophysicist who is experimentally solving the 3D structure of a protein: he/she aims at producing a final product, namely the 3D structure of the molecule. The structure submitted to the Protein Data Bank contains specific pieces of information, namely those imposed by the PDB file format. However, the difficulties handled along the process may or may not be documented. To make a long story short, a biophysicist interested in solving at the atomic level the structure of a given protein using X-ray crystallography has to go through the following steps: (i) introduction of the gene coding for this protein into a host cell, (ii) large scale protein expression and purification, (iii) crystallisation, i.e., crystal growth, (iv) X-ray diffraction analysis of this crystal and model reconstruction. Each of these steps might be challenging and may require years of work. Experiments along the way are written down in the Laboratory Information Management Systems, which is the memory of the process followed. Still for complex cases the whole process might be more similar to a handcrafted activity, and rationalising might be very difficult. It is also important to note that a person who managed to carry out a complex process will first exploit it before releasing the details. Consider now a person dealing with molecular modeling, who is working on the problem of inferring a putative complex from two unbound partners. Key questions contributing to the ability to run a reliable prediction are: which evidence of interactions between homologous proteins does one find in the biochemistry literature (this requires a thorough literature search, together with a precise compilation of experimental conditions)? What are the key regions in the interaction? Are additional molecules involved to stabilise the interactions (ions or water molecules)? Are there flexible regions? What is the role of electrostatics – which often plays a crucial role at early stages of the binding? These questions involve a subtle mix of relatively simple and complex tasks. Simple questions are those related to the search for proteins within a given homology threshold with respect to either a given reference protein, the classification and the search of protein folds, the search of protein atoms with prescribed biophysical properties (e.g., temperature factors), or the search for bibliographical pieces of information related to a molecule or a complex. 01/04/2010 21
  22. 22. FOCUS K3D D1.4.1 These tasks clearly benefit from knowledge technologies and can be easily documented. Nevertheless, there are other and clearly very difficult questions, for example handling flexible regions, computing precisely free energies, handling mutagenesis data, and understanding the precise connection between coarse and atomic modeling results. These questions in their full generality are still open. The route taken to produce a prediction cannot be fully and easily traced and there is no standard way to rationalise and document it. Finally, consider a pharmaceutical company designing a drug. The docking process just described is now a mere step in a much longer process. To begin with, the chemist will aim at identifying candidate drugs for the disease of interest. Drugs are small molecules, which underwent a wealth of experiments in the chemistry and biochemistry communities over the past five decades. The zoo of small organic molecules is rather stable. Simple rules involving the molecular weight or the solubility of a drug allow one to infer pretty reliably the chances for a molecule to be a lead – cf. Lipinski's rule of five (Lipinski et al. 2007). These pieces of information can be rationalised and organised into ontologies, and can also be easily traced in Laboratory Information Management Systems. The second step, docking, is more difficult to handle as seen above. Finally, upon identifying good hits thanks to a docking prediction, wet lab tests must be carried out, and finally clinical trials implemented. Overall, the whole process lasts more than a decade, and there is no standard way to trace it thoroughly. In fact, the Product Life Management within pharmaceutical companies is a major problem and certainly an obstacle to benefiting from the experience accumulated over a large variety of projects. This observation motivates the strategy of companies such as Dassault Systemes: having to some extent mastered the life cycle of manufactured objects, Dassault Systemes now aims at providing their tools and environments to pharmaceutical companies. To conclude, there is no general and unified way to handle the documentation of the life cycle of 3D objects encountered in bioinformatics and applications. In the most general setting, our understanding is not yet sufficient to document and elaborate easily simple knowledge technology models. The semantic interaction found in structural bioinformatics is not different from that found in other domains. As mentioned before, a number of tools have actually been developed and made accessible. Let us mention three of them, which are available from the EBI institute at, and allow running queries on the Protein Data Bank. The rationale for mentioning these three tools is that they are of increasing complexity in terms of knowledge technologies. The first one, PDBelite, allows the user to pre-define criteria to be used by a search engine. The second one, MSDpro allows the user to define its own logical queries using a drag-and- drop interface. Finally, the third one, MSDmine gives access to a complex data model allowing the user to take full advantage of the features of a relational database. Interestingly, the knowledge data made available from these services are rather elementary in terms of geometry, that is, no advanced annotation of geometric nature is provided. This is consistent with the complexity of the aforementioned mechanisms. The role and status of visualisation in the field of structural bioinformatics is rather interesting. On the one hand, molecules are 3D objects, and it is rather tempting to inspect their geometry so as to guess what the binding patches might be, or to assess the complementarities of the molecular surfaces of interacting molecules. On the other hand, the binding of two molecules is a very subtle mechanism, with a subtle interplay between enthalpic and entropic phenomena, and a mere 3D visualisation cannot account for these delicate mechanisms. However, the visualisation of 3D structures is important to develop one's intuition. It is also of prime importance to communicate with large audiences and promote this kind or research. Standards do not play a prominent role in structural bioinformatics. File formats, for example those that are used in the Protein Data Bank, come with standards. However, as seen above, the complexity of the biological processes involved, e.g., in the experimental resolution of a protein structure, is such that typically non standard problems are faced, which calls for 01/04/2010 22
  23. 23. FOCUS K3D D1.4.1 an unstructured part in the files themselves. These difficulties and also the variety of scenarios handled are clearly a hindrance to the exchange of information. 3.3 CAD/CAE and Virtual Product Modelling CAD/CAE and Virtual Product Modelling cover all ranges of man- made objects: electronics, transportation devices, tooling machines and many more. There are important differences between CAD/CAE and the other fields considered in FOCUS K3D. Medicine and Cultural Heritage are of global interest and collaboration amongst researchers, building ontologies, etc., has a long tradition, whereas product-developing companies face much harder competition amongst each other. Although the human body can be looked at as 'the most complex machine' known, it is just one type of object that Medicine and Bioinformatics are trying to decode, understand and conceptualise/formalise fully. In Virtual Product Modelling the objects (machines) modelled are much less complex than the human body but the range of different types is enormous and cannot be easily mapped into one scheme. This explains to a certain degree the different status in terms of ontologies (in virtual product modelling there are few) and standards in these areas. Nevertheless, the main problems of CAD/CAE are still comparable to those of the other fields, e.g.: • How to generate a model from a digitally acquired object? • How to model a shape easily? • How to model function or behaviour? • How to retrieve a model with certain properties? • How to handle all the information involved? Virtual Product Modelling is part of the virtual product developing process. The 3D modelling phase is typically fed with goal market characteristics, market/cost figures, requirements (aesthetic, functional, environmental, safety, etc.), concept sketches, and so on and so forth. Stylists and designers use CAD (computer-aided design) and CAS (computer-aided styling) systems to create virtual 3D models that foremost represent shape. The product functionality is usually modelled using a variety of behaviour/function modelling tools and is simulated with corresponding solvers. In the process, physical models still need to be created for styling reviews or to validate behaviour. In the styling phase, the shape of physical models is changed manually, which requires scanning the resulting 3D surface and re-creating a virtual model for it (re-engineering). In the simulation phase data from real experiments are used to tweak material models (to just name an example). The data created along the process typically reside in a multitude of data sources with a tendency to integrate product relevant data into a product data management or product lifecycle management system. Today most of the content of those systems is restricted to geometric 3D information, the whole semantics being in the heads of the engineers, and not explicitly documented in a computer-interpretable way. Although being the field where 3D media really started off (Sutherlands SketchPad 1965, company-owned CAD systems in the 1960s, etc.), the software in use today still requires a lot of manual intervention and tedious work and typically imposes certain concepts onto the user instead of giving him/her full freedom in choosing the best-suited tools at will. In the following we shortly present the state of play for the most important phases and kinds of software tools and give hints on current limitations. 01/04/2010 23
  24. 24. FOCUS K3D D1.4.1 CAD systems enable designers to create geometric models of products with the computer, to be reused and manipulated by the designers as needed. CAD systems were, and remain, highly technical software with an extremely rich feature set and functions for detailed design work. CAE tools are widely used in the automotive and aerospace industry. In fact, their use has enabled engineers to reduce product development cost and time while improving the safety, comfort, and durability of their products. The power of CAE tools has progressed to the point where much of the design verification is now done using computer simulations rather than physical prototype testing. Today’s shape representations in CAD (mainly NURBS) do not feed all purposes well, especially not those of simulations. For designing, the concept of creating a larger base surface from which certain parts are cut out by trim curves to create the final piece of surface is certainly artificial – it has no natural equivalent. Other representation schemes are needed that allow for more intuitive and easy shape manipulation and also enable easily integrating concepts of function-oriented design, etc. Simulations typically require a different kind of shape representation than provided by CAD systems, a discrete instead of a continuous one. This implies a tedious manual process of model preparation to suit different simulation needs. The integration of CAD and CAE using FEA (Finite Element Analysis) is today very complex due to the different principles for shape representation employed. In CAD, a volumetric object is described by shells described by a patchwork of mathematical surfaces representing the outer hull and inner hulls of the object, and data structures conveying the volumetric relationships. There is no requirement that adjacent surfaces match exactly, only that they match within specified tolerances. In FEA, the object has a complete mathematical description through watertight structures of trivariate parametric volumes. There is a fast growing research field in both the US and Europe concerning this discrepancy, addressing how to solve better this great interoperability challenge of CAD and FEA. This research field can be denoted as “Isogeometric representation and analysis” aiming at a common shape representation for both CAD and FEA to integrate the two disciplines and drastically improve the quality of Finite Element Analysis. Consequently, there is a need for investigating the concepts of this new approach both from a purely mathematical and from a semantic perspective to integrate it into current industrial information pipelines. Isogeometric representation and analysis employ new concepts in both the shape representation phase and the analysis phase. We will need ontologies that encompass these concepts and their relationships to the traditional concepts to facilitate interoperability between the new isogeometric systems and traditional CAD and FEM. Today, many companies use product data management (PDM) systems to store product related information, such as geometry, assembly structure, and materials. Considering the full product development process and the mechatronic disciplines (mechanics, electronics, software), there exists much more information spanning from requirements, to specifications, to mechanic, electronic and software behaviour models, to cross-domain dependencies, influences and effects, largely carrying the semantics of the product and not represented in PDM. The question is whether all the data should be in one PDM system. A more promising idea is to bridge the existing islands with semantic linking to manage knowledge effectively in order to reuse past positive experiences and solutions and avoid repeating past efforts or errors. When talking about knowledge in industrial companies, one of the important islands of information is the one that deals with the knowledge of the geometrical properties of the products manufactured, bought or integrated, and to be maintained during product changes. Therefore, models and methods for making explicit and maintaining links between geometric properties and domain specific knowledge have to be developed for improving both the specific process – either computer-aided or manually performed by the operator – and also the inter-process communication. This would be beneficial for the integration between design and simulation in product development, and in simulation during maintenance. Currently, searching in distributed information sources is hampering efficient work 01/04/2010 24
  25. 25. FOCUS K3D D1.4.1 procedures. When linking those information islands together, new search functionality is needed that spreads out over these sources, takes shape, function, metadata and semantics into account, and combines partial results to meaningful answers of the user’s questions. New partial/local 3D retrieval approaches and combined searches contribute to overcoming the current limitations of text-based or current global 3D content-based retrieval. Another aspect is the appropriate visualisation and navigation through results being returned from combined searches and user feedback for self-improving and self-learning mechanisms. Another current limitation in the downstream process of virtual assembly is the tedious way to create virtual kinematics and simulate this interactively within the virtual environment. Nowadays kinematics simulation is performed in the domain of CAD and CAE software packages, which need specially trained personnel and are designed to work on detailed product models without providing the degree of realism and interactivity virtual environments offer. Here, using semantics from CAD and parametric feature-based design for virtual kinematics in a Virtual Conceptual Design (VCD) platform would ease the process and support the early conceptual design phase as well as the embodiment design, allowing the immediate validation of assemblies and mechanisms by experiencing their behaviour during the modelling, simulation and assembling process. 3D model exchange between different systems has been and is still a great challenge with respect to model quality, as different CAD-systems and CAD-system kernels employ several approaches (and tolerances) to handle the inherit inaccuracies of STEP-type CAD-models. CAD-model check and repair have been on the agenda for many years and resulted in a number of standardisation initiatives: • AIA/ASD EN-9300 Long Term Archiving (LOTAR) addressing High Quality Geometry verification and validations and rules to execute. • ISO 14721.4, Open archival information system – Reference model. • ISO 10303-59 Product Data Quality (PDQ), Part 59: Integrated generic resource – Quality of product shape data. Since standards for CAD model quality are in place, the focus of advanced industry has now turned to the validation of the CAD-models exchanged (e.g., the STEP file, or the CAD-model generated by the receiving system), being an exact representation of or equivalent to the CAD-model in the sending system. This demand for equivalence checking poses new challenges to the CAD-model representation, to the semantic annotation of the CAD-model and to CAD-vendors in general, as industry requires more than the basic CAD-technology, CAD models, representation and algorithms to be available for validation. The ‘standard file format problem’ has to be addressed by academia and research institutes in the future. We need to develop a 3D file format addressing the following problems that are still open: • compatibility / interoperability / semantic mapping; • shape representation (procedural, parametric); • functional expressiveness; • transport of objects together with their interactively modifiable parameters. 01/04/2010 25