Your SlideShare is downloading. ×
0
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Metaphors, biases & learning partnerships
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Metaphors, biases & learning partnerships

767

Published on

From a talk to the Workshop on Integrated Strategy on Healthy Living and Chronic Diseases, Ottawa, February 2011. …

From a talk to the Workshop on Integrated Strategy on Healthy Living and Chronic Diseases, Ottawa, February 2011.

Knowledge exchange is more than just a compilation or warehousing of data or information. To generate new knowledge we must infuse data with new meaning. We do this not in an additive way from single actions and data-bits, but by creating a story about the overall pattern embedded in events and data and then using that story to understand more clearly the events and data that gave rise to it.

Published in: Business, Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
767
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
8
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • In talking about systems of KE, I wanted to spend a few minutes talking about how we use metaphors and internal narratives to learn; how our learning can be constrained by cognitive biases; how learning partnerships can help overcome both of these and help as well to create the new meta-stories that serve as the basis for innovation. Lastly, I wanted to review a few simple things to keep in mind when designing and using KE systems. click
  • KE is more than just a compilation of data or information. To generate new knowledge we must infuse data with new meaning. click We do not build new knowledge in an additive way from single actions & data-bits, but by forming ideas about the overall patterns embedded in events & data and then using these patterns to understand more clearly the events & data that gave rise to them. This hermeneutic approach builds on the idea that one's understanding of the whole is established by reference to the individual parts and one's understanding of each individual part is with reference to the whole. Neither the whole nor the part can be understood without reference to one another. click This circular character of human understanding does not inhibit new interpretations of knowledge; rather, it stresses that the meaning of new knowledge must be found within the assumptions and paradigms of its cultural, historical, and social context. Thus new knowledge must involve not only an exchange of data, but also an exchange of the organizing metaphors that help us make sense of the data we see. The more data, meaning and knowledge we exchange, the more value we create. It’s like they say in economics 101 with regard to that other great intangible, money, the more it is turned over, the more wealth is created. click
  • The problem faced by Public Health authorities everywhere is that knowledge tends to be structured in silos. This is due in part to the predispositions of public sector bureaucracies but more so by the nature of knowledge specialization. As people become more and more expert, the smaller is the community in which they feel they belong – ie the smaller the group who share the same organizing metaphors and narratives for dealing with new information. For instance, with the Knowledge Network there may be six major content areas: cancer, diabetes, obesity, cardio-vascular, lung, and mental health and the people involved in each knowledge group tend to speak to others in their own speciality. Why? Because its easier. They speak the same language. Look at things similarly. click Many organizations try to augment the silos with formal horizontal linkages, usually assuming knowledge flows in predictable channels. It doesn’t. Knowledge development is often random. We don’t know where the next innovation will come from. click Wikis, blogs and other social media presume no such predetermined channels and permit users to self select for the information and relationships they want. Given our inability to predict where innovation will come from, this latter approach of democratizing information is more in keeping with maximizing innovation. If we allow for Linus’ Law ( that’s Linus Torvald of Linux not Linus from Peanuts. Torvald is famously quoted as saying “the more eyeballs, the fewer bugs”) and get more information out there to more people, we also need to encourage ‘mashups’ of the narratives which provide a conyext for this new information and that help us make better sense of it. Being exposed to new information is no guarantee that it will be seen as relevant or even be seen at all. click
  • We absorb new information mainly by putting it into a category which we are already familiar with. Click Science works mainly by metaphor Without that categorization, we may not only under-appreciate its value but we may also not even recognize it. Click W e [do] not perceive a signal from the outside world unless it is relevant to an option for the future that we have already worked out in our imaginations There was some famous research at Harvard Medical School where they tested perception as a learned response. They raised 2 groups of cats in 2 different environments – one with no horizontal lines and the other with no vertical lines. After a year or so, the adult cats could not ‘see’ the lines from the other environment. It suggests that we see in very physical way only those things we have learned to see. As a result of these potential blind spots, de Geus recommends that organizations invest in the development of different scenarios and future narratives in order to reduce the possibility of not seeing new signals because of current operating assumptions. He refers to this approach as creating “memories of the future” click According to Donald Schon, when it comes to policy choices what’s important he says is what we think is possible than the relative objective merits of a given choice Resolving health & social policies has “more to do with the ways in which we frame the purposes to be achieved than with the selection of optimal means for achieving them” click What is possible is determined by the metaphors and stories we use that help us make sense of the world. If new knowledge and innovation is the goal, then KE must move beyond simply the warehousing of data to exercises for creating new meta-frames click
  • Research ( Tversky & Kahneman, 1981) shows that by changing the frame we use, we inevitably change the collective decisions and actions that follow from it If we believe that health is the prerogative of each individual and family, then we’ll likely view healthcare as a system to support the choices that individuals and families make. Bad health outcomes in this view are the product of bad personal choices & therefore an important incentive to change behaviour. Universal healthcare in this view is counter productive. On the other hand if we see health as a social or community good, we’ll likely view healthcare as a system for improving the health of the community even if it comes at the expense of individual freedom of choice. So we feel OK to impose restrictions on certain behaviours like smoking or drinking. click So what would prompt us to change our internal narrative? * Technology is not enough. More information is not enough. * We need to be confronted by someone whom we trust but who also holds a different or opposed view that we accept as valid. * The change evolves thru a process of meaning making, one that includes access to new information but also trusted conversations. The challenge here is that that most people do not typically interact with people who hold a different worldview click
  • There is a parable that is found in many of the world’s great traditions. It was even the subject of a popular19th century poem by American poet John Godfrey Saxe . The lesson of this parable concerns knowledge obtained from incomplete information. One feels the elephant’s tail & believes it is a pig . Another feels the leg & believes it is a tree The third feels the elephant’s side and believes it’s a wall . The fourth feels the trunk & believes it’s a snake . The fifth feels the elephant’s ear and thinks it’s a fan . The last feels the tusk and believes it’s a spear . Which of the blind men is correct? The obvious answer is of course none, but that answer is only possible from our perspective of seeing the whole picture. From the perspectives of the blind men whose senses are providing them with incomplete information, they are all correct. If under this condition of incomplete information they are all partially right, then what picture is possible if we consider all the different interpretations together? A better question than “who’s right?” is, “what can be a pig, a tree, a wall, a fan, a spear and a snake all at the same time?” What reality is it that allows all six blind men to be correct? Of course the only way an accurate picture can emerge is through their conversation with one another and their sense of the validity of each other’s claims. One can imagine them exchanging stories until eventually the notion of an elephant comes out. This parable, in fact, depicts a perennial challenge that exists with many complex health and social issues under conditions of incomplete information. The only reasonable approach is to foster a dialogue with different perspectives, where not only information is shared but also the information’s validity is tested as well as the reliability of each contributor.
  • Decision making under conditions of uncertainty and incomplete information is frequently about the learning biases that come into play in the course of making decisions When we seek solutions we speak to experts or advisors independently. The information we thus receive tends to be is fragmented & incomplete. We need a process that forces questions to be asked not just answers to be given. We need a process to generate a comprehensive picture. We must also contend with social biases that weigh the opinions of well known and powerful people more than a lesser known persons. It must be true because the ‘boss’ said it, or the that movie star supported it. This kind of subjective factor regularly influences what information we deem as important and what information we choose to ignore. Rarely is information entirely objective. So how do we ensure we’re not led astray by one subjective perspective? We are constantly subjected to cognitive biases. These are natural survival mechanisms that have evolved in us as human beings. But we need to learn how to adjust for them in our treatment of new information. Risk evaluation bias: Tendency to evaluate risk differently depending on how outcomes are framed.  Research shows that when considering losses, we prefer to gamble; when considering gains, we prefer the sure thing. Confirmation bias: Tendency to seek only confirming evidence when evaluating our world view, and fail to seek disconfirming evidence. We even discount evidence that does not fit our view. Neglect of probability – Often our most glaring failure is our tendency to completely disregard the notion probability when making a decision under uncertainty. Einstein once said that “God doesn’t play dice”. As it turns out he does.
  • Availability bias: Events that come to mind easily seem more common, more probable and more important than events that are less available in memory eg a few well publicized crimes can give impression that crime is more probable when data shows crime rate is steadily declining in recent years Framing bias: Drawing different conclusions from the same information, depending on how that information is presented. Eg. Does the information support greater independence and ‘choice’ in Canadian families or does it help to ‘nurture’ healthier Canadian families? Anchoring and adjustment bias: when making decisions we have a tendency to rely too heavily on one trait or piece of information & not consider other potential sources of important data. Eg. Health care debates focus on inputs like budgets and wait times. They tend to ignore the fact that health care system is an insurance system not a system for creating health. Creating health would put greater weight on individual responsibility for their own health outcomes. Hindsight bias: Knowledge that an event has occurred makes us overestimate the likelihood that we (or others) could have predicted it ahead of time. Eg. Predicting the current chaos in Egypt. The Israeli intelligence chief was called on the carpet last week for predicting that Egypt would not be caught up in popular uprising like Tunisia. Egypt hadn’t followed that pattern before so why would the intelligence chief believe it would this time. There is a fair degree of randomness at play. Bandwagon bias : the tendency to believe things because many other people believe them. Eg. We have to resist the temptation towards privatization in healthcare. Many people believe this. Private health care delivery has always been an integral part of Canadian healthcare. Or vaccinations are unsafe because many people so. They read it on the Internet. Contrast bias: is the tendency to reduce the weight of data from the past compared with a more recently observed but contrasting data Eg. In the good ole days we seemed to be healthier, subject to fewer diseases, ate healthy food and got sick less. This bias under-appreciates the level of sickness, disease and length of life from 70-100 years ago. These are natural learning biases we’ve developed as survival mechanisms. However, they can be serious impediments to understanding our environment and in the generation of innovative solutions. They can in fact make us blind to new data because we don’t know how to make sense of it. So we ignore it or minimize it. These biases are unlikely to be overcome in isolation. They need contrasting opinions but they also require a sufficient level of trust so that participants will be open to considering different opinion and be able to learn from one another.
  • Click The dynamics of changing community’s assumptions & organizing metaphors is essentially a cyclical process that shows how click local innovation click can influence sector practice, click knowledge frameworks and ultimately click become integrated into a changed social paradigm. The new paradigm then influences the practices of individuals and single organizations click by offering new narratives about how the world works. This process uses more than an exchange of data, it makes ample use of stories, cases, metaphors, research, teaching, and the creation of new assumptions and cultural artifacts as tools for helping the community integrate new knowledge into its existing cultural context. The most natural points of intervention are those that bring together individuals and sector networks; click sector organizations and knowledge institutions click ; and between knowledge institutions and cultural / social organizations click click
  • click On the other hand, the dissemination of community metaphors and paradigms can generate local innovation. The notion of social determinants of health (SDH) leads to click the consideration of which SDH are important, how do they work, under what conditions. This understanding leads to click sector organizations trying to apply that understanding and introducing it to local organizations click who try and assimilate it For example, Pathways to Education, is an initiative developed by the Regents Park Community Health Centre in Toronto to deal with poor community health outcomes. Knowing that low incomes levels are correlated to poor health the CHC asked, why are incomes low? They found that HS graduation levels were typically around 40%. So the CHC decided to tackle poor neighbourhood health outcomes by improving the secondary school graduation rate. Their experience has been well documented, confirming the SDH paradigm click but also encouraging similar initiatives across the country & attracting support from government ministries. Click Again t he most natural points of intervention click are those that bring different groups together click
  • If one wanted to encourage the generation of innovation and its dissemination across communities then, click these cyclical processes suggests 3 critical areas where information, knowledge and practice can be passed along from one group to another in the course of becoming part of the cultural narrative od a community: 1. Where individuals interact with networks and associations CoP’s, social media, networking events, classrooms & cafeterias & other informal settings 2. Where networks and associations interact with knowledge institutions Conferences, workshops, R&D partnerships, educational programs, professional development 3. Where knowledge institutions interact with culture Media, political discourse, film and fine art, literature The interventions involved here would likely include: convening these groups, facilitating conversations among different stakeholders; and capturing the output from these discourses (electronically or otherwise) and making them available over the Net. Click While each intervention strategy will likely differ, it should be directed towards making participants co-creators & learning partners
  • In creating these learning partnerships I suggest a need to address 5 fundamental questions: 1. What are the various needs, resources, and abilities of the people who will be directly served by the system? Identify what’s shared in common? 2. What are the needs of the information providers ? Why will they supply it? What’s in it for them? This important to ensure their continued contributions 3. How will the system be used? Will it be for information collection & storage, information dissemination, collective learning, activity coordination, performance assessment or knowledge generation? Each of these have a different level of partnership complexity & therefore requires different degrees of partner awareness 4. How can the system be shaped by users (suppliers & consumers) to encourage ownership & relevance? If either are absent you’ll likely create your own elephant, a white one. NRCan used the “no budget” approach to encourage employees to take ownership & build it themselves 5. How can the system go beyond data sharing and encourage reframing exercises among users? When Caterpillar Inc. created its Knowledge Network it had 3000 CoPs among its 70,000 employees globally. Its best return came from those CoPs that intentionally brought in outsiders with very different perspectives on the issues at hand. The result - 700% ROI for the CoPs that involved outsiders vs 200% ROI for CoPs only using internal employees click
  • In addition to the 5 development questions, consider 2 fundamental data types. Subject-matter information including primary data, research papers, news, events, policy documents, and other related content such as images, video clips, web links, etc.. Collaboration-support information including partnership specific information such as records of previous discussions; MOUs and agreements (especially contribution agreements); joint business plans; contributor rules and behaviour policies; privacy policies; decision making rules; schedules for joint work; joint reports and media announcements; overhead data, such as session transcripts and minutes (audio, video & print); as well as contact information for the various partners. This latter class of information is frequently omitted, causing participants to become unsure of the bargains that were struck and creating space for ‘free-riding’ or the perception of ‘free-riding’.
  • 1. Use active and engaging practices to involve user ownership and contribution Passive dissemination of knowledge is ineffective. 2. Provide regular interaction between stakeholders Help ensure that hierarchies of knowledge can be agreed to, that knowledge and practice evolve together, that research undertaken is aligned with practice needs, and that research can be easily translated into practice afterwards; 3. Establish the principles by which partners agree to work together Do this early so that they can begin aligning their activities. 4. Construct different strategies for formal and tacit knowledge . Formal knowledge can use weaker social links like social media, wikis, and other internet-based forums Tacit knowledge requires more face-to-face contact to transfer effectively and is more resource/time consuming but it should be considered as part of a regular schedule. Forums that mix tacit and formal knowledge should also be undertaken periodically. 5. Use effective ‘champions’ to facilitate partner trust, exchange collaborative support information, and coordinate joint action as well as explore new opportunities for exchange; 6. Keep participation in the CoP open, inclusive, and responsive An open invitation to practitioners & researchers adds value in terms of ongoing learning, innovation and good practice that is current, reliable and transferable;
  • 7. Website design, content collection and dissemination should generate a sense of belonging: - Partners should see themselves in the website , and have their contributions identifiable as contributions towards the shared community goal; - The design should reflect the type of feedback that is beneficial to local partners and their level organizational capacity; & not try to be all things to all people - The website content should be organized to help share partner contexts. The more contextual information is shared , the easier it will be for partners to adjust to each other’s actions; - Mechanisms for measuring progress should be identified at the outset and built-into the development processes of the website (eg. www.couchsurfing.org); - Design the KE system with support for ongoing learning in mind, meaning don’t over-engineer everything. Keep it modular and have few hardwired functionalities; 8. Pay attention to partner evolution As the CoP becomes engaged in working together, more social negotiation will be required, mostly through non-technical channels. Be sure & budget time & resources for this; 9. Create space for partner awareness Collaborative systems involve contingent cooperation among partners who inevitably adopt attitudes of “trust-and-verify”. This necessitates the creation of space for ongoing feedback, dialogue and social learning; and 10. Provide written or oral summaries of research for practitioners Researchers should engage practitioners early to align their research with existing practices and workflow patterns to ensure the future incorporation of their results into work of practitioners; Lastly, the fastest way to discourage input is not to recognize people’s contributions. Thank them. Profile their contributions. Try and involve them in as many ways as possible.
  • Transcript

    • 1. Taking the Lead: Applying New Strategies for Knowledge Exchange & Uptake PHAC ISHLCD Workshop February 8-9, 2011 Metaphors, Biases & Learning Partnerships 10 Ways to Promote Effective Knowledge Mobilization Christopher Wilson Research Fellow
    • 2. Knowledge Exchange <ul><li>Parts & the Whole </li></ul><ul><ul><li>We do not build new knowledge in an additive way from single actions & data-bits, but by forming ideas about the overall patterns embedded in events & data and then using these patterns to understand more clearly the events & data that gave rise to them </li></ul></ul><ul><li>Metaphors & Re-framing </li></ul><ul><ul><li>The mobilization of knowledge sufficient to alter organizational behaviours and sector practices rests on conversational processes involving many stakeholders that simultaneously re-shape practice norms together with cultural values and assumptions. </li></ul></ul>
    • 3. Towards a Knowledge Network Ca Di Ob CV L MH
    • 4. Metaphors, assumptions & cultural paradigms <ul><li>“ Science works mainly by metaphor ” - Brian Arthur, Santa Fe Institute, 1992 </li></ul><ul><li>“ W e [do] not perceive a signal from the outside world unless it is relevant to an option for the future that we have already worked out in our imaginations ” - Arie de Geus, formerly Royal Dutch Shell, 2002 </li></ul><ul><li>Resolving health & social policies has “ more to do with the ways in which we frame the purposes to be achieved than with the selection of optimal means for achieving them ” - Donald Schön, 1993 </li></ul><ul><li>The framing of problems & solutions depends upon the metaphors and stories in good currency among problem solvers. </li></ul><ul><ul><li>Can the KE move from warehousing data to exercises of re-framing & the creation of meta-frames? </li></ul></ul>
    • 5. To Change the Possibility, Change the Metaphor <ul><li>Research shows that by changing the frame we use, we inevitably change the collective decisions and actions that follow from it </li></ul><ul><li>What would prompt us to change our internal narrative? </li></ul><ul><li>Change evolves thru a process of meaning making </li></ul>
    • 6. Blind Men & the Elephant
    • 7. Decision Making Under Uncertainty <ul><li>Good decisions require asking the right questions, not just giving plausible answers. </li></ul><ul><ul><li>Just like the blind men the ‘experts’ need to learn together </li></ul></ul><ul><li>Social biases (authority bias) such as status and power differences can prevent us from considering and evaluating the quality of the data. </li></ul><ul><ul><li>We are not solely objective in our treatment of data </li></ul></ul><ul><li>Individual cognitive biases can prevent us from seeking or recognizing critical data. </li></ul><ul><ul><li>Risk evaluation bias </li></ul></ul><ul><ul><li>Confirmation bias </li></ul></ul><ul><ul><li>Neglect of probability </li></ul></ul>
    • 8. Other Learning Biases <ul><li>Availability bias </li></ul><ul><li>Framing bias </li></ul><ul><li>Anchoring and adjustment bias </li></ul><ul><li>Hindsight bias </li></ul><ul><li>Bandwagon bias </li></ul><ul><li>Contrast bias </li></ul>
    • 9. Knowledge Exchange, Innovation & Cultural Adaptation Old View Nasty mould destroys bacterial cultures Innovation Mould kills bacteria Single individual / Organization Changed Paradigm Bacterial infections easily dealt with by antibiotics Culture / Collective Understanding Dialogue Medical therapies, treatment regimens and contra-indications for antibiotics Private / Public Knowledge Institutions Affirmation Mould-derived antibiotics effective in killing many forms of bacteria in many situations Multiple individuals / Sector Organizations
    • 10. Knowledge Exchange & Dissemination of Cultural Norms Enrich Confirms & extends cultural assumption Assimilation Eg. Pathways to Education Single individual / Organization Existing Paradigm Social determinants of health Culture / Collective Understanding Contextualize What determinants, how do they affect health outcomes, when under what conditions? Private / Public Knowledge Institutions Diffusion SDH adopted by community heath practitioners as means of transforming health outcomes Multiple individuals / Organizations
    • 11. 3 Targets for KE Intervention <ul><li>Where individuals interact with networks and associations </li></ul><ul><ul><li>CoP’s, social media, networking events, classrooms & cafeterias & other informal settings </li></ul></ul><ul><li>Where networks and associations interact with knowledge institutions </li></ul><ul><ul><li>Conferences, workshops, R&D, educational programs, professional development </li></ul></ul><ul><li>Where knowledge institutions interact with culture </li></ul><ul><ul><li>Media, political discourse, film and fine art, literature </li></ul></ul><ul><li>Each intervention strategy will differ but be directed towards making co-creators & learning partners </li></ul>
    • 12. 5 Questions for Online KE <ul><li>What are the various needs, resources, and abilities of the people who will be directly served by the system? </li></ul><ul><li>What are the needs of the information providers ? </li></ul><ul><li>How will the system be used? </li></ul><ul><li>How can the system be shaped by users (suppliers & consumers) to encourage ownership & relevance? </li></ul><ul><li>How can the system encourage reframing exercises among users? </li></ul>
    • 13. Two Fundamental Data Types <ul><li>Subject-matter information </li></ul><ul><ul><li>including primary data, research papers, news, events, policy documents, and other related content such as images, video clips, web links, etc.. </li></ul></ul><ul><li>Collaboration-support information </li></ul><ul><ul><li>including partnership specific information such as records of previous discussions; MOUs and agreements (especially contribution agreements); joint business plans; contributor rules and behaviour policies; privacy policies; decision making rules; schedules for joint work; joint reports and media announcements; overhead data, such as session transcripts and minutes (audio, video & print); as well as contact information for the various partners. </li></ul></ul>
    • 14. 10 Ways to Promote Effective Knowledge Mobilization <ul><li>Use active and engaging practices to promote user ownership and contribution </li></ul><ul><li>Provide regular interaction between stakeholders </li></ul><ul><li>Establish the principles by which partners agree to work together </li></ul><ul><li>Construct different strategies for formal and tacit knowledge. </li></ul><ul><li>Use effective ‘champions’ </li></ul><ul><li>Keep participation in the CoP open, inclusive, and responsive </li></ul>
    • 15. 10 Ways to Promote Effective Knowledge Mobilization <ul><li>Website design, content collection and dissemination should generate a sense of belonging: </li></ul><ul><ul><li>Mechanisms for measuring progress should be identified at the outset and built-into the development processes of the website (eg. www.couchsurfing.org ); </li></ul></ul><ul><ul><li>Design the system with support for ongoing learning in mind; </li></ul></ul><ul><li>Pay attention to partner evolution </li></ul><ul><li>Create space for partner awareness </li></ul><ul><li>Provide written or oral summaries of research for practitioners </li></ul>
    • 16. Thank you <ul><li>Christopher Wilson </li></ul><ul><li>Research Fellow </li></ul><ul><li>Centre on Governance </li></ul><ul><li>University of Ottawa </li></ul><ul><li>[email_address] </li></ul><ul><li>Tel: 613-355-6505 </li></ul>
    • 17. Additional Resources: Computer Supported Cooperative Work <ul><li>The focus of CSCW is twofold – </li></ul><ul><ul><li>practices that allow partner coordination, and </li></ul></ul><ul><ul><li>the inter-organizational actions that permit collaborative learning to be absorbed by individual organizations so that joint work can be accomplished </li></ul></ul><ul><li>Computer Supported Cooperative Work : Concepts & Tools, Tom Gross </li></ul><ul><li>Journal of Computer Supported Cooperative Work (CSCW) – special healthcare edition </li></ul><ul><li>Building Sustainable Community Information Systems </li></ul>

    ×