Listening to the Crowd: verification of Social Media Content


Published on

This presentation was given at Tech@State in Washington DC in 2013. The presentation covers the basics of how to verify information gathered via social media.

Published in: Social Media, Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Listening to the Crowd: verification of Social Media Content

  1. 1. Listening to the Crowd Anahi Ayala Iacucci – Media Innovation Advisor
  2. 2. Crowdsourcing information for elections How to verify information collected online during elections
  4. 4. Verification and falsification The goal of verification of crowdsourced information is to assess whether the information reported is indeed referring to an actual fact that happened exactly in the same way it is reported. For this reason the verification process needs to happen as a multi-level approach, where all components are analyzed separately and then merged.
  5. 5. Types of Verification and Falsification of crowdsourced information Falsifying/Verifying the context Falsifying/Verifying the content Falsifying/Verifying the source
  6. 6. What is the content? WHO HOW who are the parties involved. Who are the victims and who are the perpetrators. WHAT what actually really happened, wha t violation has been made the context and the dynamics of the actual fact Content 5W1H WHERE WHY why did the fact happened WHEN when the fact reported happen the location of the fact reported
  7. 7. WHO HOW how did the fact happen? What is the connection in between the different components of the event? who are the parties involved? Are really the perpetrators the once mentioned in the report? Who are the victims? Verifying the content WHY The first step is to verify the content of the information, looking at the verification of the 5W1H is there any information on why the fact happen or about the intention of the perpetrators? WHAT what actually did really happen? Can we re-built the entire dynamic of the event? WHERE where did the fact happen? Is there any reference to the actual place? Addresses? Landmarks mentioned? WHEN when did the violation happen? Is the time of reporting close or not to the time of the actual fact reported?
  8. 8. Primary sources the source is the eyewitness of the fact Types of information sources Tertiary sources Secondary sources Compilation based upon primary and secondary sources the source is reporting something based upon evidence from primary sources
  9. 9. Falsifying the source The falsification of the source is often used to undermine the reliability or reputation of a source or to gain trust from the audience Falsifying to gain trust This is often done to add credibility to the information reported by attributing it to a source that will be more trusted than the reporter himself. For instance one common example of this is when secondary sources present themselves as primary sources Falsifying to undermine trust On the contrary this type of falsification is done to attribute to a source a certain information in order to make them look less credible, or even personally involved in the events
  10. 10. Falsifying the context The verification of the context is one of the most difficult things to do because certain information may be true in the content, and the source may be a reliable one, but the context may give to the same information a different angle and therefore a completely different meaning. See this picture as an example:
  12. 12. Social Media ID Direct engagement Media authentication Timing ID of the Content on the Social media Online verification of the source Location Influence ID of the Network ID of the trusting network
  13. 13. The Social media ID When getting an information from a social media source the first thing that we need to look at is the bio of the source (account) as provided in its home page. In this first step we will look only the Identity of the person to try to re-create his/her ID based on his/her social media bio. We can look at several different social medias: Twitter, Facebook, You Tube, Flickr, Linkedin, etc.
  14. 14. Bio on Social Network Is there a name and surname? Is there a link to a blog or a website? Is there a link to the Facebook/Twitter/LinkedIn/You Tube/Flickr Page? Are there pictures? Videos? Lists of previous jobs or schools attended? Look for the name/surname on Google Check the sites and see what content they have and what bios they have Check the pages and the bios there Check all of those and see what they can tell you about this person Evaluate what do you find trying to understand as much as possible about this person identity: does he/she seems to be linked to any political party? Does he/she work for a specific company and if yes what is this company doing exactly? Does he/she have a picture, and if yes, does the picture tells you something about his/her political/religious affiliation? Also make sure that all information in blogs/websites/Twitter bio and so on report the same information: any discrepancies can be a sing of an identity falsification.
  15. 15. ID of the content on social media The second step of the online verification of the source is to look at the content of the source as provided in its home page. After having checked his/her Bio from the previous section, now we will look more closely to his content on social media, what he is talking about, what his ideas may be regarding certain issues and so on. This process is done to add more information to the bio of this person and to try to get a broader picture of what this person is interested in. In this section the methodology is the same for all social media previously considered.
  16. 16. Twitter Website/Blog Linkdin page The first step here is to make a list of his/her social media, or online presence. An example can be: Flickr account Facebook page You tube channel
  17. 17. In this process make sure that you take a look at all possible type of content posted on social media: videos, photos, comments to other people content, articles, re-tweets and forums. Most of us have themes running through our interests, but a tweet/messages history that is completely consumed with one or two issues without the occasional personal reference, humor or reference to other topics is suspect. Behind every social media account should be a personality. Is the content relevant to the issue reported? Does this person seems to be committed to a certain cause or particularly interested in a certain topic? Is this a oneissue tweeter? The second step is to closely look at the content. Does he has strong opinions related to certain topics? And if yes, what his opinions are? What other topics is he/she posting about? Does the content seem reliable? Is there content that does not seems to have been supported by other sources?
  18. 18. ID of the Social Network Who are his/her friends? Do they share his/her views on specific topics or not? In this third phase we are looking at the social network that this person has. In this context we will look at the people he/she is interacting with and the friends he/she has on social networks Do they seem to be related or linked to any particular group/party or group? Do they share the same type of content?
  19. 19. ID of the trusting network This fourth step is related to the people this person trust and that he seems to be influenced by. What we are trying to see here is how to dissect the social network and look more closely to the network of people that this person trusts.
  20. 20. Who are the people that this person is retweeting most often? Who are the people that he/she is interacting with most often? Twitter/Facebook What are these people tweeting about and what are the topics he interacts with them about? Who is he/she following?
  21. 21. Who are the people that have given recommendatio ns to this person? Who are the people that he has worked for and that he knows? Linkdin/Blog Who are the people in his network?
  22. 22. Influence In this phase of the verification process we will be looking at the people that this person influence and that trust him as a source. This information will give us a idea of what is the influence that this person has on his/her social media network.
  23. 23. Who are the people that are re-tweeting this person most often? Who are the people that are interacting with this person most often? Twitter What are these people tweeting about and what are the topics they interacts with him/her about? Who are the people following him/her?
  24. 24. Location In this section of the verification phase we want to verify the location of this person in the real world. The reason why we do this is to be able to define if this person is reporting as a primary source, a secondary source or a tertiary source. The actual location of a person in the virtual world may be really tricky so we need to proceed by steps. There are two types of location services that are associated with social media: - Manual location settings - Smart phone location settings
  25. 25. Manual Location settings Some social media have the location already set up in their settings, if the user has allowed it. For example, Twitter has this option in the profile page: This function though is a manual function meaning that it is the user that manually set the place where he/she is based and where he/she is from. Most of the time this location is the home-town of the person, and not necessarily where he/she is tweeting from.
  26. 26. The same is valid for Facebook location settings, that you can find in the About page of the person:
  27. 27. The Smart Phone Location settings When people are using their smart phones to use Social Media, they can unable the automatic location of their messages, both on Twitter and Facebook. In this case the location is the one of the mobile phone, so it does correspond to the actual location of the person at the moment when the message was sent. Twitter Facebook
  28. 28. Triangulation of the location 1. Check the location of the person on the manual settings (if available) 2. Check the location on the smart phone settings (if available) 3. Cross-reference locations from section 1 and 2 (if available) 4. Cross reference the location mentioned in the message you are verifying with the location mentioned in other messages from the same person Is the location the same of the location mentioned in the message you are verifying? Is the location the same of the location mentioned in the message you are verifying? Are the locations the same? Are those location the same or in the same area? NO: the person may be on a trip or may have moved from the original place NO: this person is probably a secondary or tertiary source NO: the person may have been moving from the original location NO: if the locations are pretty far, this person is probably a secondary or tertiary source YES: pass to the verification processes2,3,4 YES: most likely this person is a primary source. Pass to step 3 and 4 YES: The person is probably a primary source YES: This person is most likely a primary source
  29. 29. Language/Location cross-reference This is a second step to be done when looking for the verification of the location of your source. In this phase you can look at verifying whether the vocabulary, slang, accents are correct for the location that a source might claim to be reporting from. For instance, if the reports are coming from a very rural area and the language is a perfect English, you may want to further verify the source. The same if a certain source is always reporting in a certain language and then suddenly you start receiving reports in another language (this may indicate that those reports are copied or taken from someone else).
  30. 30. Timing The time is an important characteristic of the information you are looking at. This will also tell you something about the possible location of the person sending the message out. In almost all social medias and blog/websites there is a time stamp of the publication, but as in the location, this stamp can also be misleading with regard to the actual event reported. For this reason we need to proceed with the triangulation of this information to get closer to an estimate of the real time of the event and therefore to the reliability of the source.
  31. 31. Check time stamp Is the time close to the time of the event? YES NO The source may be a primary source The source may still be a primary source but just had no access to communication means The source may be not a primary source but someone very much following the event closely The source may be a secondary or tertiary source
  32. 32. Time of account creation This part of the verification process is intended to look at the existence of the account before the time of the reporting. The idea is to find out if the Twitter/Facebook handle is a new account, with only a few tweets. If so, this makes authentication more difficult. Arasmus notes that "the more recent, the less reliable and the more likely it is to be an account intended to spread disinformation." In general, the longer the Twitter/Facebook handle has been around and the more Tweets/posts and people linked to this handle, the better. This gives a digital trace, a history of prior evidence that can be scrutinized for evidence of political bias, misinformation, belonging to a certain group or another and so on.
  33. 33. Media authentication Media authentication is the trace of this source in Media reports: the source in fact may be quoted by trusted/bias media outlines whether this be in the mainstream/local media or social media space. In this case the evaluation of the reliability of this media as a source, can tell us something about the person we are scrutinizing. For instance a media that is bias towards the government will use a source that is also bias towards the government, while an independent media will most likely rely on independent sources. Google or any other research engine will be useful to find out if the source you are looking for is quoted in any online medias or social network.
  34. 34. Direct engagement This is the most difficult part of the verification process and it is suggested to use it only after a very careful considerations of the risks involved in a direct contact on line with the source. For example on Twitter, you can tweet the source back and ask them to identify the source of what they are reporting (suggested to use DM for this). NPR's Andy Carvin has employed this technique particularly well. This technique will force the source of the message to either give you more information to identify themselves or to find additional sources to validate his/her information
  35. 35. Social authentication Crowdsourced verification of the source Triangulation
  36. 36. Social Authentication The Social authentication refers to the use of social networks to identify or validate the source of the report. If you're still unsure about the source's reliability, you can use: - Your own social network (Twitter, Facebook, LinkedIn) -to find out if anyone in your network know about the source's reliability - The entire social network to find out the reliability of the source, for example by tweeting something like “ Does anyone know about this person/account?”
  37. 37. Triangulation We have seen already how triangulation of information works. The main point is to put together all the information that we are able to gather by looking at the previous information, and then put them together trying to find discrepancies or compatibilities in between them. The triangulation of information needs to be based on 2 very important assumptions: - You will never have 100% certainty of the reliability of the sources unless you know them personally - Each information can tell you everything and the contrary of everything. - All the technics you are using to verify information, may also be used by someone else to falsify your information
  38. 38. For this reason there are several things that you need to keep in mind when applying all the methodologies described here: 1. Ask yourself all the possible questions, and never give anything for granted 2. As Arasmus notes, "remain skeptical about the reports that you receive. Look for multiple reports from different unconnected sources." The more independent witnesses you can get information from, the better. 3. If after doing all the necessary verifications of the report, you find out that the user reporting an event is not necessarily the original source, can the original source be identified and authenticated? In particular, if the original source is found, you still need to go through the same steps highlighted here. 4. Run multiple verifications for the same user and, even better, ask others to run the same verification processes on the same account. Personal beliefs and state of mind can change how we interpret the same information. 5. In general, keep in mind that the less you find about someone the less reliable you can consider that source.
  40. 40. Online verification of the content Once the source of an information has been authenticated, we need to verify if the content is actually true. This process is very time consuming because the description of an actual fact can be presented in four different ways: - It can be accurate in all aspects - It can be not accurate but still referring to an actual fact - It can be accurate in all factual aspects but misleading in terms of the context - It can be completely false To verify the content of an information, we need to look at the so called 5W and 1H.
  41. 41. Falsification of the Content
  42. 42. Language Content ID on Social Network Time and place Online verification of the content Triangulation Pictures/Video Follow up with source
  43. 43. WHO HOW who are the parties involved. Who are the victims and who are the perpetrators. WHAT the context and the dynamics of the actual fact what actually really happened Content 5W1H WHERE WHY the location of the fact reported why did the fact happened WHEN when the fact reported happen
  44. 44. The verification of the content starts with the verification of the different components of the information. For each of the 5W1H you will need to look at that specific information and ask yourself several questions to make sure that each of those correspond to the information reported. Some of those questions will not be answered directly in the report, but will need from your part an online search for other sources reporting the same event. See the Triangulation slides for more on this. WHO HOW How did the fact happen? Is it possible to make connections between the different components of the event? WHY is there any information on why the fact happen or about the intention of the perpetrators? who are the parties involved? Who are the perpetrators? Who are victims? The first step is to verify the content of the information, looking at the 5W1H WHAT what did actually happen? Can we re-built the entire dynamic of the event with all the details of the event? WHERE where did the fact happen? Is there any reference to the actual place? Addresses? Landmarks mentioned? WHEN when the violation happen? Is the time of reporting close to the time of the actual fact reported?
  45. 45. Language Language is a very important factor of the information reported that you may want to look at. Some of the particular things related to the use of language that may give you some indication about the veracity of the information reported are: •Use of dialectal words •Use of official language like “breaking news”, “urgent”, “confirmed” etc. •The specific vocabulary, slang, accents, which needs to be cross checked, to see if they are correct for the location that a source might claim to be reporting from.
  46. 46. Is the language used in the report neutral or the person is using lots of adjective and qualitative judgments on the information reported? Is the language used in the report technical. For example using a lot of journalistic language or human right language? Is the language used a proper language or a dialect? Are there particular inflections, slang words or grammatical mistakes in the report?
  47. 47. WHO Is there any indication of who are the people in the video? HOW What is the context and the dynamics of the actual fact? WHY Is there anything that can tell you what is the motivation of the people involved in the action? If the report you receive is also paired with a photographic or video “evidence”, you need to look into the photo/video provided to search for any clues about the 5W1H: WHAT What are they doing in the video/picture? Can you assess that with no doubts? WHERE is there any sign of where the video/picture was taken? WHEN Is there any indication of when the video/picture was taken?
  48. 48. Video What are the sounds on the background? Do people mention names of people or places in the dialogue? Is there any indications of the place based on the action taking place? • Is there any sound that you can connect with the place? For example sounds of cars passing by, people talking in the background? • If they do, in which context? Do those places/people means something in the context of the action taking place? • Can you identify buildings, signs, cars, etc., in the background? Are there objects in the video that may give you additional information? • For example, check weaponry against those known for the given country. What is the weather like in the video? • Examine weather reports to confirm that the conditions shown fit with the claimed date and time. Also looks for shadows to determine the possible time of day that a picture was taken. Which language is the dialogue taking place in? • Language used, tone of voice of the people in videos footages can also be a good indication of the place and context.
  49. 49. Pictures Is there an indication of the place where the photo was taken? A sign, a particular place marker? How are the people in the photo dressed? Does this fits with the place where the photo is supposed to be take? Where is the light coming from in the picture? Does it seems artificial? If yes, look for other pictures on line to make sure that that place really looks like that. Google street view for example is a good way to cross-check streets and public places Search on line for the meaning of those dresses: does particular outfit tells you who this person is or who he/she belongs to? Cross-check that the direction of the light is indeed in line with the position of the sun at that time (like from above if midday, etc) Are there objects in the picture that do not fit in the overall scene? Is the picture very clear and professional or does it looks like it was taken with a phone? Look very carefully at the objects on the background, the corners of the image and the floor. There are indeed phones that take very clear pictures, but in general double check if a picture it’s too perfect and too clear.
  50. 50. Follow up with source As explained before, this is the most difficult part of the verification process and it is suggested to do it only after a very careful considerations of the risks involved in a direct contact with the source. This type of verification can happen in two ways: -Online -Offline The best situation is when the verificators already have contacts in the geographic area of interest, so that they can ask them to follow up directly/in-person to confirm the accuracy of the report. Obviously this is not always possible, particularly in conflict zones. In using this technic it is very important to keep in mind that the people verifying the information on the ground may not be able to do so directly, but still they could gather first hand witnesses reports based on their local network.
  51. 51. On Line Contact The verification of the information provided online is basically based on the creation of a dialogue with the sender of the information. If for example someone has reported something on Twitter or Facebook you can simply write back to them to ask for more details about the event reported. In this case it is suggested to do it privately (via Facebook mailing box or Twitter DM). Before engaging in any online conversation with anyone it is suggested also to inquiry as much as possible if this is safe and if the person reporting will not be put in danger by you contacting them.
  52. 52. Offline Contact The idea of contacting personally someone is related to the possibility to have someone on the ground that is trusted and safe to get in contact with. If such a contact exists then this person (or network of persons) can be used to directly get in contact with the reporters to inquiry about the issue reported. Again, in this case very careful security and privacy considerations needs to be done. The use of this strategy to verify information is suggested as the last resource especially because both the trusted person on the ground and the reporter’s lives can be put in extreme danger.
  53. 53. Time and place The time and place of the event normally present the most problematic issues to verify since they are the once that can be affected by the re-reporting of the information by different sources. The idea here is that most of the time we are in front of a “Chinese whispers” type of situation and we need to make sure that those two variables are reported correctly.
  54. 54. Spelling: if the original language of the reporter it is different than the one of the report, you can find that a place may refer to several locations due to spelling issues. One example is Arabic, where the numbers of letter As in the alphabet are several, while in English there is only one. Time and Place Format of dates: the format of dates are different form place to place, so people may translate 09/08/12 as 9 August 2012, or as 08 of September 2012. If people repost a message and decide to change the format of the date, they may not know which one of the two version they are using. Edited Re-Tweets: when you find tweets reporting something try to do as many searched as possible to find out what the original tweet was saying. People tend to edit retweets when they think there is something wrong (like spelling of names) and in this way they may actually change a correct name.
  55. 55. Triangulation of content The triangulation of information is basically the act of putting together all the information you can find about the issue reported and cross-check them against the fact as reported to you. To do this it is suggested to start filtering all possible information sources available: social media, websites, blogs, local and international media, TVs, and so on. In this tasks, make sure you always remain skeptical about the reports that you receive. Look for multiple reports from different unconnected sources, since the more independent witnesses you can get information from, the better and the less critical the need for identity authentication. If after doing all the necessary verification of the report, you find out that the fact is reported by other people but with different characteristics, like time and location for example, are not matching, do not discard immediately the fact as false but try to identify which version of the report is the most accurate. If you cannot find confirmation for any of the versions, do not chose the most likely one, but make sure you report all versions and highlight which difference they have. The verification of the context often can help in the identification of the those partial falsifications. Run multiple verifications for the same user and, even better, ask others to run the same verifications processes on the same account. Personal beliefs and state of mind can change how we interpret the same information. In general, keep in mind that the less you find about something that less reliable you can consider that information, and the more you have to rely on people on the ground to confirm it.
  56. 56. Content ID on Social Network When you are crowdsourcing information about a specific event or place, try to look in the social network about that same event. If you do not find anything related to that event, search for events in the same area or for issues that may be related to that event. In the social networks normally events tend to be repeated and reported in several platforms (Twitter and Facebook for example) so this process will also help you identifying possible editing of the event reported done by different users.
  57. 57. Social authentication The creation of Virtual Trusted Networks (active and passive) Crowdsourced verification of the content Trusted Network on the ground
  58. 58. Social authentication When you are crowdsourcing information about a specific event or place, you should start finding an online social networks related to that specific issue. For example in Twitter, that specific social network will be the one twitting very on a specific hashtag (#). You can use that social network to ask information about how unusual the event reported is, to get a sense of how likely it is to happened in the first place. This online community, for example, can help you translate, triangulate, and track down key information. They can enable you to crowdsourced verification but you must always check and challenge what you are told. Ask people what they think, what do they know about that area, or about that event. You have also to remember that you must NEVER give an indication of what your source is, if it is not public already, and try not to give too many details of the information reported if that information is not public already.
  59. 59. Trusted Network on the ground When verifying information online the use of an offline network can be very helpful. As mentioned before, this network can be your ears and eyes on the ground, but they can also incur in considerable risks by doing so. For this reason those networks need to be very trusted, possibly people that you personally know and it should be open only to other people known and trusted personally. As already said in the module, security issues should be prioritize over everything else.
  60. 60. The creation of Virtual Trusted Networks (active and passive) In several occasion, the creation of a trusted virtual network has been done by several activists around the world to verify crowdsourced information. This network has been created by starting with a group of very trusted people on the ground (personally known and contacted) and then expanded only to people vetted by this same group of people, and only if personally met. To do this there are several tools that have been used, like Skype (better if encrypted since Skype is indeed vulnerable to monitoring by third parties), Google groups, encrypted emails mailing lists, etc.
  61. 61. The creation of Virtual Trusted Networks (active and passive) It is also possible (and suggested) when doing crowdsourcing online to create a list of the sources that provide information online (directly to you or via social networks) that you can keep updated during your project. This list can also be compiled by adding comments to each of the accounts/people listed and in this way starting to create your reliable list of sources and “scoring” them according to your verification. This list will also be very useful for all the processes done before, and especially once you have done the source verification for those account and can attest the reliability of the source.
  63. 63. Context Matters The understanding of the context of an information it is crucial to give a sense to what has really happened. Specifically in the context of crowdsourced information the context become extremely important because we may not know the source and we may have to find out if the information is actually true. On the other side though, even if the fact reported really did happened and the source is reliable, we may be in the situation where the context of the information can give us a completely different idea of what is the overall picture surrounding the event.
  64. 64. Why Context matters? People develop their opinions based on the information available to them, and this is how we evaluate the verification processes of an information. Sometime though if the fact is not presented it in the right context, the same information may be evaluated in very different ways. The idea here is that once we have attested that a fact has happened, we need to make sure that every fact relevant to the story, which is a necessary part of the story, is also reported. In the context of this module we will look at two different ways that you can use contextualization technics to verify information: 1. Find data that can further support or give you an idea of the veracity of the information 2. Find data that can give a meaning to the information reported
  65. 65. Factual context Time and Location On line Verification the context Global Context
  66. 66. Time and Location We have already seen before how time and location can help us identify the veracity of an information, but in this case we will look at them as important factors for the contextualization of the event reported. Sometimes the time and location of an event make a huge different in considering the reliability of the information reported. What we can look at here is if the event reported makes sense considering the time and location reported.
  67. 67. Time Does the time of the event make sense for the event reported? For example: if an attack to a street vendor is reported, does the time of the event reported make sense considering the type of economic activity? Also, does the time gives a specific meaning to the event? Like an attack to a mosque on Friday? Or to a Synagogue on Saturday?
  68. 68. Location Does the location of the even make sense for the type of event reported? For example, does the event take place in a location where it is very unlikely to find the actors involved? Does the location have a specific meaning for the type of event reported? Does the location have access or closure times that do not fit with the time of the event reported?
  69. 69. Factual context The first thing that we need to look at when looking at the context is to try to place the event we are analyzing in a greater picture. The picture in this page is an example. If you think about how you would describe the picture, the context of the event is giving you a completely different information depending on if you look at the right or left part of the picture. The event happening is always the same though. This is what we mean when we say that you have to look at the factual context.
  70. 70. Global context The global context refer to the international context around that event. Sometimes political crisis outside a country can trigger events in a third country. One example can be the sudden explosion of tensions in between two neighboring countries that can push a third one to close its borders or move contingents of the army towards the borders. The global context is also something related to the important facts that are happening in the same country or in the same location of the event reported. The national context of a reported event in fact and when it does happen, can have a relationship with the reasons behind the event, that can help us understand if the event is likely to be true or if it can be triggered by other events happening in the same place or at the same time.
  71. 71. Is there any other event similar to this one that has happened in the same time or area? For example something very similar reported in the same area Or something that could have triggered that event (like a retaliation or an attack) Like if an attack is reported, are there other attack been reported in the same area before or after? For example, massive arrests of activists may be following the dead of a police man during a demonstration Or something that could have influenced the way people described what they have seen (for example something very shocking or inflammatory) In this case people may report event to you exaggerating the event because they are in a very emotional state of mind following a very shocking event
  72. 72. Look at what is going on the region, but also globally in the world Is there any political issues happening that can be related to the country? Like important bilateral meetings? Or peace treaties? Or important speeches being delivered? Are there economical issues/ factors that can have an influence in the country? Sudden rise in unemployment? Sudden rise in price of oil? There has been any events happening in neighboring countries that may have lead to reaction in the place you are considering? For example sudden rise in tension in between two countries? Or an economical agreement reached?
  74. 74. The 9 golden rules 1. Unless you see it and you hear it, it is not 100% verified. 2. After rule I, chose your verification criteria in advance and make them public (if possible) 3. Chose your trusted network and make clear rules for being part of it 4. Doubt and question everything, always 5. Be always ready to change your mind about the reliability of a source 6. Be always ready to change your mind about the veracity of an information 7. You don’t need to verify everything. Choose you battles 8. Always admit when you fail or when you do a mistake 9. What you know is also what your enemies know. Keep this always in mind when verifying information
  75. 75. Keep your reputation intact Since we said that rule number one for the verification of crowdsourced information is that unless you see and hear it, it is not 100% verified, you need to protect your reputation and your credibility every time you use, republish, collect or advertise crowdsourced information. To this, you need to be very transparent about your verification criteria and about what degree of verification you can achieve with the methodologies you are using. This also means that you do not have to decide just in between verified and not verified (like 0-100) but you can also choose to use several degrees of verification in the middle. The main important things is that you make very clear to whoever is taking that information from you what criteria you used and how are you verifying your sources and information. A second very important point, as expressed before, is that you need to always be ready to admit when you did a mistake in your verification process and be ready to change your protocols according to the situation.
  77. 77. Step I: Chose what you need to verify You do not necessarily need to verify everything. Choose what are most important information by looking at the typology and amount of information that you received. Make a considerate evaluation of your capacity. It is better to have few very well verified information, then a lot of quasi verified information.
  78. 78. Step II: Choose your verification criteria Create your criteria for verification in advance and by looking precisely at what means you have to verify information. Ask yourself: 1. When is that you will consider an information verified? 2. When is that you will not consider it verified? 3. What degrees of verification are you willing to have in the middle?
  79. 79. Step III: draft your trusted sources list Choose very carefully who you will consider a trusted source. This does not necessarily need to be mainstream media or known people. In addition to this you need to update your trusted sources list periodically (I would say on a daily bases) to make sure that you add new sources and review the trustfulness of old sources.
  80. 80. Step IV: Draft your verification protocols and stick to it Once you have drafted your verification protocols, make sure you always apply them to all information you process. Do not allow yourself to do exceptions to it, and make sure that all your information are verified following the same procedures. This is very important to assure consistency and to be able to always get back to your verification process to refine where it is not working.
  81. 81. Step V: make it public One of the main important things about your verification protocol is to make it as public as possible. The reason behind this is not to inform people how they can go around it, but to assure that people do know what sort of credibility your information have. You do not have to make the all details public, but people using your information need to know exactly how much they can trust your information or not. The more they are aware of the verification protocols you use, the more you will be able, if necessary to explain eventual mistakes or failure in verifying information, as well as managing expectations about the reliability of the information posted.
  82. 82. EXAMPLES
  83. 83. Blogging the Egyptian Election “BEE” 2010
  85. 85. Step 1: what to verify One of the main problem of verification of crowdsourced information is related to the fact that the overload of information can make it impossible to actually do a meaningful verification of the information received. For this reason, the U-Shahid team’s first step was to define concrete criteria for what types of reports required verification. They didn’t attempt to verify all the content that was available, but only the one that was important or critical. 1. Reports related to an immediate threat or act of violence 2. “Grave electoral fraud” the gravity of which was assessed by the importance of the parties involved (celebrities, known PMs, government officials) or the gravity of the fraud (involving violence, or massive votes exchange) or it’s location (inside the pooling stations)
  86. 86. The report was supported by a video or pictures that clearly confirmed what was been reported (especially location); At least one of the sources of the information was clear and known (i.e., 2 SMSs from unknown sources could not verify each other) The U-Shahid team also created some criteria that all reports need to be checked against to be considered as verified. If one or more of the criteria was met, then the report would have been considered verified Messages coming from social media (Twitter, Facebook and blogs) needed to be confirmed by an SMS, a media report or a direct witness; It was been reported by two or more independent sources (the list of independent sources was created in advance and updated by the team) ;
  87. 87. Step 3: Direct contact with the source By asking to provide a video or picture of the event but only if it was safe to do so. If the report was sent by SMS, that number was called to verify the person’s identity. The witness was asked if they observed the event themselves or if they simply learned about the event from someone else. More specifically, details on the 5w1h are asked. If the event being reported was still unfolding, the witness was asked if anyone else nearby was able confirm the information. If the report came from Twitter itself, the account of the tweeter was reviewed. Simple content analysis of previous tweets and the account holder’s bio was carried out online, in addition to revised the Tweeter’s followers. For reports that had to be verified but did not meet the criteria outlines before, the verification team was engaging directly with the source in 4 different ways By tweeting back to the original Twitter user asking for more information—again using the “who did what, to whom, how and where” format, asking followers to confirm if the information was indeed correct.
  88. 88. Step 4: In person verification The U-Shahid team determines whether one of their election monitors (trained by them) was close to the area referenced in a report that required verification. If a monitor was on site or close enough, that person was asked to verify the report. The second core strategy involves inperson verification via a trusted network of monitors on the ground: If U-Shahid did not had any monitors in the area, the team was activating their NGO partners who may had monitors in the area. If so, those individuals were asked to confirm the validity of the report being investigated. If no one of the two possibilities was possible, UShahid was using a network of trusted journalists in the area to ask them to go to the place to confirm the information for them.
  89. 89. Step 5: online triangulation In addition to this the U-Shahid team was then triangulating the report being investigated with other reports they had already received. If no one of the previous possibilities was available, the team was using the mainstream media and online tools for confirmation. They used web-based research to look for any evidence that is specific to the event that was reported as well as that location. They looked for article, blogs, social media, video or pictures, mainstream media articles and TV shows that could confirm the information reported.
  90. 90. OUTCOMES Ushahid received on election day November 28, 2010 during voting period 1252 reports, documented with 287 videos and 111 images. By the end of the project on Dec 28, 2010 a total of 2700 were uploaded on the platform with an average of reports per day of 27 and a percentage of verified reports of 91.26%.
  91. 91. Electua 2012
  92. 92. THANK YOU Anahi Ayala Iacucci Media Innovation Advisor @anahi_ayala @info_innovation