Uploaded byCloudera, Inc.
Using Hadoop to Drive Down Fraud for Telcos
The document discusses the use of Apache Hadoop for real-time fraud analytics in communication service providers, highlighting the significant financial losses due to various types of fraud, such as premium rate service and roaming fraud. It contrasts traditional legacy systems with a Hadoop-based approach, emphasizing faster detection and improved accuracy in identifying fraud. The presentation outlines the solution benefits provided by Cloudera and Argyle Data, including enhanced fraud discovery and reduced false positives.
More Related Content
Similar to Using Hadoop to Drive Down Fraud for Telcos
More from Cloudera, Inc.
Using Hadoop to Drive Down Fraud for Telcos
- 1. Using Apache HadoopTo Drive Down Fraud for Communication Service Providers Hadoop Based, Real-Time Fraud Analytics Platform for Telcos April 2016
- 2. 2© Cloudera, Inc.All rights reserved. Your Speakers Vijay Raja Solutions Marketing Manager Cloudera Dr. Ian Howells Chief Marketing Officer Argyle Data
- 3. 3© Cloudera, Inc.All rights reserved. Agenda • Telco Fraud Basics • Fraud & Associated Impact on Telcos • Traditional or Legacy Approach to Fraud • Hadoop Based Approach to Fraud • Cloudera – Argyle Data Solution • Demo – Fraud Solution • Q&A
- 4. 4© Cloudera, Inc.All rights reserved. Revenue Threats & Fraud are a Global Problem Profit/Risk Not Geography Matters - The Global Attacker • Web Premium Rate Number • Auto-Forward • High Speed Computer • 100’s of Simultaneous Calls • 220 minutes/minute • Victims • New York - $166,000 weekend • Albany - $200,000 3 Days • 34 Years Worth of Calls • To Gambia, Somalia, Maldives
- 5. 5© Cloudera, Inc.All rights reserved. Premium Rate Service Fraud The Wangiri Attack – “One Ring & Cut” Premium Rate Number/ Line Fraudsters can easily set up a Premium Rate Number in any part of the world Drive unsuspecting users to the Premium rate line High Cost Calls Premium Rate ($1 - $4/min) High Interconnect Fees Missed Calls from masked numbers More Calls = More Revenues
- 6. 6© Cloudera, Inc.All rights reserved. Fraud & Associated Financial Impact for Telcos Mobile and fixed-line carriers lose $38B per year to fraud & revenue threats Premium rate service fraud cost victims $3.8B globally last year Roaming Fraud cost the industry $6.1B globally last year Source- Communications Fraud Control Association (CFCA)
- 7. 7© Cloudera, Inc.All rights reserved. Fraud Types & Methods Premium Rate Service Fraud Roaming Fraud International Revenue Share Fraud Subscription Fraud Domestic Revenue Share Fraud Wangiri Attack SMS Phishing PBX Hacking Identity Theft …… FRAUD TYPE FRAUD METHOD Increasing Complexity of Fraud Types & Nature of Attacks
- 8. 8© Cloudera, Inc.All rights reserved. Legacy Systems Don’t Work Anymore… The Legacy Defense Approach The New Sophisticated Global Attack Threat • Batch File Ingestion • Discover Threats Too Late • Rules Based • Don’t Discover Zero-Day Attack Methods • False Positive Overload • Data Silos • No Crime 360 Signals • Flat World Forensics • Discover Incident not Crime Rings • High Cost Proprietary Architecture • Limited Data due to cost constraints
- 9. 9© Cloudera, Inc.All rights reserved. Legacy Defense Approach vs. Hadoop & Machine Learning The Legacy Defense Approach • Batch File Ingestion • Discover Threats Too Late • Rules Based • Don’t Discover Zero-Day Attack Methods • False Positive Overload • Data Silos • No Crime 360 Signals • Flat world Forensics • Discover Incident not Crime Rings • High Cost Proprietary Architecture • Limited Data due to cost constraints The Hadoop & Machine Learning Approach • Real-Time Packet Ingestion • Discover in Seconds vs. Hours or Days • Real-Time Anomaly Detection • Discover 250% to 350% More Fraud • 20 to 30 Times Less False Positives • Hadoop based Enterprise Data Hub • Discover Crime 360 Signals • Graph based Visual Analytics • Discover Crime Rings • Native Hadoop Architecture • Unlimited Data Storage & Analytics
- 10. 10© Cloudera, Inc.All rights reserved. Polling Question What are your plans for using Hadoop for Fraud Analytics o We already use Hadoop today for Fraud Analytics o We currently use Hadoop, but not for Fraud o We would look to potentially utilizing Hadoop for Fraud o We have no plans for Hadoop o Not Applicable/ Don’t Know
- 11. 11© Cloudera, Inc.All rights reserved. Real-Time Fraud Analytics Solution DPI/File Ingestion – Files, Packets & Database Feature Enrichment Mobile Call Fixed Call VoIP Call Data BSS Systems ISUP, TD.35, TD.57, MAP, SMS, BSSAP ISUP, Diameter SIP, H.323, Diameter Viber, Whatsapp GTP, Diameter Billing, CRM, Others BSS Data Sources
- 12. 12© Cloudera, Inc.All rights reserved. Real-Time Fraud Analytics Solution Security and Administration Ingest Sqoop, Flume Transform MapReduce, Hive, Pig, Spark Analytic Database Impala Search Solr Machine Learning SAS, R, Spark, Mahout NoSQL Database HBase Streaming Spark Streaming Unlimited Storage HDFS, Hbase, Kudu YARN, Cloudera Manager, Cloudera Navigator Cloudera Enterprise Hadoop Platform Process Discover Model Serve DPI/File Ingestion – Files, Packets & Database Feature Enrichment Mobile Call Fixed Call VoIP Call Data BSS Systems ISUP, TD.35, TD.57, MAP, SMS, BSSAP ISUP, Diameter SIP, H.323, Diameter Viber, Whatsapp GTP, Diameter Billing, CRM, Others BSS Data Sources
- 13. 13© Cloudera, Inc.All rights reserved. Real-Time Fraud Analytics Solution Security and Administration Ingest Sqoop, Flume Transform MapReduce, Hive, Pig, Spark Analytic Database Impala Search Solr Machine Learning SAS, R, Spark, Mahout NoSQL Database HBase Streaming Spark Streaming Unlimited Storage HDFS, Hbase, Kudu YARN, Cloudera Manager, Cloudera Navigator Cloudera Enterprise Hadoop Platform Process Discover Model Serve DPI/File Ingestion – Files, Packets & Database Feature Enrichment Mobile Call Fixed Call VoIP Call Data BSS Systems ISUP, TD.35, TD.57, MAP, SMS, BSSAP ISUP, Diameter SIP, H.323, Diameter Viber, Whatsapp GTP, Diameter Billing, CRM, Others BSS Data Sources Fraud Threat Analytics Machine Learning Profit/ SLA Threat Analytics Forensic Analytics (Graph) Argyle Data - Fraud Analytics Platform
- 14. 14© Cloudera, Inc.All rights reserved. Domestic Fraud Wangiri Attack Forensics Crime Ring Discovery Roaming Fraud SLA Revenue Clawback Live Demo - Three Fraud Scenarios - Brazilian Operator
- 15. 15© Cloudera, Inc.All rights reserved. Real-Time Fraud Analytics - Demo
- 16. 16© Cloudera, Inc.All rights reserved. Solution Benefits – Cloudera|Argyle Data • Discover fraud in seconds versus hours or days • 250% to 350% better fraud discovery • 20 to 30 times less false positives • Prevents millions in revenue leakage • Visual patterns that make fraud, profit or SLA threats obvious • Interactive dashboards that discover crime rings and flows otherwise not detectable Wangiri Attack – Visual Graphic
- 17. 17© Cloudera, Inc.All rights reserved. Thank you
Editor's Notes
- #5 Telco Fraud is a $38 Billion industry today – huge market and it is a global problem – It is not a third world problem http://www.nytimes.com/2014/10/20/technology/dial-and-redial-phone-hackers-stealing-billions-.html?_r=0 Hackers had broken into the phone network of the company, Foreman Seeley Fountain Architecture, and routed $166,000 worth of calls from the firm to premium-rate telephone numbers in Gambia, Somalia and the Maldives. It would have taken 34 years for the firm to run up those charges legitimately, according to a complaint it filed with the Federal Communications Commission. The firm, in Norcross, Ga., was the victim of an age-old fraud that has found new life now that most corporate phone lines run over the Internet. http://www.ibtimes.com/inside-story-how-pakistan-took-down-fbis-most-wanted-cybercriminal-1860808 http://www.premiumrateinternational.com/countries.php
- #7 IRSF- International revenue share fraud (IRSF) - involves artificially inflating traffic to a foreign number. IRSF, is usually combined with premium rate service fraud. ISRF is driven by both connection charges and by the number of minutes consumed. Certain countries have particularly high interconnection fees and are a focus for IRSF. These fees are highest in the Caribbean (including Cuba) and many small countries in the Pacific. Two individuals were arrested recently for gaining access to business telephone systems and using the systems to place international telephone calls to premium rate numbers. This cost the victims more than US$50 million.
- #8 Every year, the telco industry loses $38 Billion to fraud and Roaming Fraud takes the biggest hit: about $6 billion globally and almost $2 billion in North America and Western Europe. Criminals are adopting newer techniques to perpetrate fraud quickly and more efficiently than ever before. A cyberfraud gang can set up, go to work and disappear in 24 hours or less, often before an operator even knows the fraud or arbitrage attack is happening. Modern, sophisticated attacks are mutating and continually evolving. ---------------------------------- Roaming Fraud: When roaming, the call data is collected by the visiting network, which can be a nearby operator just as easily as it could be one half way across the world. The call detail record (CDR) for these roaming charges doesn’t arrive to the home network until days sometimes weeks later, leaving a large window of opportunity for fraudulent attacks. Domestic revenue share fraud pertains to the abuse of carrier interconnect agreements and is very similar to international revenue share fraud and premium rate service fraud. In all three scenarios, there is an artificial inflation of traffic to a premium rate phone number. The scheme is fairly simple: A fraudster gets hold of a premium rate service number – a phone number where a portion of the charges goes to the operator and not only the phone carrier like with regular phone numbers – and inflates the traffic to the service to generate more revenue. Roaming fraud is the responsibility of the home network of the subscriber, providing that the visiting network meets its Service Level Agreement (SLA) commitments (typically 4 hours). This time window is critical, because: If the SLA is met the financial responsibility is in the hands of the home network. If the SLA is not met the financial responsibility is in the hands of the visiting network.
- #9 Many carriers are trying to defend themselves with legacy approaches and technologies that don’t work anymore. Legacy systems rely on: • Batch processing – Typically discovers threats 24 – 48 hours after it has occurred – discovers threats too late- Goal for operators should be to reduce the window from 24 hours to discovering fraud within seconds or minutes… • They are Rules based - and discover only old known, patterns of fraud – Cannot detect net new attacks or they overload the analyst with false positives – So the real attack may be 1000 on the list of alerts • Silos of data that sits across multiple systems – Legacy systems are able to look at only a fraction of data and can never really give you a 360 degree view of fraud. In terms of data we are talking about an Average of 35 – 60 Billion CDRs a day & 1.5 Trillion events/ day for a Tier 1 Mobile organization So how do you look for fraud signals in this mountain of data and bring out meaningful threat analytics? This is really where a Hadoop & Machine learning comes into picture. Using Hadoop you can utilize all of your data and you look for fraud signals in more data, more often and in real-time This is where Hadoop & machine Learning really excels in– You Need a Big Data platform to look at this scale of data in real-time and drive Machine learning & real-time analytics on this data. Existing systems simply don’t work anymore. They either: • Fail - Don’t discover new threats or “zero-day” fraud attacks • Overwhelm – Bombard users with false positives • Operate in Batch – Discovering fraud threats too late Scale of Data in Telcos - Average of 35 Billion CDRs a day – for a NA Service Provider 1.5 Trillion events/ day – This is really where a Big Data Platform or an Enterprise Data Hub comes into picture
- #10 You can discover Fraud that you didn't’t know existed earlier You can also discover net new attacks
- #11 At this point I want to open up to a polling question, to try and understand from you – where you are with respect to the IoT journey within your organization. We’ll give it about 20sec so that everyone can answer – I already see a lot of responses coming in ------------------- Very consistent with what we see in the market In line with what we hear from our customers as well.
- #12 This is where Hadoop & machine Learning really excels in– You Need a Big Data platform to look at this scale of data in real-time and drive Machine learning & real-time analytics on this data. Cloudera has teamed up with Argyle Data to deliver a next generation, native-Hadoop, real-time, fraud analytics platform that is tailored for today’s Communication Service Providers. This enables is CSPs to take advantage of all the data they have at their disposal to easily and effectively identify fraud and revenue threats in real-time. Real-Time Data Ingest: we can look at data from a multitude of sources including - mobile call packets, fixed call packets, VoIP and Data packets (including TD.35, CDR, TAP 3, ISUP, BSSAP, MAP, Diameter packets) straight from the switch – and effectively combine these data to uncover fraud. Looking at more data in real-time across multiple sources or silos enables better fraud detection. All of this data is stored in a an Cloudera enterprise data hub, where you can combine, Process, Analyze all this data to detect anomalies. The Argyle Solution sits on top – tightly integrated with Cloudera – extensively utilize Impala and also Sentry for security. Argyle Data Solution- The first is the Fraud analytics application – used for real-time fraud detection, the second is around profit threat or SLA analytics (important because most fraud happens in roaming and there is an SLA between Service providers when a subscriber is roaming) and the third one is Forensic Analytics – this basically enables you look at an attack and see if it is an individual doing the damage or if there is a crime ring involved. The application lets you detect crime rings in a very intuitive and visual fashion and it has the ability to show 1,2,3or 4 degrees of separation (like in LinkedIN) to discover crime rings. Real- Time Machine Learning & Anomaly Detection: As entries are stored, probabilities are generated by comparing the new entries with historic patterns in real-time enabling quicker fraud detection Real- Time Analytics: Machine learning driven threat scoring prioritizes fraud threat attacks, allowing analysts to effectively use their time inspecting real threats. Real-time fraud alerts are sent based on anomalies detected by the machine learning system Real- Time Threat Visualization: Platform combines massive amounts of roaming data, real-time network data, and business data into one simple easy-to-use graph analysis dashboard – makes fraud detection easy & obvious.
- #13 This is where Hadoop & machine Learning really excels in– You Need a Big Data platform to look at this scale of data in real-time and drive Machine learning & real-time analytics on this data. Cloudera has teamed up with Argyle Data to deliver a next generation, native-Hadoop, real-time, fraud analytics platform that is tailored for today’s Communication Service Providers. This enables is CSPs to take advantage of all the data they have at their disposal to easily and effectively identify fraud and revenue threats in real-time. Real-Time Data Ingest: we can look at data from a multitude of sources including - mobile call packets, fixed call packets, VoIP and Data packets (including TD.35, CDR, TAP 3, ISUP, BSSAP, MAP, Diameter packets) straight from the switch – and effectively combine these data to uncover fraud. Looking at more data in real-time across multiple sources or silos enables better fraud detection. All of this data is stored in a an Cloudera enterprise data hub, where you can combine, Process, Analyze all this data to detect anomalies. The Argyle Solution sits on top – tightly integrated with Cloudera – extensively utilize Impala and also Sentry for security. Argyle Data Solution- The first is the Fraud analytics application – used for real-time fraud detection, the second is around profit threat or SLA analytics (important because most fraud happens in roaming and there is an SLA between Service providers when a subscriber is roaming) and the third one is Forensic Analytics – this basically enables you look at an attack and see if it is an individual doing the damage or if there is a crime ring involved. The application lets you detect crime rings in a very intuitive and visual fashion and it has the ability to show 1,2,3or 4 degrees of separation (like in LinkedIN) to discover crime rings. Real- Time Machine Learning & Anomaly Detection: As entries are stored, probabilities are generated by comparing the new entries with historic patterns in real-time enabling quicker fraud detection Real- Time Analytics: Machine learning driven threat scoring prioritizes fraud threat attacks, allowing analysts to effectively use their time inspecting real threats. Real-time fraud alerts are sent based on anomalies detected by the machine learning system Real- Time Threat Visualization: Platform combines massive amounts of roaming data, real-time network data, and business data into one simple easy-to-use graph analysis dashboard – makes fraud detection easy & obvious.
- #14 This is where Hadoop & machine Learning really excels in– You Need a Big Data platform to look at this scale of data in real-time and drive Machine learning & real-time analytics on this data. Cloudera has teamed up with Argyle Data to deliver a next generation, native-Hadoop, real-time, fraud analytics platform that is tailored for today’s Communication Service Providers. This enables is CSPs to take advantage of all the data they have at their disposal to easily and effectively identify fraud and revenue threats in real-time. Real-Time Data Ingest: we can look at data from a multitude of sources including - mobile call packets, fixed call packets, VoIP and Data packets (including TD.35, CDR, TAP 3, ISUP, BSSAP, MAP, Diameter packets) straight from the switch – and effectively combine these data to uncover fraud. Looking at more data in real-time across multiple sources or silos enables better fraud detection. All of this data is stored in a an Cloudera enterprise data hub, where you can combine, Process, Analyze all this data to detect anomalies. The Argyle Solution sits on top – tightly integrated with Cloudera – extensively utilize Impala and also Sentry for security. Argyle Data Solution- The first is the Fraud analytics application – used for real-time fraud detection, the second is around profit threat or SLA analytics (important because most fraud happens in roaming and there is an SLA between Service providers when a subscriber is roaming) and the third one is Forensic Analytics – this basically enables you look at an attack and see if it is an individual doing the damage or if there is a crime ring involved. The application lets you detect crime rings in a very intuitive and visual fashion and it has the ability to show 1,2,3or 4 degrees of separation (like in LinkedIN) to discover crime rings. Real- Time Machine Learning & Anomaly Detection: As entries are stored, probabilities are generated by comparing the new entries with historic patterns in real-time enabling quicker fraud detection Real- Time Analytics: Machine learning driven threat scoring prioritizes fraud threat attacks, allowing analysts to effectively use their time inspecting real threats. Real-time fraud alerts are sent based on anomalies detected by the machine learning system Real- Time Threat Visualization: Platform combines massive amounts of roaming data, real-time network data, and business data into one simple easy-to-use graph analysis dashboard – makes fraud detection easy & obvious.
- #16 A live dashboard showing Roaming Fraud in Colombia