Telcom data mining


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Telcom data mining

  1. 1. 486 Section: ServiceData Mining in the TelecommunicationsIndustryGary M. WeissFordham University, USAINTRODUCTION Telecommunication companies maintain data about the phone calls that traverse their networks in the form ofThe telecommunications industry was one of the first call detail records, which contain descriptive informa-to adopt data mining technology. This is most likely tion for each phone call. In 2001, AT&T long distancebecause telecommunication companies routinely gener- customers generated over 300 million call detail recordsate and store enormous amounts of high-quality data, per day (Cortes & Pregibon, 2001) and, because callhave a very large customer base, and operate in a detail records are kept online for several months, thisrapidly changing and highly competitive environment. meant that billions of call detail records were readilyTelecommunication companies utilize data mining to available for data mining. Call detail data is useful forimprove their marketing efforts, identify fraud, and marketing and fraud detection applications.better manage their telecommunication networks. Telecommunication companies also maintain exten-However, these companies also face a number of data sive customer information, such as billing information,mining challenges due to the enormous size of their as well as information obtained from outside parties,data sets, the sequential and temporal aspects of their such as credit score information. This informationdata, and the need to predict very rare events—such as can be quite useful and often is combined with tele-customer fraud and network failures—in real-time. communication-specific data to improve the results The popularity of data mining in the telecommunica- of data mining. For example, while call detail datations industry can be viewed as an extension of the use can be used to identify suspicious calling patterns, aof expert systems in the telecommunications industry customer’s credit score is often incorporated into the(Liebowitz, 1988). These systems were developed to analysis before determining the likelihood that fraudaddress the complexity associated with maintaining a is actually taking place.huge network infrastructure and the need to maximize Telecommunications companies also generate andnetwork reliability while minimizing labor costs. The store an extensive amount of data related to the opera-problem with these expert systems is that they are ex- tion of their networks. This is because the networkpensive to develop because it is both difficult and time- elements in these large telecommunication networksconsuming to elicit the requisite domain knowledge have some self-diagnostic capabilities that permit themfrom experts. Data mining can be viewed as a means to generate both status and alarm messages. Theseof automatically generating some of this knowledge streams of messages can be mined in order to supportdirectly from the data. network management functions, namely fault isolation and prediction. The telecommunication industry faces a number ofBACKGROUND data mining challenges. According to a Winter Cor- poration survey (2003), the three largest databases allThe data mining applications for any industry depend belong to telecommunication companies, with Franceon two factors: the data that are available and the busi- Telecom, AT&T, and SBC having databases with 29, 26,ness problems facing the industry. This section provides and 25 Terabytes, respectively. Thus, the scalability ofbackground information about the data maintained by data mining methods is a key concern. A second issuetelecommunications companies. The challenges as- is that telecommunication data is often in the form ofsociated with mining telecommunication data are also transactions/events and is not at the proper semanticdescribed in this section. level for data mining. For example, one typically wants to mine call detail data at the customer (i.e., phone-Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
  2. 2. Data Mining in the Telecommunications Industryline) level but the raw data represents individual phone was MCI’s Friends and Family program. This program,calls. Thus it is often necessary to aggregate data to long since retired, began after marketing researchers Dthe appropriate semantic level (Sasisekharan, Seshadri identified many small but well connected subgraphs& Weiss, 1996) before mining the data. An alternative in the graphs of calling activity (Han, Altman, Kumar,is to utilize a data mining method that can operate on Mannila & Pregibon, 2002). By offering reduced ratesthe transactional data directly and extract sequential or to customers in one’s calling circle, this marketing strat-temporal patterns (Klemettinen, Mannila & Toivonen, egy enabled the company to use their own customers1999; Weiss & Hirsh, 1998). as salesmen. This work can be considered an early use Another issue arises because much of the telecom- of social-network analysis and link mining (Getoor &munications data is generated in real-time and many Diehl, 2005). A more recent example uses the interac-telecommunication applications, such as fraud identi- tions between consumers to identify those customersfication and network fault detection, need to operate in likely to adopt new telecommunication services (Hill,real-time. Because of its efforts to address this issue, Provost & Volinsky, 2006). A more traditional approachthe telecommunications industry has been a leader in involves generating customer profiles (i.e., signatures)the research area of mining data streams (Aggarwal, from call detail records and then mining these profiles2007). One way to handle data streams is to maintain a for marketing purposes. This approach has been usedsignature of the data, which is a summary description of to identify whether a phone line is being used for voicethe data that can be updated quickly and incrementally. or fax (Kaplan, Strauss & Szegedy, 1999) and to clas-Cortes and Pregibon (2001) developed signature-based sify a phone line as belonging to a either business ormethods and applied them to data streams of call detail residential customer (Cortes & Pregibon, 1998).records. A final issue with telecommunication data Over the past few years, the emphasis of marketingand the associated applications involves rarity. For applications in the telecommunications industry hasexample, both telecommunication fraud and network shifted from identifying new customers to measuringequipment failures are relatively rare. Predicting and customer value and then taking steps to retain the mostidentifying rare events has been shown to be quite dif- profitable customers. This shift has occurred because itficult for many data mining algorithms (Weiss, 2004) is much more expensive to acquire new telecommunica-and therefore this issue must be handled carefully in tion customers than retain existing ones. Thus it is usefulorder to ensure reasonably good results. to know the total lifetime value of a customer, which is the total net income a company can expect from that customer over time. A variety of data mining methodsMAIN FOCUS are being used to model customer lifetime value for telecommunication customers (Rosset, Neumann, EickNumerous data mining applications have been de- & Vatnik, 2003; Freeman & Melli, 2006).ployed in the telecommunications industry. However, A key component of modeling a telecommunica-most applications fall into one of the following three tion customer’s value is estimating how long theycategories: marketing, fraud detection, and network will remain with their current carrier. This problemfault isolation and prediction. is of interest in its own right since if a company can predict when a customer is likely to leave, it can takeTelecommunications Marketing proactive steps to retain the customer. The process of a customer leaving a company is referred to as churn,Telecommunication companies maintain an enormous and churn analysis involves building a model ofamount of information about their customers and, due customer attrition. Customer churn is a huge issue into an extremely competitive environment, have great the telecommunication industry where, until recently,motivation for exploiting this information. For these telecommunication companies routinely offered largereasons the telecommunications industry has been a cash incentives for customers to switch carriers. Nu-leader in the use of data mining to identify customers, merous systems and methods have been developed toretain customers, and maximize the profit obtained from predict customer churn (Wei & Chin, 2002; Mozer,each customer. Perhaps the most famous use of data Wolniewicz, Grimes, Johnson, & Kaushansky, 2000;mining to acquire new telecommunications customers Mani, Drew, Betz & Datta, 1999; Masand, Datta, Mani, 487
  3. 3. Data Mining in the Telecommunications Industry& Li, 1999). These systems almost always utilize call was augmented by comparing the new calling be-detail and contract data, but also often use other data havior to profiles of generic fraud—and fraud is onlyabout the customer (credit score, complaint history, signaled if the behavior matches one of these profiles.etc.) in order to improve performance. Churn predic- Customer level data can also aid in identifying fraud.tion is fundamentally a very difficult problem and, For example, price plan and credit rating informationconsequently, systems for predicting churn have been can be incorporated into the fraud analysis (Rosset,only moderately effective—only demonstrating the Murad, Neumann, Idan, & Pinkas, 1999). More recentability to identify some of the customers most likely work using signatures has employed dynamic clusteringto churn (Masand et al., 1999). as well as deviation detection to detect fraud (Alves et al., 2006). In this work, each signature was placedTelecommunications Fraud Detection within a cluster and a change in cluster membership was viewed as a potential indicator of fraud.Fraud is very serious problem for telecommunica- There are some methods for identifying fraud thattion companies, resulting in billions of dollars of lost do not involve comparing new behavior against arevenue each year. Fraud can be divided into two profile of old behavior. Perpetrators of fraud rarelycategories: subscription fraud and superimposition work alone. For example, perpetrators of fraud oftenfraud (Fawcett and Provost, 2002). Subscription fraud act as brokers and sell illicit service to others—and theoccurs when a customer opens an account with the illegal buyers will often use different accounts to callintention of never paying the account and superim- the same phone number again and again. Cortes andposition fraud occurs when a perpetrator gains illicit Pregibon (2001) exploited this behavior by recognizingaccess to the account of a legitimate customer. In this that certain phone numbers are repeatedly called fromlatter case, the fraudulent behavior will often occur compromised accounts and that calls to these numbersin parallel with legitimate customer behavior (i.e., is are a strong indicator that the current account maysuperimposed on it). Superimposition fraud has been be compromised. A final method for detecting frauda much more significant problem for telecommunica- exploits human pattern recognition skills. Cox, Eicktion companies than subscription fraud. Ideally, both & Wills (1997) built a suite of tools for visualizingsubscription fraud and superimposition fraud should data that was tailored to show calling activity in suchbe detected immediately and the associated customer a way that unusual patterns are easily detected by us-account deactivated or suspended. However, because it ers. These tools were then used to identify internationalis often difficult to distinguish between legitimate and calling fraud.illicit use with limited data, it is not always feasibleto detect fraud as soon as it begins. This problem is Telecommunication Network Faultcompounded by the fact that there are substantial costs Isolation and Predictionassociated with investigating fraud, as well as costs ifusage is mistakenly classified as fraudulent (e.g., an Monitoring and maintaining telecommunication net-annoyed customer). works is an important task. As these networks became The most common technique for identifying super- increasingly complex, expert systems were developedimposition fraud is to compare the customer’s current to handle the alarms generated by the network elementscalling behavior with a profile of his past usage, using (Weiss, Ros & Singhal, 1998). However, because thesedeviation detection and anomaly detection techniques. systems are expensive to develop and keep current, dataThe profile must be able to be quickly updated because mining applications have been developed to identifyof the volume of call detail records and the need to and predict network faults. Fault identification can beidentify fraud in a timely manner. Cortes and Pregi- quite difficult because a single fault may result in abon (2001) generated a signature from a data stream cascade of alarms—many of which are not associatedof call-detail records to concisely describe the calling with the root cause of the problem. Thus an importantbehavior of customers and then they used anomaly part of fault identification is alarm correlation, whichdetection to “measure the unusualness of a new call enables multiple alarms to be recognized as beingrelative to a particular account.” Because new behavior related to a single fault.does not necessarily imply fraud, this basic approach488
  4. 4. Data Mining in the Telecommunications Industry The Telecommunication Alarm Sequence Analyzer the telecommunications industry, but as telecommuni-(TASA) is a data mining tool that aids with fault iden- cation companies start providing television service to Dtification by looking for frequently occurring temporal the home and more sophisticated cell phone servicespatterns of alarms (Klemettinen, Mannila & Toivonen, become available (e.g., music, video, etc.), it is clear1999). Patterns detected by this tool were then used to that new data mining applications, such as recommenderhelp construct a rule-based alarm correlation system. systems, will be developed and deployed.Another effort, used to predict telecommunication Unfortunately, there is also one troubling trend thatswitch failures, employed a genetic algorithm to mine has developed in recent years. This concerns the in-historical alarm logs looking for predictive sequential creasing belief that U.S. telecommunication companiesand temporal patterns (Weiss & Hirsh, 1998). One are too readily sharing customer records with govern-limitation with the approaches just described is that mental agencies. This concern arose in 2006 due tothey ignore the structural information about the under- revelations—made public in numerous newspaper andlying network. The quality of the mined sequences can magazine articles—that telecommunications companiesbe improved if topological proximity constraints are were turning over information on calling patterns toconsidered in the data mining process (Devitt, Duffin the National Security Agency (NSA) for purposes ofand Moloney, 2005) or if substructures in the telecom- data mining (Krikke, 2006). If this concern continues tomunication data can be identified and exploited to allow grow unchecked, it could lead to restrictions that limitsimpler, more useful, patterns to be learned (Baritchi, the use of data mining for legitimate purposes.Cook, & Lawrence, 2000). Another approach is to useBayesian Belief Networks to identify faults, since theycan reason about causes and effects (Sterritt, Adamson, CONCLUSIONShapcott & Curran, 2000). The telecommunications industry has been one of the early adopters of data mining and has deployed numer-FUTURE TRENDS ous data mining applications. The primary applications relate to marketing, fraud detection, and networkData mining should play an important and increasing monitoring. Data mining in the telecommunicationsrole in the telecommunications industry due to the industry faces several challenges, due to the size oflarge amounts of high quality data available, the com- the data sets, the sequential and temporal nature of thepetitive nature of the industry and the advances being data, and the real-time requirements of many of themade in data mining. In particular, advances in mining applications. New methods have been developed anddata streams, mining sequential and temporal data, existing methods have been enhanced to respond toand predicting/classifying rare events should benefit these challenges. The competitive and changing naturethe telecommunications industry. As these and other of the industry, combined with the fact that the industryadvances are made, more reliance will be placed on the generates enormous amounts of data, ensures that dataknowledge acquired through data mining and less on the mining will play an important role in the future of theknowledge acquired through the time-intensive process telecommunications industry.of eliciting domain knowledge from experts—althoughwe expect human experts will continue to play animportant role for some time to come. REFERENCES Changes in the nature of the telecommunicationsindustry will also lead to the development of new ap- Aggarwal, C. (Ed.). (2007). Data Streams: Models andplications and the demise of some current applications. Algorithms. New York: Springer.As an example, the main application of fraud detectionin the telecommunications industry used to be in cellular Alves, R., Ferreira, P., Belo, O., Lopes, J., Ribeiro, J.,cloning fraud, but this is no longer the case because the Cortesao, L., & Martins, F. (2006). Discovering telecomproblem has been largely eliminated due to technologi- fraud situations through mining anomalous behaviorcal advances in the cell phone authentication process. patterns. Proceedings of the ACM SIGKDD WorkshopIt is difficult to predict what future changes will face on Data Mining for Business Applications (pp. 1-7). New York: ACM Press. 489
  5. 5. Data Mining in the Telecommunications IndustryBaritchi, A., Cook, D., & Holder, L. (2000). Discov- Liebowitz, J. (1988). Expert System Applications toering structural patterns in telecommunications data. Telecommunications. New York, NY: John Wiley &Proceedings of the Thirteenth Annual Florida AI Re- Symposium (pp. 82-85). Mani, D., Drew, J., Betz, A., & Datta, P (1999). StatisticsCortes, C., & Pregibon, D (1998). Giga-mining. Pro- and data mining techniques for lifetime value modeling.ceedings of the Fourth International Conference on Proceedings of the Fifth ACM SIGKDD InternationalKnowledge Discovery and Data Mining (pp. 174-178). Conference on Knowledge Discovery and Data MiningNew York, NY: AAAI Press. (pp. 94-103). New York, NY: ACM Press.Cortes, C., & Pregibon, D. (2001). Signature-based Masand, B., Datta, P., Mani, D., & Li, B. (1999).methods for data streams. Data Mining and Knowledge CHAMP: A prototype for automated cellular churnDiscovery, 5(3), 167-182. prediction. Data Mining and Knowledge Discovery, 3(2), 219-225.Cox, K., Eick, S., & Wills, G. (1997). Visual data min-ing: Recognizing telephone calling fraud. Data Mining Mozer, M., Wolniewicz, R., Grimes, D., Johnson, E.,and Knowledge Discovery, 1(2), 225-231. & Kaushansky, H. (2000). Predicting subscriber dis- satisfaction and improving retention in the wirelessDevitt, A., Duffin, J., & Moloney, R. (2005). Topo- telecommunication industry. IEEE Transactions ongraphical proximity for mining network alarm data. Neural Networks, 11, 690-696.Proceedings of the 2005 ACM SIGCOMM Workshopon Mining Network Data (pp. 179-184). New York: Rosset, S., Murad, U., Neumann, E., Idan, Y., & Gadi,ACM Press. P. (1999). Discovery of fraud rules for telecommu- nications—challenges and solutions. Proceedings ofFawcett, T., & Provost, F. (2002). Fraud Detection. In the Fifth ACM SIGKDD International Conference onW. Klosgen & J. Zytkow (Eds.), Handbook of Data Knowledge Discovery and Data Mining (pp. 409-413).Mining and Knowledge Discovery (pp. 726-731). New New York: ACM Press.York: Oxford University Press. Rosset, S., Neumann, E., Eick, U., & Vatnik (2003).Freeman, E., & Melli, G. (2006). Championing of Customer lifetime value models for decision LTV model at LTC. SIGKDD Explorations, 8(1), Data Mining and Knowledge Discovery, 7(3), 321-27-32. 339.Getoor, L., & Diehl, C.P. (2005). Link mining: A survey. Sasisekharan, R., Seshadri, V., & Weiss, S (1996). DataSIGKDD Explorations, 7(2), 3-12. mining and forecasting in large-scale telecommunica-Hill, S., Provost, F., & Volinsky, C. (2006). Network- tion networks. IEEE Expert, 11(1), 37-43.based marketing: Identifying likely adopters via con- Sterritt, R., Adamson, K., Shapcott, C., & Curran, E.sumer networks. Statistical Science, 21(2), 256-276. (2000). Parallel data mining of Bayesian networks fromKaplan, H., Strauss, M., & Szegedy, M. (1999). Just telecommunication network data. Proceedings of thethe fax—differentiating voice and fax phone lines us- 14th International Parallel and Distributed Processinging call billing data. Proceedings of the Tenth Annual Symposium, IEEE Computer Society.ACM-SIAM Symposium on Discrete Algorithms (pp. Wei, C., & Chiu, I (2002). Turning telecommunications935-936). Philadelphia, PA: Society for Industrial and call details to churn prediction: A data mining approach.Applied Mathematics. Expert Systems with Applications, 23(2), 103-112.Klemettinen, M., Mannila, H., & Toivonen, H. (1999). Weiss, G., & Hirsh, H (1998). Learning to predict rareRule discovery in telecommunication alarm data. events in event sequences. In R. Agrawal & P. StolorzJournal of Network and Systems Management, 7(4), (Eds.), Proceedings of the Fourth International Confer-395-423. ence on Knowledge Discovery and Data Mining (pp.Krikke, J. (2006). Intelligent surveillance empowers 359-363). Menlo Park, CA: AAAI analysts. IEEE Intelligent Systems, 21(3),102-104.490
  6. 6. Data Mining in the Telecommunications IndustryWeiss, G., Ros, J., & Singhal, A. (1998). ANSWER: Call Detail Record: Contains the descriptive in-Network monitoring using object-oriented rule. Pro- formation associated with a single phone call. Dceedings of the Tenth Conference on Innovative Ap- Churn: Customer attrition. Churn prediction refersplications of Artificial Intelligence (pp. 1087-1093). to the ability to predict that a customer will leave aMenlo Park: AAAI Press. company before the change actually occurs.Weiss, G. (2004). Mining with rarity: A unifying frame- Signature: A summary description of a subset ofwork. SIGKDD Explorations, 6(1), 7-19. data from a data stream that can be updated incremen-Winter Corporation (2003). 2003 Top 10 Award Win- tally and quickly.ners. Retrieved October 8, 2005, from http://www. Subscription Fraud: Occurs when a opens an account with no intention of ever paying forTenwinners.asp the services incurred. Superimposition Fraud: Occurs when a perpetra- tor gains illicit access to an account being used by aKEY TERMS legitimate customer and where fraudulent use is “su- perimposed” on top of legitimate use. Bayesian Belief Network: A model that predictsthe probability of events or conditions based on causal Total Lifetime Value: The total net income a com-relations. pany can expect from a customer over time. 491