Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Traffic Insight Using Netflow and Deepfield Systems


Published on

Traffic Insight Using Netflow and Deepfield Systems by Mohd Rizal

Published in: Internet
  • Login to see the comments

Traffic Insight Using Netflow and Deepfield Systems

  1. 1. Traffic Insight NETFLOW & DEEPFIELD SYSTEMS Prepared by: Mohd Rizal Mohd Ramly, TM Berhad Saiful Akmal Towfek, TM Berhad Mohd Farhan Tajwid , TM Berhad
  2. 2. There is no visibility of traffic per user OurProblemStatement Need to have better understanding & info on network behavior ie Video Streaming. IMPACT: Planning and dimensioning will be difficult relying only on utilization data (totally network physical behaviour) Category/ApplicationVisibility Costly for full-blown deep packet inspection BUT do we really need all the information given by DPI? Would netflow information be enough for basic network information? DeepPacketInspection(DPI)Cost Cheaper solution combining existing `network definition map` from DEEPFIELD with granularity of per user from netflow v9. 1cloudgnome is product from DEEPFIELD (Nokia) with simple network signature but in aggregate view
  3. 3. Traffic category based on netflow information NetflowInformation NetflowCapability Interface ToS Protocol Source IP Address Destination IP Address Source Port Destination Port Link Layer Header IP Header TCP/UDP Header NETFLOW ü Monitors data in Layer 2 thru 4 ü Determines applications by port ü Utilizes a seven- tuple for flow ü Flow information who, what, when where v With new DEEPFIELD systems, the definition for category done by cloudgnome (DEEPFIELD).
  4. 4. Comparison between netflow and DPI NETFLOWvsDPI Up to homepass (PPPoE dialer) with public IP. GRANULARITY Cannot be done, only for traffic utilization QoE Only for peering with AS number. IP source may not be accurate COVERAGE No near realtime or realtime. ONLY DESCRIPTIVE ANALYTIC ANALYTICDETAILS Client devices view. Able to identify brand/type of devices Able to identify each QoE measurement based on Layer 7 performance Accurate based on DPI signature. Platform DPI ready up to PREDICTIVE ANALYTIC and QoE assessment Netflow DPI Impact Planning and network design can be done in an optimize way. Personalization cannot be done Identify total homepass view up to category traffic ie: streaming, web browsing (no quality measurement) Only can identify top peering performance ie: Netflix, Apple, Google, Facebook As and when descriptive analytic for planning purpose and brief understanding for each top applications category streaming.
  5. 5. NameNodex6 International Gateway MasterNode MSAN xDSL FTTH COPPER Transmission COPPER FOC Transmission OLT Access Gateway 10G/100G 10G/100G 10G/100G DEEPFIELD SYSTEMS Contain gnomecloud IP CORE CLOUD Collector Netflow Collector NetworkTopology 3 core members with support of 7 staff. Manpower Create a temporary Hadoop with 6 name node. How All ingress traffic at TM’s gateway routers to eliminate to reduce complexity and velocity data Strategy
  6. 6. Huge task waiting to massage/exploratory huge data with high velocity TheProcess Network protocol for collecting IP traffic information (up to layer 4) from DEEPFIELD with traffic category mapping Challenge Raw IP data with no user login information. Huge data (700MB for 5 minutes) Netflow Data exploratory and logic creation to understand the behavior of customers and application/network trending Exploratoryandlogic Mapping for source IP to domain/sites/application & category DEEPFIELDcloudgnome To map the customers login with IP and get the package information (subscriber info) RadiusData
  7. 7. Exploit the Deepfield capability utilizing Cloud Genome NetflowClassification 01 02 03 04 Deepfield Capability to distinguish each network flow based on signature content. Different category content (ie: video, dns, cache etc) within same port, IP source and protocol Based on aggregation data not single customer flow DEEPFIELDprovidetheintelligence We able to identify the user from the radius info RADIUSgivesusidentification With 5 minutes volume, we able to derive the throughput performance and with identification of source and destination NETFLOWgiveustheflowandusage With this integration, it will help to map the geolocation and others marketing requirement. ie: 30Mbps with quota vs 30Mbps unlimited BSS&InventoryIntegration
  8. 8. TheChallenges Critical to Further Achievements ProcessingPower 01 Huge data from Radius and Netflow will be difficult to handle cloudgnomeLimitation 02 Only on selected routers CHALLENGE Need high powered Hadoop servers with speed & large storage Mapping with cloudgnome will be tricky due to multiple category within same IP + port CURRENT MITIGATION Data just for 4 weeks due to storage and speed. (descriptive analytic) Mathematic modelling with assumption of normal distribution 03 Skillset and ability to analyse huge and high velocity data Ability to understand the network with analytic thinking ability Training and practical lesson Manpowerandteam 04 If we take every datapoint each timestamp, it will be huge Difficulties to mapping with radius and will be massive task and takes time Take first 5 minutes for each ½ an hour SamplingRateDecision
  9. 9. Data for 2 weeks: 26/XX/201X to X/XX/201X StreamingOTTInsights Netflixstats q Unique users stream – 2XX,X9X* q Daily max – 1X,7X9 unique user AND between 9.25 pm to 10.00 pm q Average stream traffic @ 3.954Mbps IFLIXstats q Unique users stream – XX,547 q Daily max – X,5X2 unique users AND between 9.25 pm to 10.00 pm q Average stream traffic @ 1.372Mbps VIUstats 0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 16,000 18,000 20,000 22,000 viuiflixneflix q Unique users stream – X3,X0X* q Daily max – XX6 unique users AND between 9.25 pm to 10.00 pm q Average stream traffic @ 0.185Mbps XX/XX/201X XX/XX/201X The capability can be extended to geographical location, ie: more users from SEGAMBUT watch VIU compare to IFLIX & we can identify exact user watch which content group XX/XX/201X XX/XX/201X XX/XX/201X XX/1XX201X *data plot for 5 days only INSIGHT 1
  10. 10. How Much distribution of DNS server for TM’s Unifi Subscribers (public IP only) DNSServerUsage 5X.5X% 0.X2% XX.1% X.4X% 0.XX% X1X.2K subscribers confidence with GOOGLE DNS X1.5K subscribers used cloudfare DNS Less than half (6XXK) still maintain their TM DNS configuration 3XK subscribers config their RG with OpenDNS X.5K subcribers used comodo DNS Most probably, customers config with other DNS due to accessibility to block websites by MCMC/government * With assumption that if specific users already have records using others DNS, we eliminate the possibility they using TM’s DNS. INSIGHT 2
  11. 11. From Deepfield Data TopDomainVisit TopWebVisitbyTM’sSubscriber From aggregate data we able to distinguish the TOP 15 of the domain visit by our subscribers contribute in 80% from overall domain reach by all users. Youtubecontribute3X.X% Behaviour of Malaysian watching content via youtube with the booming of independent content provider PaidContent:Netflix Eventhough paid service, but with variety and impressive quality of content, users will subscribed . MessagingApps 77% from detected messenging apps is WHATSAPP with peak traffic 39Gbps Games No 7 in top category with VALVE, Nintendo and ROBLOX is the top 3 contributors Top websites by TM’s Subscribers others With netflow data, we able to distinguish each user details based on cloud genome from DEEPFIELD. INSIGHT 3
  12. 12. From Data Derive by NETFLOW, the planning become more realistic StrategicPlanning NetworkPlanning During Deployment 115,52k 35,52k Distribution of mass home user and business (example) FTTH Metro ethernet OLT OLT OLT OLT Access OLT business home 01 We able to determine network capacity based on multivariate linear regression up to CORE network. During Campaign Determine new predictive traffic based on marketing campaign. Ease the planning and prevent any congestion up to CORE network 02 Scenario INSIGHT 4