Tera stream for datastreams


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • In 2009, Younglimwon Soft Lab set a record expanding its sales by 50%. In addition, the company is making efforts to diversify by as developing new products and improving quality through R&D investment and holding a CEO forum and a CEO workshop for corporate customers.   These efforts have paid off. The company has enjoyed ratio of 90% or higher signing maintenance contracts with corporate customers. In 2010, the company recorded 20 win-backs beating larger competitors.
  • Tera stream for datastreams

    1. 1. © 2012 DataStreams Corp. All Rights Reserved.
    2. 2. Data Integration as Data Infrastructurecontent Why TeraStream™? Case Studies Company Profile Q&A © 2012 DataStreams Corp. All Rights Reserved.
    3. 3. Data IntegrationAsData Infrastructure © 2012 DataStreams Corp. All Rights Reserved.
    4. 4. Enterprise Data Management SuiteMinimal implementation effort and producing high-qualityBusiness Intelligence output. Real Time Monitoring • Performance Tuning • Emergency Management Source ReportsSystems Integrated ODS / DW Report Mart MERGE SPLIT Aggregation Calculation ETL / Real Time / Near Real Time SORT JOIN Conversion Reformat Data Governance Architecture Master Data Meta Data Data Quality Impact Analysis Management © 2012 DataStreams Corp. All Rights Reserved.
    5. 5. Possession of Key Technology © 2012 DataStreams Corp. All Rights Reserved.
    6. 6. DataStreams Products ETL Test Data Security Change Data Capture Near Real Time Meta Data Data Quality Master Data Management Impact Analysis Common Benefit of DataStreams Products  Simple Mgt. & Upgrade  Stability  TCO Reduction  One-stop support  Saving time cost by Speed  Extreme Performance © 2012 DataStreams Corp. All Rights Reserved.
    7. 7. Why TeraStream™ ? © 2012 DataStreams Corp. All Rights Reserved.
    8. 8. Floods of Massive Data To get Business intelligence on the Data.. Need Data Integration ERP Systems Other Operational Apps Legacy Systems Web Logs etc Call Center Apps RDBMSsFile based information Unstructured Data © 2012 DataStreams Corp. All Rights Reserved.
    9. 9. Data Integration LandscapeBusiness Challenges Impact Factors Result  Multiple versions of the  Disparate data  Bad decisions “Truth”  Inaccurate data  Wasted time and  Lost revenue  Incomplete data resources aggregating  Lost productivity information  Untimely data  Lost market opportunity  Difficult to use Data  Fragmented  Delayed Decision making  Bad Citizen relationships Integration Approach  Uninformed management This is more than 30 percent of corporate IT budgets so data integrity is used to emphasize what is important. © 2012 DataStreams Corp. All Rights Reserved.
    10. 10. Well PositionedTeraStream is very well positioned in the Data Integrationmarketplace. Powerful Performance (TeraSort™) High-speed extraction (FACT™) Reuse of data (ETL and Batch Hub) Over 200 Customers Serving multiple industries including banking, government retail Over a decade of experience Intuitive GUI Easy to Operate Easy to maintain Fast implementation Easy customization Low resource use © 2012 DataStreams Corp. All Rights Reserved.
    11. 11. TeraStream™ Positioning  A TeraStream™ bundle, one Unload tool, one Sort tool and ETL metadata repository  An end to end, High Performance driven data integration platform. • Basel II • Risk Management • Data Warehouse • Data Migration • Data Synchronization • Batch Execution • Master Data Management © 2012 DataStreams Corp. All Rights Reserved.
    12. 12. TeraStream™ Data IntegrationExtraction, Transformation, Load with High performanceSORT Engine Data extraction from various High Performance SORT engine resolves time bottleneck due to transform Automatic generation of scripts can be used for loading to various commercial DBMS in high speed LARGE DATUM DBMSs Text Text Data Data MERGE SPLIT Aggregation Calculation High Performances SORT Engine SORT JOIN Conversion Reformat EXTRACT TRANSFORM LOAD © 2012 DataStreams Corp. All Rights Reserved.
    13. 13. TeraStream™ Data Integration Benefits3times Elapsed time improvement against its competitorResult on Data Migration in Shinhan Bank, Korea TeraStream™ IBM DataStage FILE → FILE FILE → DB FILE → DB DB → DB  thread MAX for sort =3  Parallel = 4  File manipulation : 35% CPU usage  File manipulation : 58% of CPU usage.  Load : 80% of peak CPU usage  Load: 58% of peak CPU usage Conclusion Conclusion  Elapse time : 20 minutes  Elapse time : 59 minutes  Wasted System Resource : 800  Wasted System Resource : 3000 ( 40% Avg. CPU usage X 20 mins ) (50% Avg. CPU usage X 60 mins) © 2012 DataStreams Corp. All Rights Reserved.
    14. 14. TeraStream™ Near Real Time DataIntegration Processing BenefitsHigh Performance improvement on Real Data processingResult on DW Benchmark Shinhan Bank, Korea (Minutes) 50 DataStage 40 30 10 03 TeraStream 02 01 0 1 10 100 200 1,000 2,000 (Ten thousand records) © 2012 DataStreams Corp. All Rights Reserved.
    15. 15. TeraStream™ Large Data BenefitsHigh Performance improvement on Very Large Data processingResult on Batch Job Benchmark National Health Insurance Corporations, Korea (2005) 55% Reduction 50% Reduction 90% Reduction 70day 11day 4day 7day 5day 2day © 2012 DataStreams Corp. All Rights Reserved.
    16. 16. TeraStream™ Batch Processing BenefitsHigh Performance improvement on Batch Jobs executionthan DB processingResult on Batch Job Benchmark Post Insurance, Korea 40,000 400,000 4,000,000 Records Records Records 12x 3x © 2012 DataStreams Corp. All Rights Reserved.
    17. 17. TeraStream™ EBH BenefitsMassive volume of files extracted from source systemsare stored in EBH, and reuse in next stepETL Time is reduced by avg. 56% In case of LG Telecom Legacy ODS EDW MART ORACLE 8.0.6 ORACLE Teradata Sybase EBH (ETL Batch Hub)EBH (ETL and Batch Hub)stores temporary or result files which is shared for further table generation in EDW and DATA MART. © 2012 DataStreams Corp. All Rights Reserved.
    18. 18. TeraStream™ FeaturesIntuitive User Interface Intuitive task flow Metadata Repository GUI for developers Impact analysis Change history manager Scheduler Editor window Scheduling by time / Mapping Creation period / business calendarConfiguration Management Project Version Execution log Web Monitoring File View © 2012 DataStreams Corp. All Rights Reserved.
    19. 19. TeraStream™ FeaturesHigh speed data extraction with FACT unload engine DBMS Supported SQL is supported High Speed Extraction SQL Database API optimized Split data Filtering Export Format  Time  Timestamp  Fixed / Variable  CLOB / BLOB  NULL  FRAME  FILL © 2012 DataStreams Corp. All Rights Reserved.
    20. 20. TeraStream™ FeaturesDatabase Connectivity and Diverse manipulation processand functions DBMS Supported N:N mapping File to DB or DB to File Built-in Function User Function Supported Function Types  INSERT  DELETE  UPDATE  INSERT - UPDATE  DELETE – INSERT  EBCDIC ↔ ASCII  CLOB / BLOB  NDB (Unisys 9-bit)  HDB (IBM) © 2012 DataStreams Corp. All Rights Reserved.
    21. 21. TeraStream™ FeaturesUses various transportation method according to filestructure, security, amount of record and etc. DBMS File to DB data load for bulk data  Provide Bulk Load to process large data  Partition Load Bulk Load  Parallel Load DBMS Load scripts automatically  Automatically Generate Script by using DBMS Load Script Schema information Automatic  Fixed / Variable  Provide DBMS Option Encryption while transporting  Data Readable Symmetric Encryption  Maintain original data characteristic  Support various encryption pattern © 2012 DataStreams Corp. All Rights Reserved.
    22. 22. Case Studies © 2012 DataStreams Corp. All Rights Reserved.
    23. 23. Kookmin BankEDW and integrated DM installation  Various DBMS(IMS HDB, HOST DB2, Oracle, DB2 UDB) integration by using Expected TeraStream™ Result  Meeting batch target time of 2 hours and 30 minutes for 4TB of EBCDIC data. Issues System configuration• M/F and IMS HDB conversion File process flow DB QUERY• Processing changed data in absence of time-series column Source system EDW• Processing large size data within batch process IMS HDB Sybase ASIQ 12.7 time(10TB/day based on source data)• How to process high volume files in parallel ETL A-SOR DM Informover Plans IBM M/F - Seg. split - conversion & Array split• Converting main frame data into data in Unix - logic applied environment (10TB → 25TB) within 18 hours.• Various data conversion and processing including HDB, DB2 ETL Korean character conversion - conversion• ETL task from accounting system server to new ODW Informover - logic applied server(extracting appx. 200 GB of daily changed data within 1 hour and 30 minutes by using FACT module of Server RDB TeraStream™) TS(FACT) ETL• ETL and Batch process in unified way.• Batch job in core banking system within 6 hours. - Logic applied © 2012 DataStreams Corp. All Rights Reserved.
    24. 24. Samsung ElectronicsGlobal Database Integration using NRT ETL  Efficiency is maintained despite cross transportationExpected  Bi-directional NRT integration allows the same service regardless of system type Result and country (Time from extraction to loading is 20 minutes.)  Bi-directional remote data transportation using TeraStreamIssues System Configuration• Registration in one country should have the same Smart Phone Smart Phone System in China service at other country. System in Germany• duplicated record should be avoided due to cross transportation. DBs in DBs in• 20 minutes Near Real-time Service Service• Perfect Recovery scheme should be presentedPlans Sam To DB UPSERT• Rea-time data transportation between Germany and SAM To DB NRT Extract NRT Extract UPSERT China.• Bi-directional synchronization between TeraStream of Germany and China.• 20 min. MAX loading time for transported data is implemented using TeraStream NRT. 프로그램 성공, 실패 등 실행 결과 Web Monitoring• Web Monitoring is developed © 2012 DataStreams Corp. All Rights Reserved.
    25. 25. LG TelecomLG Telecom new billing system data transfer  The working hours shortened to D +3 and D +1 in reducing the system load Expected  On average, 56% of the effect of reducing working hours Result  Emergency response system rework due to delay in securing and providing data to minimize Impact Issues System Configuration• Solution provided by „I‟ company requires more than twelve hours for processing every billing and call data. ETL EDW Server• It delays entire processes and often requires re- processing of data. D+1• Efficient unique key generation for entire business tasks Custo Call Data Billing Legacy ODS Server merPlans Contacts PPS/ BSS• Transition from old to new billing system AR SRDF ODS Mining Input Campaign MOLAP - Data size: 3TB→ 3.5TB, Object: Transition in Oracle Oracle/ Variables Analysis Analysis Billing Informatica ETL 30 minutes Mining Teradata Analysis• Move unchanged data among large dataset three CSM CCS CTI NCR 10Node days prior to the new system open date. MPS PPS DM Server• Separate files that will be loaded to EDW and DM ERP NMS and load them in different business tables. OLAP TeraStream loads data transformed Mart• Unique key generation for entire business process Sybase ASIQ is done first. in ODS to EDW and DM at the same time. IBM P Series © 2012 DataStreams Corp. All Rights Reserved.
    26. 26. Company Profile © 2012 DataStreams Corp. All Rights Reserved.
    27. 27. Company ProfileCompany DataStreams Corp.CEO Young-sang LeeEstablished 2001.9.19 Data Integration Solutions Development and Sales - ETL/data migration/high performance batch product -Deferred (Near Real Time) data integration solution - Real time data integration product Data Quality Solutions Development and SalesBusiness -Metadata and data quality management solutions - Application impact analysis product  Master Data Management Solution  Data Warehouse & BI Development / SNS Analytics  Data Standardization & Quality Management Consulting  High Performance Batch Job Technical ConsultingInvestor JAFCO ASIAHeadquarter Chungho-Nais B/D 6F, 1597-3, Seocho-dong, Seocho-gu, Seoul, Korea #601, U-Spacemall #2, Sampyoung-dong, Bundang-gu, Seongnam-si,R&D Center Gyounggi-do , KoreaHomepage www.datastreams.co.kr © 2012 DataStreams Corp. All Rights Reserved.
    28. 28. Business AreaDataStreams provides “data integration and quality solutions” that enable your company tostand firmly on concrete data infrastructure. It allows you to analyze and utilize yourinternal or external data to gain competitiveness by empowering with timely, relevant andtrustworthy data for your top business imperatives. Based on leading data integration andquality solutions, DataStreams has expanded its business area to Master Data Managementand Information Analytics as well as big data performance improvement, data migration,various data integration and metadata management. Data Integration Solutions Meta Data & Quality Development / Sales Management Solutions - Metadata and data - ETL / Data Migration / High-speed Batch Tool quality management solution - Real-time data Integration Product Business - Application impact analysis product - Deferred data integration solution(NRT) Areas Data Quality Consulting D/W BI Development Master Data High-speed batch Management Solution & processing technical Consulting SNS Analytics consulting © 2012 DataStreams Corp. All Rights Reserved.
    29. 29. Mission of DataStreamsDataStreams is evolved toward data management company based on over 13 years ofexperience in data integration and quality management. DataStreams helps informationusers to build concrete data infrastructure for integration and quality management. Basedon superb quality of on-hand data, DataStreams‟s mission to satisfy customers by raisingenterprise value and competitiveness will be accomplished. No.1 end-to-end data integration and quality management solution provider Help your enterprise by providing data governance capability Consult and build systems how to utilize your accumulated data for long years © 2012 DataStreams Corp. All Rights Reserved.
    30. 30. Market Share & ReputationDataStreams is No.1 data integration and metadata management solution provider in Korea. DataStreamshas been selected for 30 Korean software company worthy of global recognition in 2012.Market share Good ReputationNo. 1 Share in Domestic Market Etc 6% SAS 3% SAP 5% DataStreams ORACLE 28% 6% INFOMATICA, 10% Wise IBM 9% 10% Encore Gtone BTL 7% 10% 6% <Domestic M/S> * Domestic M/S is researched by DataStreams (2008) • M/S of Global Vendors is quoted by "Worldwide DI and Access Software 2006 Vendor Shares: Market and Segments," IDC. © 2012 DataStreams Corp. All Rights Reserved.
    31. 31. CredentialsDataStreams is recognized as leading company in data integration & quality management in domesticmarket. DataStreams is the only solution provider who has expertise in both solution market. DataStreamshas developed its own methodologies for master data management and data governance technologies forEDM. As a leader, DataStreams owns good references in various industrial fields (especially, more than 80%of market share in domestic banking industry). Data Integration Data Quality Methodology & Experiences Data quality technologies with metadata Data Integration technologies Its own methodology for MDM & lot management, data quality and application with ETL, CDC and NRT solutions. of technical experiences impact analysis solutions. Progress No. 1 Market Share Professional Support DataStreams 80% Perfect data governance for data administrator is progressive approach for EDM . It is based on product line-up from 80% of M/S in domestic Banking Market, Technical support and quality assurance ETL to real-time and metadata to impact Market Share, ETL(55%)#1 in with Over 60 Consultants and over 34 analysis solutions, Data Migration(60%)#1 in Korea R&D personnel © 2012 DataStreams Corp. All Rights Reserved.
    32. 32. DataStreams is the SW Market Leader in KoreaTeraStream™ has No.1 market share and stellar references in ETL and Migration market in Korea. 5% 15% ETL & 55% Migration 25% TeraStream IBM Data Stage Informatica Other - 31 - © 2012 DataStreams Corp. All Rights Reserved.
    33. 33. DataStreams dominates Korean banking industryMost of major banks in Korea use TeraStream™ as their major ETL product. TeraStream isunique market leader in Korean banking industry. © 2012 DataStreams Corp. All Rights Reserved.
    34. 34. The Market Leader in Korean is going globalDataStreams already has a strong global presence with many of its products and plans to rollout its entire data management product suite internationally Merklenet.Inc. Accenture Gmbh Airweb CSC Consulting Sungard Bisys American Air Telefonica Soluciones Reliance Industry Comcast Highmark, Inc Procecard Excelcom Merkle Data Tech Techocom Sodexhopass Telkomsel Aviva Banco Colpatria Credibanco © 2012 DataStreams Corp. All Rights Reserved.
    35. 35. Strong Track Record of Success2012 05 Open Pangyo Techno valley R&D Center 2005 04 TeraStream™ won New Technology Certification from Ministry of Knowledge Economy2011 12 Datastreams China. established 04 Acquired KDB Solution Co., Korea‟s first meta-data 11 Received SW achievement award by Korean Government management solution company. 06 Selected as an ATC(Advanced Technology Center) 03 Awarded for Product Competitiveness from SMBA2010 12 TeraStream™ V.3.2 released 2004 08 TeraStream™, trademark registered (No.0589927) 03 CEO, Young S. Lee elected as chairman of 05 First worldwide sales contract of FACT™ KOSEA(Korea Software Enterprise Association). 04 Selected as technically innovated company of 01 MetaStream™ V.2.7 released 2004 by Small & Medium Business Administration2009 06 QualityStream™, ImpactStream™ released 2003 10 Presented FACT™ to Oracle Open World 2003 in San Francisco, USA 02 DeltaStream™ V.1.1 released 08 KEB selected TeraStream™ as standard Batch/ 06 Contracted with Intellectual Property Office ETL solution for Next Generation Banking System.2008 for Enterprise Data Quality Management. 03 Company‟s name change to DataStreams 2002 12 First TeraStream™ V.1.1 contract at National Statistics Office 11 JAFCO, Japan invested 4 million USD. 07 TeraStream™ exposed in SEK2002, Seoul 07 Awarded for Excellent Venture Company by 06 TeraStream™ Version 1.1, Released2007 Deputy Prime Minister 01 Registered TeraStream™ as a trademark 06 TeraStream™ Version 2.0 released 05 MetaStream™, awarded for Digital Business 2001 09 Innovative Data Solutions Corp., established Innovation by SMBA2006 12 TeraStream is selected as standard ETL for 16 2000 04 TeraStream development started 01 CoSort was successfully applied to a DW Province Public Administration Office project for the first time at LG company 11 TeraStream™ V.1.4 won GS(Good Software) Certification 1999 09 CoSort Korea, established © 2012 DataStreams Corp. All Rights Reserved.
    36. 36. Marquee Accounts in Korea and Beyond Banks Insurance Securities Card/ Finance Government/ Public Sector/ Universities Telco Manufacturing /Services © 2012 DataStreams Corp. All Rights Reserved.
    37. 37. Thank youwww.datastreams.co.kr