16/09 – replaced ‘clients’ by ‘customers’ everywhere (telcos jargon)
Integrationisbroad and functionallycomplex(much more thanthishighlevelviewwhichalready shows somelevel of complexity)Questions to Iberdrola:Is your system integrated? With WMS? With AM? With CIS? How?How do you set up the data in the HES and MDM?How do you commission meterstoday?
BI adaptadoparatrabajar en memoria.In-memory Cache: decide quédatoscachearTimesten – DB generalEssbase – analítica multidimensional, predicciones, en memoria.Interfaces de conexiónnInfiniband: HA con otrosExalytics o paraExadata
Exalytics & Exadata: plataforma de BI más rápida del mundo.
Hadoop: was inspired by Google's MapReduce and Google File System (GFS) papers.Apache Cassandra: was developed at Facebook to power their Inbox Search feature (it is a columnar nosql DB)Cloudera: . It contributes to Hadoop and related Apache projects and provides a distribution for Hadoop for the enterpriseHive: DW on top of HadoopHbase: Nosql (columnar) – based on Googles BigtableVoldemort is a distributed data store that is designed as a key-value store used by LinkedIn for high-scalability storage (NoSql key value)
Business intelligence will be also extremely challenged: more info then ever, more complexity than before, and more operationality than before (less for off line BI, more participating to operational processes).BI is key to provide cohesive view of the world : assets, crews, outages, and take the right decisionsAll components will need to support this.
Qué pasa con los datos? va a existir muchísima necesidad de Integración de Datos, Distribución Vs Comercial, movimiento de ficheros, contingencia,…GG: Réplica casi en tiempo real, no intrusiva, heterogénea.
CEP for me focusing on criticalevent drivent near real time smart grids scenarios-Night Expend > 10 KW-average # of Elect. Power expended in the last 24 hours-Outage A, B and C occurred within 30 minute window
Technology in support of utilities challenges
<Insert Picture Here>Technology for Utilities Transformation and newChallenges: Integration, Big Data, ExtremePerformance.Aitor IbañezEnterprise Architect for Utilities
Contents <Insert Picture Here>• Utilities Business Transformation• Need for Technology• Extreme Performance• No-limit Integration• Applications: Architectural Challenges 2
Data Growth defines new Challenges… DATA GROWTH IMPACT IT CHALLENGES • Multiple Devices: • Agility for Business Meters, Apps, SCAD Processes on As,… Transactions between Devices and Applications • Info Processed by multiple • New Scalable Departments: Risk Architecture supporting Mgmt., Billing, Dema new Analysis and nd Transaction Growth Analysis, Network. Reqs. • Growth of Corporate • New Data Mgmt Reqs. DW for Reports and for Storage, Master Trend Analysis. Data, Synchronization & Sharing of Information Fuente: Accenture “Challenges for Utilities” 2010, Oracle Insight4 4
Challenges that arise new Biz & IT Needs Business Perspective IT Perspective External Pressure Strategy Levers Implications Managing efficiently Scalable growing Data Volumes Infrastructure Managing Data Explosion Improving Customer Business Processes without cost increments Care & Go-to-market Processes Automation Growing SalesGrowing Market Real-time Data Improving Availability Access, Load Share balancingDefense against Improving Competency Performance and On-demand security of growing, Business Cost Savings Transactions Continuity Reducing Operational Simplifying Storage Costs and improving Energy Efficiency and Servers Faster processing to improve Billing, Demand More Power Analysis, … Utilities need New Technology to face up New Challenges 5 5
… Needs translated into new Requirements • Scalable Infrastructure • Horizontal growing Performance • Business Processes Automation • Faster deployment of processes • Process Governance • Transactions • Real-time Data Access, Load Balancing Protecting • Fast response to data needs Investment • Grid computing • On-demand growing, Business continuity • Right-sizing, growing infrastructure as the Business demands • SLAs Compliance Scalability • Simplifying Storage and Servers • Reducing HW Installed base: best platform for every purpose • Energy Savings • More Power • Unprecedented Processing Power Manageability • Driving new Business opportunities6 6
Smart Grid Challenges Managing the Exponential Growth of Data Volumes Much more data… Much more processing… Break Point Cost System Cost inviable Processing Power required for Interval based billingSource: EPRI Scalable Scalability break points: System storage, scalability cost Meters 8
Integration beyond BoundariesResponding to events and information from avariety of sources 9
So… Need for Technology!• What do we need? Extreme Performance Engineered Massive Data Volume Systems Unstructured Data Analysis No-Limit Integration Data Integration Biz Processes Automation Services Architecture BPM, SOA, EDA Events Processing 10
Technologies in SupportExaMDM Concept: Technical Architecture 10 M Meters Benchmark • ½ Exadata + ½ Exalogic + Oracle MDM + CC&B • 500.000 bills 1 hour • Daily meter reads upload 2,6 hours • VEE processing 4 hours • Usage Aggregations 30 min. • Daily Total: 8 hours. • Scale: ½ Full ExaMDM = 4 hours 18
Oracle Exalytics – Under the Hood TimesTen for Exalytics 1 TB RAM Essbase 40 Processing Cores InfiniBand High Speed Networking Adaptive In-Memory Tools In-Memory In-Memory Oracle Business Intelligence Foundation Suite Analytics Software Analytics Hardware Speed-of-thought enterprise Extreme performance Business Intelligence platform via ExadataFor information purposes only, not a commitment to the development, release, and timing of any features or functionality. 19
Oracle ExalyticsExtreme Performance for Enterprise Analytics• Speed-of-Thought Analysis at Enterprise Scale • 20x better response times • Sub-second performance enables speed-of-thought analytics • Grow to 50,000 enterprise users on one appliance• What-if Analysis and Forecasting • 16x better response times • More accurate – use more detail, include more scope, evaluate wider operations • More scale – more users and higher volumes at busy periods • Greater agility – do daily instead of monthly• Enterprise Planning • Reduce planning cycle times with 5x better response time and 6x better throughput • Run more planning models with finer grained operational detail to improve accuracy • Extend beyond Finance to every LOB, with 10,000 Planning users on one appliance 20
Big Data for Smart Grid Analysis from unstructured data• Unstructured data coming from • Smart Meters & Devices, Domotic,… • Info coming from Network & Generation SCADAs • Substations, other equipments,… • Meteo Data.• BigData Appliance • 18 nodes, 216 CPU cores. • 864 GB RAM, 648 TB storage • 40 Gb/s Infiniband• Extreme Performance for Unstructured Data processing. 21
Big Data for Smart Grid Think about the Future…• Driving Immediate Analysis with unprecedented processing power • Social Networks: mentions [ourCompany] • Sentimental Analysis, Complaints • Social Networks: mentions #Meter, [ourCompany], #power Trending Topics. • Targeted programs for each consumer • Understanding Customer: CRM, Meter load, Internet logs, Twitter,… • Power Demand Planning: processing trends and actual consumption by geography/concentrator. • Prediction of Network failures or bottlenecks. • Geographical & Meteo Data 22
Divided Solution Spectrum Data VarietyUnstructured Distributed NoSQL File Systems Flexible MapReduce Specialized Solutions Developer Transaction (Key-Value) Centric Stores SQL DBMS ETL DBMS Advanced Trusted (DW) Analytics Secure Schema (OLTP) Administered Information Density Acquire Organize Analyze 23
Oracle’s Big Data Solution Data Variety Big Data ApplianceUnstructured HDFS• Hadoop Hadoop • NoSQL Database In-DB • Oracle Loader for hadoop Loader Analytics Oracle • Oracle Data Integrator Hadoop Oracle for “R” Exalytics NoSQL DB Mining • Speed of Text Thought Oracle Graph Analytics Data Integrator Spatial Oracle Exadata • OLTP & DW Oracle Oracle Database Database • Data Mining & Oracle R (DW) Oracle (OLTP) BI EE Schema • Semantics • Spatial Information Density Acquire Organize Analyze 24
Smart Metering Architecture DecisionsBig Data Business Intelligence SOA Framework Middleware & Bus. Orchestration Services AMI Self Service / portal Work & Asset Mobile meter Head-End NRT MDM CIS TBs Distribution management Outage Management 25
Oracle GoldenGate The Solution for Enterprise-wide Real Time Data Needs Database and Mission Criticalapplications, Mixed Applications &sources, distributed Real-time Data, Business systems, legacy, information Intelligence, Reporti OLTP, OLAP ng for Customers, Partner s & Employees Real-time Access Oracle GoldenGate delivers real-time access of real-time information, enabling companies to dramatically improve the availability, reliability, and performance of critical data across enterprise systems. 27
Oracle Data Integrator Data Integrator• Data Integration Solution E-LT Transformation Realtime Data Business Enterprise SOA | BAM | Hubs Intelligence Warehouse CEP• Declarative Design• No-intrusive• Change Data Capture Batch Data Flow Oracle Data Integrator Real Time Data• Architecture hot- Flow Services changes E-LT Events• Multi-DB support: CDC Knowledge modules Logs Transactional Logs Logs CDC CDC RDBMS 28
Supply Real Time Business IntelligenceNon-invasive Capture + E-LT ProcessingApplication Real Time BI Analytic BI (using Data Copy) (Facts & Dims) Consistency Window E-LT (Mini-Batch + Transforms) 29
Technologies in Support Oracle Integration Platform for Utilities AIA Foundation Pack for UtilitiesObjective: Reference Process ModelProvide a comprehensive, consistent, and flexibleintegration approach across the Utilities SuiteApproach: Industry Common Object Model & Services • Built on Oracle’s Fusion Middleware Standard s • Leveraging AIA Methodology • Expand AIA Foundation Pack for Utilities: –Common Objects and Enterprise Services –Compliant with utility industry standards –Deliver best-practice process models for key processes across utility segments (water, gas, electric, waste) Oracle Utilities Integrated Solution Suite - WAM MWM ML / AMI CC&B CRMBenefits: • Provide a common integration approach across SOA Suite, AIA (ESB + FP for CIM and MultiSpeak) all Oracle applications. • Accelerate deployments and provide flexibility NMS SCADA QM MDM GIS / Spatial Model for project-specific extensions. 31
Oracle Utilities Integration Strategy Oracle AIA Integrations Benefits $ Value Open Ecosystem Pre-built integrations AIA Centers of using leading Oracle Excellence Fusion middleware Extensibility Methodology AIA Partner Co- Lower TCO Loose coupling of apps development Industry best Supported products through SOA practices UGBU Partners to Designed for next Focus on Prescriptive certify with SAP + generation apps configurations not other 3rd Parties rebuilds methodology compatibility Successfully deployed Industry standards Reusability adoption such as in complex, high- volume environments CIM for Utilities Proven by Oracle 32
Business Process Management Processes Life Cycle Business Analyst Process Developer Process Modeling, simulating and Documentation Process Development Process and Systems Integration Execution and ManagementBusiness Owner Users Interaction Business Participants Corporate Applications Dashboards, tools and historical And Databases trends 33
Oracle Technologies in support Event-Driven Architecture Event Sources Response Services event Complex Event Streams ProcessorCoherence eventReal Time • Event sourcingListeners • Continuous query language to event define event BAM Dashboards • Correlation Applications • Sequencing event • Window of time evaluationDatabases • Pattern matching event • Event publishing BAM Alerts • Real time Java infrastructure BPM Processes BPM/BPEL Processes JMS closed loop Messaging 34
Business Impact of Events, Oracle BAM Meaningful, Event-driven Intelligence for End-Users Monitor business processes & services in real-time Key Performance Indicators (KPIs) Service-Level Agreements (SLAs) Analyze events as they occur Correlate events & KPIs Identify trends as they emerge Alert users to bottlenecks & solutions Act on current conditions Event-driven alerts Real-time dashboards BPEL processes & web services integration 35