Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CWIN17 India / Insights platform architecture v1 0 virtual - subhadeep dutta

350 views

Published on

Cloud architecture & Platforms

  • Be the first to comment

CWIN17 India / Insights platform architecture v1 0 virtual - subhadeep dutta

  1. 1. Cloud Architectures & Platforms CWIN17, September 27th 2017 Presenter : SHUVADEEP DUTTA, Insights Platform, Insights & Data
  2. 2. 2Copyright © Capgemini 2017 All Rights Reserved 2Copyright © Capgemini 2017. All Rights Reserved AGENDA Platform Architecture Views Cloud Platform Security Platform Logging Platform DevOps
  3. 3. 3Copyright © Capgemini 2017 All Rights Reserved 3Copyright © Capgemini 2017. All Rights Reserved Future Looking Data Lake – Conceptual Architecture
  4. 4. 4Copyright © Capgemini 2017 All Rights Reserved 4Copyright © Capgemini 2017. All Rights Reserved Big Data and Analytics Platform Logical Architecture – Cloud & Technology Agnostic View
  5. 5. 5Copyright © Capgemini 2017 All Rights Reserved 5Copyright © Capgemini 2017. All Rights Reserved Big Data and Analytics Cloud Platform – MS Azure / AWS Native Services
  6. 6. 6Copyright © Capgemini 2017 All Rights Reserved 6Copyright © Capgemini 2017. All Rights Reserved Big Data and Analytics Cloud Platform – Custom Built Stack (Opensource Hortonworks)
  7. 7. 7Copyright © Capgemini 2017 All Rights Reserved 7Copyright © Capgemini 2017. All Rights Reserved Cloud Deployment Considerations Data topology, governance & security capabilities need to be evaluated taking into consideration the key considerations to define a target state hybrid cloud platform architecture Focus Area Consideration Data Location Location of data storage in alignment with regional regulatory compliance directives Analytics Use Cases Type of analytical workloads going to be executed on the data repository located at a specific region / country Data Access, Authentication and Authorization Mechanism of data and underlying platform resource access based on specific user role, user location, time of access etc. Network Latency Volume of data transfer over network ensuring appropriate bandwidth and SLAs being met Data Ingestion Type of data ingestion mechanisms used to transfer data from on-premise to cloud and vice-versa Security and Regulatory Compliance Implementation of controls and mechanisms to meet platform security (perimeter, data and application), regional regulatory compliance directives such as PCI, SOX etc. Platform Governance Platform wide metadata management, audit logging, master and reference data management capabilities; considerations for bottom-up / top-down /Hybrid approaches: Data Catalog driven data discovery and knowledge sharing (bottom-up) vs. Data Governance Council (Data Stewardship) driven information asset management and knowledge sharing (Top-down) Platform Infrastructure Considerations for High Availability, Fault Tolerant and Disaster Recovery capabilities Considerations from platform performance and scalability based on specific Big Data Analytics workload use cases Considerations for Platform resource management and orchestration Data and Application Portability Considerations for data and application portability across different platform environments e.g. On-premise, Private, Public cloud
  8. 8. The information contained in this presentation is proprietary. Copyright © 2016 Capgemini. All rights reserved. Rightshore® is a trademark belonging to Capgemini. www.capgemini.com 8Copyright © Capgemini 2017. All Rights Reserved Capgemini’s DaaS (Data as a Service) Solution Framework Meet all data and analytics management service needs from data ingestion, preparation, discovery, through till data analysis using opensource or commercial tools Leverage client’s current investments and integrate with technology of client choice or extend / customize existing framework capabilities
  9. 9. 9Copyright © Capgemini 2017 All Rights Reserved Cloud Platform Security
  10. 10. 10Copyright © Capgemini 2017 All Rights Reserved 10Copyright © Capgemini 2017. All Rights Reserved Big Data and Analytics Platform – Cloud Security Framework Physical Asset Security Management Protection for physical assets and locations including networks and data centers Cloud Governance Cloud specific security governance including directory synchronization and geo locational support Information Asset Protection Protection of data at rest or in transit Governance & Compliance Security governance, maintenance of security policy, audit and compliance Threats and Vulnerability Management Management of vulnerabilities and mitigations with Network and endpoint protection Incident Management Management and responding to expected and unexpected events Identity and Access Management Authentication of users and management of identity
  11. 11. 11Copyright © Capgemini 2017 All Rights Reserved 11Copyright © Capgemini 2017. All Rights Reserved Cloud Security Capability Framework – Shared Responsibility Model
  12. 12. 12Copyright © Capgemini 2017 All Rights Reserved 12Copyright © Capgemini 2017. All Rights Reserved AWS VPC Security Reference Architecture VPC Security Architecture Scenarios Scenario 1 : VPC with a Single Public Subnet only • Instances run in a private, isolated section of the AWS cloud with direct access to the Internet • Network ACLs and security groups can be used to provide strict control over inbound and outbound network traffic to EC2 instances Scenario 2 : VPC with Public and Private Subnets and NAT • In addition to public subnet, a private subnet is added whose instances are not addressable from the Internet • Instances in the private subnet can establish outbound connections to the Internet via the public subnet using Network Address Translation (NAT Gateway or Instance) Scenario 3 : VPC with Public and Private Subnets and hardware VPN access • IPsec VPN connection between Amazon VPC and customer data center, while also providing direct access to the Internet for public subnet instances in Amazon VPC • VPN appliance on customer corporate data center side Scenario 4 : VPC with Private Subnet only and hardware VPN access • Instances run in a private, isolated section of the AWS cloud with a private subnet whose instances are not addressable from the Internet • Private subnet is connected to customer data center via an IPsec VPN tunnel
  13. 13. 13Copyright © Capgemini 2017 All Rights Reserved 13Copyright © Capgemini 2017. All Rights Reserved AWS VPC Security Reference Architecture – Security Groups  VPC security groups to firewall each EC2 instance  Each instance can be in up to 5 security groups  Separate security groups for applications and management  Security groups are stateful with ingress and egress rules  Max. 50 rules per security group  VPC Router will allow any subnet to route to another in VPC  Network Access Control Lists are used to restrict internal VPC traffic  Elastic load balancers are used to distribute traffic between instances  Elastic load balancers are also placed in security groups  Platform security can scale up and down with solution; instances can be added into security groups during launch time  Use NAT instances to provide internet connectivity for Private Subnets; allow backend servers to route to AWS APIs – Ex. storing logs in S3 bucket or using DynamoDB, SES  Access AWS API endpoints through the Internet Gateway like S3, SES, DynamoDB, SNS etc.
  14. 14. 14Copyright © Capgemini 2017 All Rights Reserved 14Copyright © Capgemini 2017. All Rights Reserved AWS Identity and Access Management  Securely control access to AWS services and resources  Fine grained control of user permissions, resources and actions  Support for RunInstances  Multi Factor Authentication – Hardware token or Smartphone Apps  Segregation of roles using IAM AWS Account Owner (Master) Network Management Security Management Server Management Storage Management
  15. 15. 15Copyright © Capgemini 2017 All Rights Reserved 15Copyright © Capgemini 2017. All Rights Reserved AWS Identity and Access Management Role Based Security
  16. 16. 16Copyright © Capgemini 2017 All Rights Reserved 16Copyright © Capgemini 2017. All Rights Reserved AWS Identity and Access Management using Enterprise’ existing Active Directory Flow 1 2 3 4 The enterprise user accesses the identity broker application The identity broker application authenticates the users against the corporate identity store The identity broker application has permissions to access the AWS Security Token Service (STS) to request temporary security credentials Enterprise users get a temporary URL that gives them access to the AWS APIs or the Management Console AWS Identity Federation with Temporary Security Credentials
  17. 17. 17Copyright © Capgemini 2017 All Rights Reserved 17Copyright © Capgemini 2017. All Rights Reserved AWS Data Storage Security – Capabilities  AWS S3 Capabilities –  Access controls at bucket and object level  Cryptographic capabilities such as SSL for data in motion, Server/Client side encryption, MD5 checksums  AWS Redshift Capabilities –  Full disk encryption; CloudHSM to store keys  Back-up access logs to S3 for analysis  Security groups and VPC for deployment; data loading using SSL from S3 with restricted access to S3  SSL encryption for data accessed over internet  AWS RDS Capabilities –  Restricted access to RDS instances using Security groups and IAM permissions  Data encryption (Data at rest and in motion)  Automatic patching for minor updates  AWS DynamoDB Capabilities –  Fine grained security access to columns and rows using IAM role and access policies  AWS EBS Volume Capabilities –  Option to use own encryption or commercial solutions Ex. Windows BitLocker or Linux LUKS for encrypted volumes and TrueCrypt for containers; Commercial : Safenet Protect-V, Trend Secure Cloud etc.
  18. 18. 18Copyright © Capgemini 2017 All Rights Reserved 18Copyright © Capgemini 2017. All Rights Reserved Securing AWS Applications Process Flow
  19. 19. 19Copyright © Capgemini 2017 All Rights Reserved Platform Logging
  20. 20. 20Copyright © Capgemini 2017 All Rights Reserved 20Copyright © Capgemini 2017. All Rights Reserved Platform Monitoring – Centralized Troubleshooting, Security, Audit and Monitoring
  21. 21. 21Copyright © Capgemini 2017 All Rights Reserved 21Copyright © Capgemini 2017. All Rights Reserved Platform Monitoring – Solution Options Shippers Queue Logstash Elasticsearch Elasticsearch Kibana Log Parsing Indexing & Curation Dashboards & ReportsRedis, RabbirMQ, KafkaSyslog, Rsyslog, Logstash, Fluentd etc. JSONEvents Extract Option 1 – Custom Built using ELK stack (Elasticsearch, Logstash and Kibana) Option 2 – SaaS Solution – Loggly (alternate leading tool : Sumologic) Shippers Syslog, Rsyslog, Logstash, Fluentd, cloud plug-ins etc. Application Framework INGEST PROCESS INDEX Search & Other Services • Cloud-based SaaS for easy central log collection, aggregation, management • Easy set-up • Dynamic parsing - Real-time, JSON support, parsing/tagging, self-documenting • Regular Expressions based • Dashboards, pre-configured and customizable, shareable • Anomaly Detection • Alerts that can be sent to HipChat, Slack, PagerDuty, HTTP endpoints, others • JIRA Software integration, point-and click ticket creation without leaving Loggly
  22. 22. 22Copyright © Capgemini 2017 All Rights Reserved Platform DevOps
  23. 23. 23Copyright © Capgemini 2017 All Rights Reserved 23Copyright © Capgemini 2017. All Rights Reserved DevOps Reference Architecture for Big Data Analytics Workloads
  24. 24. The information contained in this presentation is proprietary. Copyright © 2015 Capgemini. All rights reserved. Rightshore® is a trademark belonging to Capgemini. www.capgemini.com About Capgemini With more than 145,000 people in over 40 countries, Capgemini is one of the world's foremost providers of consulting, technology and outsourcing services. The Group reported 2014 global revenues of EUR 10.573 billion. Together with its clients, Capgemini creates and delivers business and technology solutions that fit their needs and drive the results they want. A deeply multicultural organization, Capgemini has developed its own way of working, the Collaborative Business Experience™, and draws on Rightshore®, its worldwide delivery model Learn more about us at www.capgemini.com.

×