Automating Splunk at Large Scale with Cloudify

•Download as PPTX, PDF•

0 likes•403 views

Slides for the webinar presented by Risk Focus on automating large scale Splunk deployments with Cloud Orchestration. More details available here: http://cloudify.co/webinar/Automating-Splunk-Large-Scale

Software

Agenda
About Us
Splunk Adoption Path & Challenges
Use Case Examples
Summary
2

Risk Focus
Expertise inTrading, Risk Management & CloudTransformation
Right-sized:
– technical organization big enough to deliver
– small enough to be efficient
Capital Markets specialists
TopTen Most Promising Cloud Banking Solution Providers of
2015 by Banking CIO Outlook Magazine
Cloudify partner with large, successful implementations
Splunk Premier partner with certified architects, consultants
3

Splunk Adoption Path
4
Log Aggregation
Operational
Intelligence
Automation and
Machine Learning

Challenges
5
Deployment
Normalization
andVisualization
Scalability

APIsMessaging
layer
Standard Capital Market business process
6
Transaction
Data
Logs
Risk / Margin,Valuation Engine
Market Data
Trade Capture
Client reporting
Post-trade
Reference Data Liquidity provider

Cloud Infra monitoring case study
Build a private cloud atTop 5 N.A. Bank:
– Hosting over 10,000VMs
– Tools to manage provisioning, rollout process
Splunk monitors cloud service layer, and allVMs.
Delivers system and application logs across entire fleet in every
environment.
Met retention requirements for ephemeral infrastructure.
7

Scaling Splunk
Segregated data to support information barriers
Role-based access to allow some users to search across tenants
Leverages administration benefits of common environment
Manage varying retention requirements
Scale to adapt to Splunk growth

Manage the Splunk SDLC
Splunk Infrastructure as CODE
– ProvisionVM
– Install Agents
– Bootstrap Splunk install
– Configure indexers, search-heads
Follow the same SDLC for Splunk as for other software

Multi-Tenant Splunk for a firm-wide data lake
A rich topology with
many different
components
Deployment
sequencing can be
complex

Multi-Tenant Splunk Management Console
11

Blueprint Overview
Number of nodes configurable
at deployment
“Surge” capacity or additional
tenants can be deployed in
similar manner at
scheduled/arbitrary time
Installs are automated with
minimal manual intervention

Cloudify and Ansible roles and responsibilities
13
Cloudify
• Interprets DSL
• ProvisionsVMs
• Defines
networking,
access policy and
permissions
• Sets up auto-
scaling policy
• Calls Ansible/SALT Ansible/SALT
• Installs
Application
Server’s packages
• Configures
Application Server
• Installs
Applications

How We Do It
Automation….
Infrastructure as code
Fleet configurable at deployment
Installs are automated with minimal manual intervention
“Surge” capacity or additional nodes can be deployed in a
similar manner

Summary
Splunk can solve….
We can help:
– Deploy
– Create Intelligence
– Scale
15

ThankYou
Subir Grewal, CFA
RiskFocus, Inc.
Subir.Grewal@riskfocus.com
splunk@riskfocus.com
www.riskfocus.com
16

Cloud, Cost, Complexity, and threat Coverage are top of mind for every security leader. The Lakehouse architecture has emerged in recent years to help address these concerns with a single unified architecture for all your threat data, analytics and AI in the cloud. In this talk, we will show how Lakehouse is essential for effective Cybersecurity and popular security use-cases. We will also share how Databricks empowers the security data scientist and analyst of the future and how this technology allows cyber data sets to be used to solve business problems.

Threat Detection and Response at Scale with Dominique Brezinski

Databricks

Security monitoring and threat response has diverse processing demands on large volumes of log and telemetry data. Processing requirements span from low-latency stream processing to interactive queries over months of data. To make things more challenging, we must keep the data accessible for a retention window measured in years. Having tackled this problem before in a massive-scale environment using Apache Spark, when it came time to do it again, there were a few things I knew worked and a few wrongs I wanted to right. We approached Databricks with a set of challenges to collaborate on: provide a stable and optimized platform for Unified Analytics that allows our team to focus on value delivery using streaming, SQL, graph, and ML; leverage decoupled storage and compute while delivering high performance over a broad set of workloads; use S3 notifications instead of list operations; remove Hive Metastore from the write path; and approach indexed response times for our more common search cases, without hard-to-scale index maintenance, over our entire retention window. This is about the fruit of that collaboration.

Talend introduction v1

Softnix Technology

Softnix Security Data Lake

Softnix Technology

November 2013 HUG: Cyber Security with HadoopYahoo Developer Network

02 a holistic approach to big dataRaul Chong

Recently, deep learning has delivered ground-breaking advances in many industries by delivering human-like understanding for difficult cognition problems. We will share our empirical experiences of applying deep learning to some real-world security challenges, together with leant lessons and suggestions. 1. Examples We are going to explain our innovative User & Entity Behavior Analytics (UEBA) solution which includes 2 deep learning examples: 1. user and entity behavior anomaly detection using Convolutional Neural Network (CNN), 2. stateful user risk scoring using Long Short Term Memory (LSTM), in order to detect slow-gestating and multi-stage targeted attacks. We are also going to share several real-life use cases of successfully detecting compromised users and malicious insiders in big enterprises. 2. Infrastructure The production data processing and analytics workflow is developed using Spark, Spark Streaming and TensorFlow. We will share the experience of managing and tuning distributed TensorFlow and Spark on a middle/small size cluster in both SAS and on-premises deployments. This includes how to manage and split resources between Spark and TensorFlow, how to split and tune workloads between parameter servers and worker servers in TensorFlow, etc. 3. Challenges and Guidance At the end, we are going to discuss the special challenges of applying deep learning (or general ML) into security than most other consumer industries, e.g., lack of large volume of high-quality labeled data, interpretation of models, fast detection, high cost of inaccurate detections. Human intelligence – including knowledge of both enterprise business context and security heuristics – is a very precious resource to help cover these gaps. Thus any effective security ML solution has to have well integrated human and machine intelligence. To achieve this partnership, there are several suggestions based on our current experiences, e.g., mix of complex and simple models, reinforcement learning based on human feed, pairing probabilistic ML results with deterministic forensic data.

How Virtual Reality and Machine Learning Are Powering the New Age of Network ...

DataStax

Introduction to Big Data

trendwiseanalytics1

Importance of Big Data Analytics

Impetus Technologies

Sqrrl Enterprise: Integrate, Explore, Analyze

Sqrrl

Splunk .conf2011: Search Language: IntermediateErin Sweeney

Building a Real-Time Security Application Using Log Data and Machine Learning...

Sri Ambati

Enterprise Data Hub: The Next Big Thing in Big Data

Cloudera, Inc.

If you missed Strata + Hadoop World, you missed quite a bit. This year's event was packed with Big Data practitioners across industries who shared their experiences and how they are driving new innovations like never before. Just because you weren't there, doesn't mean you missed out. In this session, we'll touch on a few of the key highlights from the show, including: Key trends in Big Data adoption The enterprise data hub How the enterprise data hub is used in practice

Building a future-proof cyber security platform with Apache Metron

DataWorks Summit

Qsight IT gives you insight in how we use Metron in securing our customers by continuously analyzing and monitoring users, applications, data, and networks. We show you how we implemented Metron as a replacement for our former security platform based on rule-based security. Since we are dealing with a non-conventional use case “serving many customers with one platform,” we developed a business classification module that enables us to score threats according to the customer’s input. To be future ready, we are working on extending this rule-based way of detection with machine learning models like web defacement, suspicious URL’s, UEBA, and many more to come. In order to provide all the necessary information to the SOC analysts at a glance, we are developing a custom SOC application from where they can handle security alarms, analyze captured data, and have historical data at hand. We regard our new Metron based Security Platform as an emerging giant—a future-proof cyber security platform! Speaker Bas van de Lustgraaf, Big Data Engineer, QSight IT Machiel van Tilborg, BI Engineer, QSight IT

Grace Hopper Open Source Day Findings | Thorn & Cloudera Cares

Cloudera, Inc.

Get full visibility and find hidden security issues

Elasticsearch

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Panel - Interactive Applic...

Data Con LA

In this interactive panel discussion, you will hear from these Spark experts as to why they chose to go "all-in" on Spark, leveraging the rich core capabilities that make Spark so exciting, and committing to significant IP that turns Spark into a world-class enterprise data preparation engine. Raymond and David will explain specific cases where capabilities were built on top of core Spark to provide a true interactive data prep application experience. Innovations such as creating a Domain Specific Language (DSL), an optimizing compiler, a persistent columnar caching layer, application specific Resilient Distributed Datasets (RDDs), on-line aggregation operators to solve the core memory, pipelining and shuffling obstacles to produce a highly interactive application with the core user and data volume scale-out benefits of Spark.

Xanadu Big Data Platform Technology BMT@ Rackspace Cloud

Alex G. Lee, Ph.D. Esq. CLP

Big Data Tech Stack

Abdullah Çetin ÇAVDAR

Case study: Hadoop as ELT for Leading US Retailer - Happiest Minds

Happiest Minds Technologies

Threat Hunting with Elastic at SpectorOps: Welcome to HELK

Elasticsearch

Gopalakrishna: big data consultant

Gopalakrishna Palem

Gopalakrishna, widely referred to as GK, is a Technology Management & Strategy Consultant specialized in Big data, M2M and Predictive Analytics. For more than a decade he is being actively engaged by companies like Microsoft, Oracle and Symphony on various tasks ranging from applying Machine Learning to predict component failure and machine availability, to designing RESTful Single-Sign-On (SSO) architectures for Oracle IDM Fusion Middleware, distributed CRM systems on Windows Azure for high-avaialbility, low-cost Render Farm solutions on Hadoop for high-scalability and so on. Contact Gopalakrishna at http://gopalakrishna.palem.in/

Data Science Driven Malware Detection

VMware Tanzu

Malware detection within enterprise networks is a critical component of an effective information security strategy. Instances of malware attacks are increasing – making them especially important to detect – and data science can help. This presentation outlines data science driven approaches to finding domains that have time and user-based co-occurrence relationships. It also includes a demonstration of a scalable and operationalizable framework to detect domain associations by analyzing the web traffic of users in any organization. Additional information: http://www.datasciencecentral.com/video/dsc-webinar-series-data-science-driven-approaches-to-malware

Advanced Analytics for Any Data at Real-Time Speed

danpotterdwch

Using Graphs to Enable National-Scale Analytics

Neo4j

Gluon Consulting - Specialized Software Development for Finance

Dennis Cabarroguis

Gluon offers specialized software development and consultancy services for fintechs and financial services SMEs. Our unique delivery model combines London based analysis, project management and QA with delivery from our team of talented developers in the Philippines. Results, service and care levels are those you would expect from the best front-office tech teams in London, New York and Silicon Valley. Costs? Refreshingly close to your average total outsourcing initiative, but without the risk and the hassle. OVER 40 YEARS OF EXPERIENCE IN LONDON Our core team members are veteran software consultants and enterprise architects with a combined 40 years of experience writing high-performance, robust and scalable software for some of the world's largest financial organizations and independent technology providers. TECH BEFORE FINTECH In our journeys, we built trading and portfolio analytics platforms, distributed and parallel calculation and workflow engines, high volume messaging middleware, dashboards with advanced interaction and visualization features. We did numerical and statistical computing, machine learning and semantic processing. We produced real-time, interactive scenario analysis for one of the most profitable trading desks on the planet. We helped e-commerce and brokerage companies to scale their operations without breaking a sweat. We contributed to pioneering robotics and field automation platforms before IoT and event streams were cool. For more info, please contact Dennis Cabarroguis dennis.cabarroguis@gluonconsulting.com

Losing Control to the CloudRochester Security Summit

What's hot

Deep Learning in Security - Examples, Infrastructure, Challenges, and Suggest...

DataWorks Summit

How Virtual Reality and Machine Learning Are Powering the New Age of Network ...

DataStax

Introduction to Big Data

trendwiseanalytics1

Importance of Big Data Analytics

Impetus Technologies

Sqrrl Enterprise: Integrate, Explore, Analyze

Sqrrl

Splunk .conf2011: Search Language: IntermediateErin Sweeney

Building a Real-Time Security Application Using Log Data and Machine Learning...

Sri Ambati

Enterprise Data Hub: The Next Big Thing in Big Data

Cloudera, Inc.

Building a future-proof cyber security platform with Apache Metron

DataWorks Summit

Grace Hopper Open Source Day Findings | Thorn & Cloudera Cares

Cloudera, Inc.

Get full visibility and find hidden security issues

Elasticsearch

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Panel - Interactive Applic...

Data Con LA

Xanadu Big Data Platform Technology BMT@ Rackspace Cloud

Alex G. Lee, Ph.D. Esq. CLP

Big Data Tech Stack

Abdullah Çetin ÇAVDAR

Case study: Hadoop as ELT for Leading US Retailer - Happiest Minds

Happiest Minds Technologies

Threat Hunting with Elastic at SpectorOps: Welcome to HELK

Elasticsearch

Gopalakrishna: big data consultant

Gopalakrishna Palem

Data Science Driven Malware Detection

VMware Tanzu

Advanced Analytics for Any Data at Real-Time Speed

danpotterdwch

Using Graphs to Enable National-Scale Analytics

Neo4j

What's hot (20)

Deep Learning in Security - Examples, Infrastructure, Challenges, and Suggest...

How Virtual Reality and Machine Learning Are Powering the New Age of Network ...

Introduction to Big Data

Importance of Big Data Analytics

Sqrrl Enterprise: Integrate, Explore, Analyze

Splunk .conf2011: Search Language: Intermediate

Building a Real-Time Security Application Using Log Data and Machine Learning...

Enterprise Data Hub: The Next Big Thing in Big Data

Building a future-proof cyber security platform with Apache Metron

Grace Hopper Open Source Day Findings | Thorn & Cloudera Cares

Get full visibility and find hidden security issues

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Panel - Interactive Applic...

Xanadu Big Data Platform Technology BMT@ Rackspace Cloud

Big Data Tech Stack

Case study: Hadoop as ELT for Leading US Retailer - Happiest Minds

Threat Hunting with Elastic at SpectorOps: Welcome to HELK

Gopalakrishna: big data consultant

Data Science Driven Malware Detection

Advanced Analytics for Any Data at Real-Time Speed

Using Graphs to Enable National-Scale Analytics

Similar to Automating Splunk at Large Scale with Cloudify

Gluon Consulting - Specialized Software Development for Finance

Dennis Cabarroguis

Losing Control to the CloudRochester Security Summit

Cloud cpmputing and busness processesMinka Fudulova

Logicalis Cloud Briefing

Logicalis Australia

Analyze your application portfolio to know where the quality and risk issues ...

Sogeti Nederland B.V.

How to develop a multi cloud strategy to accelerate digital transformation - ...

Senaka Ariyasinghe

Secure, Strengthen, Automate, and Scale Modern Workloads with Red Hat & NGINX

NGINX, Inc.

Learn how to support your application delivery – no matter where you are on the journey from monolithic apps to microservices. Join this webinar to learn: - About important considerations around digital innovation in FSI - How to leverage automation and Ansible to deliver apps faster - About keys to delivering modern apps securely and reliably anywhere - How OpenShift takes the complexity out of containers https://www.nginx.com/resources/webinars/secure-strengthen-automate-scale-modern-workloads-with-red-hat-nginx/

GigaSpaces XAP for Financial Services

Uri Cohen

Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...

Flink Forward

As many industries, banking is undergoing a fundamental change because of the software revolution. No longer are banks competing only on interest rates and having the best traders, these days customer experience and having the best engineers are the focus. In this changing world, banks compete with new start-ups, the so-called Fintechs, and with large platform organisations such as Google, Facebook and Apple. At ING, we believe that staying ahead of the game means changing how we interact with our customers, no longer a traditional model of waiting for the customers to come to the bank through our website or apps, but to actively reach out to the customer with information that is relevant to him or her in order to make their financial life frictionless. Many of these changes are driven by reacting to all events that are relevant to the customer, and using streaming analytics to be able to reach out to the customer in milliseconds after the event occurs. Apache Flink is key for ING to achieve this. This presentation addresses how ING approaches the challenge, the role that Apache Flink plays, and the consequences regulations have on how we work with Open Source in general, and with Apache Flink (and data Artisans) in particular. This keynote takes place at Kino 3.

ITAM UK 2017_Implementing a global award winning SAM program_Jochen Hagenlocher

Martin Thompson

Gain Deep Visibility into APIs and Integrations with Anypoint Monitoring

InfluxData

Assembling your cloud orchestra: A field guide to multi-cloud management

IBM

Operating in a multi-cloud environment is a reality for most organizations today. Whether it’s human resources recruiting candidates, manufacturing tracking shipments, or marketing enticing customers, business units are doing an end-run around their own IT departments to directly access services on the cloud. Instead of ignoring or attempting to stifle organizational forays into multiple clouds, IT needs to facilitate, orchestrate and optimize enterprise multi-cloud footing. Enterprises that assemble harmonized multi-cloud platforms now can magnify their business advantage while optimizing costs. But, it won’t be easy. Join the virtual event to hear how organizations are benefiting and mitigating challenges along with best practices for a successful multi-cloud management system.

Espion and SureSkills Presentation - Your Journey To A Secure Cloud

Google

Neev Application Performance Management Services

Neev Technologies

- Application Performance Management (APM) solutions manage the performance, capacity and availability of dynamic applications from the Cloud or a traditional data center. - APM aims to diagnose application performance issues to ensure that an expected level of service is maintained. - As part of this monitoring, two specific sets of parameters are closely tracked. - The first being performance metrics that define end user experience for an application, the second being metrics for computational resources used by the application for a specific load. - APM solutions not only monitor and analyze logs but also diagnose problems and assist in pro-active performance management. - APM is most commonly used for web applications where its components can also be individually monitored to pinpoint reasons for possible delays in the system. - Neev has partnered with APM solutions like AppDynamics and Splunk to offer them to our customers.

Should it be outsourced then?

Proact Netherlands B.V.

Quest Software - Dan FalconerIDG Romania

Advance Your IoT Strategy with Integration of PLM to Oracle Product Hub

Jared Pettit

[Agile PLM users] Data silos are silent productivity killers. Integrating PLM to up and downstream applications via a centralized hub is key to advancing insights, improving decisions and assuring data harmony across the extended enterprise. In these slides, which come from a Domain Systems webinar (with the same title) we explore the tools and best practices involved in driving an IIoT strategy forward.

CloudOps with OpsRamp: From Discovery to Resolution

OpsRamp

Are you able to observe, monitor and act upon cloud assets - wherever they are? In this Tech Talk, we'll discuss the role of cloud operations and review OpsRamp's unique capabilities for hybrid infrastructure discovery, monitoring, service maps, remote consoles and remediation. This Tech Talk will showcase the collective value of a unified IT operations management platform, reviewing multiple OpsRamp features and tying them to the most common CloudOps workflows. Watch the recording here: https://www.brighttalk.com/webcast/17791/424253 Learn more at https://www.opsramp.com Also, follow us on social media channels to learn about product highlights, news, announcements, events, conferences and more: Twitter - https://www.twitter.com/OpsRamp LinkedIn - https://www.linkedin.com/company/opsramp Facebook - https://www.facebook.com/OpsRampHQ/

CRTC Cloud- Scott Sadler

KrisValerio

ORbyte Trade 2.0 - Cloud Crowd 2009

Francis Otshudi

Similar to Automating Splunk at Large Scale with Cloudify (20)

Gluon Consulting - Specialized Software Development for Finance

Losing Control to the Cloud

Cloud cpmputing and busness processes

Logicalis Cloud Briefing

Analyze your application portfolio to know where the quality and risk issues ...

How to develop a multi cloud strategy to accelerate digital transformation - ...

Secure, Strengthen, Automate, and Scale Modern Workloads with Red Hat & NGINX

GigaSpaces XAP for Financial Services

Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...

ITAM UK 2017_Implementing a global award winning SAM program_Jochen Hagenlocher

Gain Deep Visibility into APIs and Integrations with Anypoint Monitoring

Assembling your cloud orchestra: A field guide to multi-cloud management

Espion and SureSkills Presentation - Your Journey To A Secure Cloud

Neev Application Performance Management Services

Should it be outsourced then?

Quest Software - Dan Falconer

Advance Your IoT Strategy with Integration of PLM to Oracle Product Hub

CloudOps with OpsRamp: From Discovery to Resolution

CRTC Cloud- Scott Sadler

ORbyte Trade 2.0 - Cloud Crowd 2009

More from Cloudify Community

Webinar: Dealing with automation tool overload!

Automating Splunk at Large Scale with Cloudify

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Automating Splunk at Large Scale with Cloudify

Similar to Automating Splunk at Large Scale with Cloudify (20)

More from Cloudify Community

More from Cloudify Community (20)

Recently uploaded

Recently uploaded (20)

Automating Splunk at Large Scale with Cloudify

Editor's Notes