815121835-Introduction-to-DevOps units.pdf

What is DevOps?
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor

Organizational structure
Online products are complex
Expensive to develop and maintain
Thousands of engineers
How to collaborate efficiently?

DevOps
DevOps is a combination of
methods,
tools, and
cultural behavior
that improves how software is developed and maintained.
It helps organizations deliver higher-quality online products faster.

Traditional Change Management
Independent teams
Different teams have different goals
Slow development
DevOps
Different teams work together
Software Development + IT Operations =
DevOps
Different teams have similar goals
High speed development

Traditional release

Minimum Viable Product
A minimum viable product (MVP) is an
early version of a product with limited
functionality.
Cheaper to build
High speed time-to-market

Product improvements

DevOps benefits

Let's practice!

Use Cases for
DevOps
Cem Sakarya
DevOps Risk Advisor

DevOps
DevOps: Software Development + IT
Operations
Collaborative and Shared Responsibilities
MVP Releases
Adaptable to various use cases

Infrastructure Engineering
Design, Develop, and Maintain the IT
infrastructure
This infrastructure requires power from the
cloud or from hardware the company owns
Infrastructure engineers take care of the
hardware, network, and cloud components

Product Engineering
Design, develop, and maintain the Software
Products
Customer serving components

Data Engineering
Data engineering refers to the building of systems to enable the collection and usage of
data.
https://en.wikipedia.org/wiki/Data_engineering
1

DataOps
Software is powered by data
Data is moved in data pipelines
Ensuring data moves smoothly
Data is not lost while moving
Move the data for the use of data scientists
and ML Engineers
MLOps
Predicting the future using the past data
Data Preparation (Historical data)
Model Training
Model Testing
Deployment
Maintenance

No competition

Project
Management
Methodologies for
DevOps
Cem Sakarya
DevOps Risk Advisor

Why is project management important to
DevOps?
Defines how the team will operate
Drives change
Timeline/Resource Management
Collaboration within/across teams
What is project management?
Project management is the use of
specific knowledge,
skills,
tools, and
techniques
to deliver something of value to people.

Waterfall
An old methodology
No return back to earlier stage
Develop all of the product at once

Agile
Series of cycles
Achieving one small goal at a time
First MVP, then improvements
Going through the cycle each time

Choosing the right project management model
The best methodology depends on:
Team's skill set
Budget
Complexity of the project
Expectations
Agile is the de facto standard for most
software development teams.

Scrum
A sprint in Scrum is a two-to-four week
timeframe with a light development goal for
the team.
Kanban
No sprints in Kanban, instead tracking
improvements continuously.

Scrum
Regular, fixed-length sprints (i.e., two
weeks)
Learn through experiences
Sprint planning, sprint, daily standup, sprint
review, sprint retrospective
Product owner, scrum master, development
team
Kanban
Continuous flow
Use visuals to improve work-in-progress
Visualize the flow of work, limit work-in-
progress, manage flow, incorporate
feedback loops
No defined roles
https://www.atlassian.com/agile/kanban/kanban-vs-scrum
1

Both Scrum and Kanban are under Agile methodology
Both Scrum and Kanban improves collaboration
Both powerful methodologies when applied to correct use case

The DevOps Change
Management Model
Cem Sakarya
DevOps Risk Advisor

Requirements
Developing a new product
Leadership: Sets business expectations
Product Experts: Define user experience
requirements

Design
Product Engineers: Designing the customer
experience
Infra Engineers: Designing the parts that
product engineers will use
Data Engineers: Designing the data
pipelines for the new product

Develop and test
Product Engineers
Develop for the customer experience
Customer interaction
Infra Engineers
Supporting the product
Providing testing environment
Data Engineers
Providing test data

Testing
Testing and developing at the same time
Testing before customer interaction
Otherwise poor user experience and
security issues

Experimentation
Limited set of users
Early user interactions
Observe if the users like the product
Real user feedback

Cyclical development

Main Software
Architecture
Systems
Cem Sakarya
DevOps Risk Advisor

Software architecture
Software architecture refers to the
fundamental structures of a software
system and the discipline of creating such
structures and systems. Each structure
comprises software elements, relations
among them, and properties of both
elements and relations.
https://en.wikipedia.org/wiki/Software_architecture
1

Application Programming Interfaces (APIs)
Data flows through APIs
They are essential for software and architecture

Various architectural systems
Factors to consider
Type of software
In machine or in the browser
Complexity of the software
Simpler software parts can be treated as a single unit
Separate the pieces for more complex software

Microservices
Most important architectural system for
complex software
Common in modern organizations
Rather than a big single unit, have many
smaller independent units
Smaller units are called the microservices
They communicate to each other via APIs

DevOps for microservices architecture

Benefits of DevOps for microservices
Software is changed constantly in large organizations
Change them one at a time
Microservices are more secure and reliable
If anything goes bad with one microservice, the rest will still function well

Modern IT
Infrastructure
Concepts
Cem Sakarya
DevOps Risk Advisor

Main components of IT infrastructure
Combination of various components
Customized for organizations specific
needs

Hardware
Can be hosted on organization's own
premises
Or it can be accessed through a cloud
service provider

Network
Communication with other parts of the
software
Communication with the users through the
Internet

Software
We need software to:
Manage the hardware and network
Configure them for the organization's use
The goal of infrastructure software:
enable the product teams to develop and
serve customers.

Change management
Software needs constant maintenance and improvement
These changes could be very expensive and disruptive to the services of the organization
Infrastructure Engineering provides the tools that's necessary to build products

Developer platform

Codebase & version control
Version Control: Git

DevOps CI/CD pipelines
DevOps in narrow meaning - CI/CD
Code needs to be built
Turn code file into machine executable file

Deployment platform

Change management in microservices

Elements of DevOps
Cem Sakarya
DevOps Risk Advisor

DevOps change management model
Continuous Integration & Continuous Delivery

Principles of CI
1. Version Control Software
2. Frequent integrations

Version control
Main branch as the source-of-truth for the codebase
Multiple teams working on the same software
Peer Review
Testing in every step

Principles of CD
1. Automated builds
2. Automated tests
3. Predictable and short change times

Experimentation
Limited set of users
Collect their feedback
Happens once a change is deployed

Feedback loops
Continuously improving the product
Use the review stage for the feedback loop
Turn the feedback into action
Change the software if necessary

Data Engineering
Implications of the
DevOps Architecture
Cem Sakarya
DevOps Risk Advisor

Microservices architecture
Microservices are small-scale software
programs
Microservices deployed separately
Each microservice take care of a different
functionality
Each microservice has its own data and
logic
They store the data in private databases

Monolithic architecture
Opposite of the microservices architecture
A big single unit
Much simpler compared to microservices
Maintenance and change is very hard and
risky

Monolithic Architecture
Limited number of databases
All application uses the same databases
Could be a viable option for small scale
applications
Microservices Architecture
A database for each microservice
Microservices must do API calls to reach
another services database
Microservices are effective in large
organizations and complex products

Microservices private databases
Some product functionalities require collaboration from multiple microservices
Microservices can not access each others databases freely, so they do API calls
Microservices always communicate with each other.

Data Engineering applications in microservices

Data Pipelines
Cem Sakarya
DevOps Risk Advisor

Data pipelines

ETL
https://www.istockphoto.com/nl/vector/data-extract-transform-load-gm1361894912-434102842?phrase=ETL
1

Batch processing
Move offline data in batches
Batch Processing: A large amount of data

Recap
Data Engineering is an integral part of
Infrastructure Engineering
Data pipelines are robust tools important
for handling data
Batch processing works on regular
schedules, stream processing works
continuously
Batch processing is useful for handling the
accumulated data
Stream processing handles the real-time
d t

Analytics &
Reporting with
DevOps
Cem Sakarya
DevOps Risk Advisor

DevOps benefits for reporting
Handling Multiple Data Sources in harmony
Producing many logs during the change management process

Reporting architecture For DevOps
DevOps produces a lot of data
Data sources are spread out
Data needs to be collected and moved to a database
Data Scientists can use the data for insights, reports, and dashboards

Tools: Ecosystem
Cem Sakarya
DevOps Risk Advisor

Requirements
Business and engineering interaction
Used throughout the software change
management process
Project Management Tools
Communication Tools
https://atlassian.design/foundations/logos https://slack.com/media-kit https://discord.com/branding
1 2 3

Version Control Software
Main change management technology is
git
Git is a protocol used for version control
Git ensures multiple developers can work
on the same software
Git has many implementations, most
common ones are GitHub and GitLab
https://git-scm.com/downloads/logos https://github.com/logos https://about.gitlab.com/press/press-kit/
1 2 3

Build Tools
Software needs to be built to become
executable
Main build tools are: Maven and Gradle
https://maven.apache.org/ https://gradle.com/brand/
1 2

CI/CD Tools
CI/CD pipelines are the main principles of
DevOps
CI/CD pipelines ensure automated building,
testing, and deployment of software
Main CI/CD tools are Jenkins and CircleCI
https://www.jenkins.io/press/ https://circleci.com/legal/trademark-guidelines/
1 2

Deployment
Microservices are developed and deployed
independently from each other
Containers imitate separate machines
Microservices are deployed on separate
containers
Containers: Docker and Podman
Container orchestration: Kubernetes
https://www.docker.com/company/newsroom/media-resources/ https://podman.io/ https://kubernetes.io/
1 2 3

Monitoring Tools
Products need to be closely monitored and
observed for quality and reliability issues
Monitor the DevOps health and change
management metrics
Example tools used for monitoring are
SignalFX and AppDynamics
https://www.splunk.com/en_us/products/observability.html?301=/en_us/devops.html
1

Data management tools: Kafka
Kafka is a message publishing system
Kafka is heavily used in microservices
architecture
Microservices keep a journal of the work
they do on Kafka
https://kafka.apache.org/trademark
1

Data pipeline management tools
Most tools used for both batch and
streaming processing
Main tools used for data pipeline
management are: Apache Airflow, Hevo
Data, and Prefect
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow%20logos https://design.hevodata.com/
https://www.prefect.io/newsroom/logos/
1 2 3

Ensuring Data
Quality Standards
with DevOps
Cem Sakarya
DevOps Risk Advisor

DevOps ensures good code
Most software handle data
Good software drives high-quality data
DevOps helps maintain high-quality in software

Data quality
How trusted is the information?
Most recent and accurate information
Can be hard and costly to achieve
Not all data requires the same quality
There are many elements to Data Quality

Elements of data quality
Elements breaks down and define Data
Quality
They should be traced and standardized

Is the data correct in all details?
Should be closely monitored

How comprehensive is the data?
Check if data is lost when it is stored in a
database
Check if data is lost when it is moved to
different databases

Consistency is about the reliability of the
information
It is also referred to as Data Integrity
Should be closely monitored and tested

Keeping and handling only the relevant
information
Irrelevant information wastes time and
resources

How up to date is the information?
Old data could be replaced by more recent
information
Raw data within microservices is not used
in data analysis
Microservices talk to each other via API
calls to get the most recent information
The data within microservices should be
replicated at an acceptable frequency

Observability and
Reliability
Cem Sakarya
DevOps Risk Advisor

Observability
Observing inner components of software
Components might consist of:
Microservices
APIs
Databases
Other components
Check they are working correctly.
https://www.istockphoto.com/nl/vector/het-lijnpictogram-van-de-verrekijker-oceaanconcept-verrekijkerteken-
op-witte-gm1245624425-363092800?phrase=field%20glass
1

Why is observability needed?
Many different parts tied to each other
Spot specific problems and their root cause
Monitoring systems effectively

Alerting
Alerts help detect problems early
Set alerts if software fails to run
Setting up alerts enable teams to react fast
when problems happen
https://www.istockphoto.com/nl/vector/notification-bells-icon-gm1364077374-435399123?phrase=alerting
1

Why is reliability important?
Examples:
Government services
Defense systems
Banking software
Many other software also require high
reliability

How to increase reliability?
Two main principles:
Testing
Observability
DevOps automates testing, and enforce
observability.

Testing
Testing is used to detect the problems before
they go to market.
DevOps ensures automated testing.
Observability
Observability is used to detect the problems
as they happen in user interaction.
Observability is needed even when the
software is tested enough.

Reliability in microservices

DevOps Cultural
Concepts
Cem Sakarya
DevOps Risk Advisor

Concepts
Collaboration
Autonomous Teams
Shared Responsibilities
Automation
Post-Mortem

Collaboration
Heart of DevOps
Collaboration between Development and Operations

Autonomous teams
Teams are diverse
Self sufficient teams
Teams have members who specialized in:
Product Engineering
Operations
Data Engineering, etc.

Shared responsibilities
Autonomous teams take care of specific
parts of the software
They are responsible for developing and
running
You build it, you run it

Automation
CI/CD pipelines helps automate change
management, testing, and deployment
Automation is a time-saver
Developers focus on creativity, not the
simple tasks
Without CI/CD pipelines, testing and
deployment must be done manually

Post-mortem
All software is prone to errors
Something can always go wrong
When a software problem is identified:
First solve it
Then review what went wrong
This review is called Post-Mortem
No one is blamed in the Post-Mortem

How this issue could have been
prevented?
Common Answers
More testing
More code review
How this issue could have been
better handled?
More observability helps with:
Earlier detection of issues
Earlier detection of the root cause

Tying it all together
Cem Sakarya
DevOps Risk Advisor

What is DevOps after all?
DevOps is a combination of:
tools
technologies
cultural elements
Software Development -> DevOps
Data Engineering -> DataOps
Machine Learning Engineering -> MLOps
It helps ensure high productivity, high quality,
and managed costs.

What is DevOps about?
Reorganizes into autonomous teams
Collaboration of teammates with different
backgrounds
Teams being responsible for different parts
of software
Automating change management
Testing
Experimentation

Why is DevOps necessary?
It is very efficient compared to other systems.
Automation and collaborations boost productivity
Testing and observability is necessary for reliability
Experimentation helps with product quality

When is DevOps applicable?
DevOps principles are applicable most of
the time.
It is necessary for large organizations.

What are the components of DevOps?

CONGRATS!
Cem Sakarya
DevOps Risk Advisor

What did we learn?
1. Introducing ourselves to DevOps and its
main components
2. Looking deeper into modern software
architecture and infrastructure
3. Exploring the data engineering
applications of DevOps and DevOps
reporting
4. Ensuring reliability and data quality

You can use your skills to...
navigate through software development
for software design, development, and testing
apply data engineering implications of DevOps

Time to celebrate!

815121835-Introduction-to-DevOps units.pdf

More Related Content

Similar to 815121835-Introduction-to-DevOps units.pdf

Recently uploaded

815121835-Introduction-to-DevOps units.pdf