Agile Operations - Xpdays France 2009

Agile and Operations for people with a good taste Gildas Le Nadan - Patrick Debois - Xpdays France 2009

Gildas Le Nadan Gildas comes from France

He has been working on servers for several years

Patrick Debois Patrick comes from Belgium

He works as freelancer always looking for good opportunities

We both come from sysadmin land

But we have been looking with great interest to the agile developer community

Finally we decided to give our own version of agile for operations

Today we have two stories for you

The ﬁrst one is about the impact of agile development on operations

The second one is what it means to do agile in Operations

So letʼs start with the ﬁrst great story. It starts off like a beautiful project

Once upon a time there was a team that had a lot of Agile Developers with good intentions

They had prepared all their frameworks and IDEʼs

They had all the tools they needed

They even had experts on Usability

They worked in close cooperation with the customer

They worked hard

Using their Post-its in abundance

They dutifully continued holding their standup meetings

They monitored their progress using a backlog

And this is the result after their ﬁrst sprint. A fully working product!

And the second sprint again, but still room for a lot of improvement

They worked and they worked, but after a while they realized ...

That they were adminstrating their systems in an adhoc manner and that they were not super sysadmins

So They called in a Sysadmin to ﬁx it

But It turned out their development environment ...

was quiet different from their test environment.

and nowhere near production environment!

They were still using manual installations

The sysadmins also cleanup up the development and test environment, providing a good base for further development

While doing so, Operations followed their ofﬁcial ITIL guidelines,

While they cleaned up things a few testers found some bugs

But overall most clients were happy with the quality

But still it felt that there was something missing.. It required more Tasting euh testing by a senior person

There was a real nasty bug that came up once in a while

But the problem was Hard to catch

Eventually they nailed it

Conﬁdent now, they decided to make their ﬁrst public release

There were some minor usability problems.

But they were easily solved by the project team with a temporary ﬁx

They created some workarounds

Delivering a great result

Inspired by the success Marketing wanted to put out lots of new features

But then Operations Team shouted NOOOOOO!!!!!

because in reality, things started to get ugly in production

First they had dealt with it, doing some emergency patches

But now things got REALLY ugly

Customers were experiencing bad response times

When they ﬁnally activated the logging , carefully checking the impact, the logs were full of useless debug messages

Eventually it appeared, that the usage mix was different: real users would use it synchronous way but the request by the APIʼs were asynchronous

So operations decided to put this in the FAQ

And they put on a ticket system

They installed larger servers

But the platform stayed fragile and required frequent restarts

One of the classic problems is that projects think they are the only one

But this project was not the only one operations had to support

They also had to document things, not loose any information

because, when the project was ﬁnished, some developers were assigned maintenance ﬁxes. At some moment, nobody of the original team was still there, and junior staff was trained to step in

The development team had the following view on the subject

Some people, probably the most senior, had a broader view of the platform operations was operating.

In reality it really looked this way

Ops giving speciﬁcations To avoid this kind of surprises in the future, they invited operations people during the design phase. This way they could transmit their knowledge of the productions environment and it was written down in the request for proposals.

Ops wrote down every requirement they could think of

But this Big Design UpFront resulted in and over complicated, overly designed and over engineered solutions.

The solutions seems to be to integrate operations IN the project phase. This both in the beginning and during the project. So both in good and bad times...

Because these people will constantly think about ... logs

They will check that sizing is done correctly

They will think of emergency procedures

Make sure Parallel Processing works

That your applications are packaged nicely

that your data can be archived and that the backup AND restore works

take the necessary security measures

think of good deployment tools

They will think about reporting. Find relations with other systems Think of reports management will request for SLA reporting

In the end everybody will be proud of what they prepared

And that includes the serving staff as well!

If you think now. Yeah but Iʼm in another business

You will always require some kind of log ﬁles

You will always need infrastructure

More good tools

Someone who needs to deal with angry customers

Good End User Manuals

The need for archiving

Cleanup Routines

dealing with capacity peeks

Monitoring the health of your systems

Some who takes care of supplies to keep your systems going

Hopefully you will see the light in the end

but off course , disasters can still happen!

Agile Manifesto Ok, the operations team needs to be agile, and it needs to be integrated in the project. How would the agile manifesto apply to YOUR work as an operations member

http://agilemanifesto.org We value the items on the left more then on the right

Individuals and interactions over processes and tools

ITIL vs. Agile ITIL has lot of practices for keeping things running. It used to be a change moderator, but as development is more agile we need to adapt. ITIL v3 is has introduced the notion of continuous improvement too.

Operations as a cost centre Increase in Maturity can bring Value (Gartner scale) 0 ADhoc 1 Reactive 2 Proactive 3 Service 4 Value

There’s no magic tool There’s no magic tool that can save you from bad organization. It still requires you to think!

Working software over comprehensive Documentation Working software over comprehensive Documentation Working means working in operation (Scope Problem , Dev) / Working Service

Customer Collaboration over Contract Negotiation Who’s the customer (Internal, External / Different ASP, Normal company, internal support)

Responding to change over following a plan Operations has been doing this for years. Every incident / issue requires us to react/ adapt things

Avoid the “Big Design UpFront”.

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software Our highest priority is to satisfy the customers: endusers but also developers What is early for the customer? 4d for a server, 2 min for a new account? What is value is for customer?

Risk Mgt DEV /Project = Creating value Loss of Value (protect value) = OPS

Welcome changing requirements even late in development. Agile processes harness change for the customer’s competitive advantage

Ops are often very resistive to change. Bussines might require constant adaptation.

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference for the shorter timescale Do things often so you get better at it.

Avoid Big bang migrations. Go in small steps.

Business people and developers must work together daily throughout the project

Have operations in your project and afterwards In good and bad times...

Build projects around motivated individuals. Give them the environment and the support they need and trust them to get the job done. Different environments : dev, test, prepod, training, trial, prod, qa Do you trust your developers to do deployment? Do you any secrets/super power they don’t have?

The most efﬁcient method of conveying information to and with a development team is face-to-face Don’t lock yourself in a small room with only email communication!

Working software is the primary measure It things work, and people are satisﬁed, you’re doing a good job!

Agile Processes promote sustainable development. The sponsors, developers, and users should be able to maintain constant pace indeﬁnitely Shared projects Specialists On call + daily job extend deployment power beyond ops team to spread the load

Continuous Attention to Technical Excellence and good design enhances agility Keep your skills sharp! You never know who is looking at you.

Scalability Thinks of Scalabilty

M A N A A G E B Y I T L I Manageability (start, stop subparts/ monitor progress)

Maintainability Maintenability = changed the text depending on the environment that changes

abi lity ec ur S Securability

Reliability Reliability

F l e x i b i l i t y Flexibility

Why is it important Ops has limited control over the elements they need to integrate or take care or

A Loose Coupling C B A’ C B A’ C D B E F G Noodle Soup Loose coupling

Butterﬂy Effect Butterﬂy effect

KPI and Monitoring KPI and Monitoring

Simplicity -- the art of maximizing the amount of work not done http://farm1.static.ﬂickr.com/78/168397680_01673102c2.jpg?v=0 Don’t go over engineer, Pragmatic

Simplicity Design Issues: Keep things Simple Stupid (KISS) Donʼt over cluster, loop networks, ...

Best Architectures, requirements and designs emerge from self-organizing teams. Use the tools you can adapt to your needs as you require them. Not because they have a good marketing.

Closed Software Closed Hardware Avoid Closed Source Software or Closed Appliances

Multiple Projects • One Product Owner? • = Program Manager Be clear on who is your customers. Your boss, project manager(s), tickets?

Incidents vs Projects Avoid being a Shared resource, pick the phone, take complaints, and new projects So you can commit to your work better.

Pair System Administration Operations decided to go for pair sysadminstration Project but also for incidents Learning , spreading the knowledge (vs. specialist / hero culture) http://www.ﬂickr.com/photos/mitikusa/2504868526/

Continuous Improvement • Burndown charts vs Qos • Target, no absolute/ Estimation Always try to improve yourself

Virtualized Hardware Go virtual on your hardware. Stop your emotions ;-)

Automated Deployment Automate things, that you don’t want to do over and over again. DRY: don’t repeat yourself

Conﬁg Mgt Version control your stuff, use tools like puppet, chef in stead of custom scripts

Doing Incremental Steps Work in small steps Changes in conﬁgurations: better traceability

Refactoring You need to correct mistakes.

Test Driven Administration Be sure that you can test/monitor what you need to have things working. Otherwise you are blind when changes happen.

Trend analyze for better prediction of when things will fail

Even if project ﬁnishes, environment will changes (patches, new hardware). So you need to able to test

Sometimes cleaning is easy. But if there are Legacy systems with lots of dependencies or no clear owner

Work together on your teams Continuous Integration system. You will learn a lot

So the next time youʼre celebrating a new project release

Maybe youʼll remember us

Make Operations Fun Again So that your operations team will be happy

More? agile-system-administration@googlegroups.com IEEE-paper: http://www.jedi.be/presentations/IEEE-Agile-Infrastructure.pdf patrick.debois@jedi.be gildas@endemics.info

Thanks you for listening!

Agile Operations - Xpdays France 2009

by Patrick Debois on Nov 27, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 9

Statistics

Agile Operations - Xpdays France 2009 — Presentation Transcript