Sky TV and BMC have been working together for almost 20 years. BMC has supported Sky TV to operate the most comprehensive multi-channel, multi-platform television service across the UK and Ireland. Sky TV has recently lifted the lid on the Control-M black box process by rolling out Self-Service to our customers; the demand has been very encouraging and overwhelming. We named our project “Sunrise – The New Dawn of Batch Processing,” and we would like to share our experience with you.
Jack
HOD IT Services at Sky
Based in Scotland / LondonResponsible for front line services
In IT for 30 years / 12 at Sky
JD Managerial / SA Technical
Q&A throughout but preferred at end.
Stevie
Batch Operations Manager at Sky TV
Been at Sky for over 17 years
Based in Scotland
Have worked with Control-M for the past 16years.
Control-M experience has progressed from first line support using the Control-M GUI to a scheduling role to the administration of the Control-M platform at Sky
Jack - Explain that we are here to talk about the significant role that Control-M has played in the success of the Sky Entertainment business, and the journey between BMC and Sky over the past 16 years.
Jack
Let me throw a throw a few pieces of information to help you paint a picture and set the scene
Sky is Britain and Ireland’s leading home entertainment and communications provider.
Providing entertainment to approximately 40% of homes since it was launched 25 years ago, way back in 1989
The Sky ethos from the outset was the belief that people wanted a better choice of television content
This audience may find this hard to understand but prior to Sky’s launch we had just 4 TV channels in Britain and just 2 in Ireland!
Due to links with Sky’s parent company, 21st Century Fox (formerly News International) we brought popular American shows like the Simpsons to the UK for the first time.
Sky expanded into the Broadband and Telephony market in 2006 and very quickly went from zero to the 2nd largest broadband provider in Britain
Our product range includes Video on Demand, Sky Go, Sky+ (TIVO), Broadband, Telephony, High Definition, 3D TV, Sky Atlantic, Sky Sports, Sky Movies, Sky Arts
We also believe the way we do business is as important as what we do. Our ‘Believe in better’ ethos means doing the right thing and taking responsibility for our actions day to day.
Our latest acquisition is in central Europe with the full takeover of our sister companies in Italy and Germany.
This is still subject to regulatory approval and will create bigger headroom for growth.
Sky Promo Video – TV Products Sizzle v3_14_05_14.mp4 - 1min 18s 108Mb
Stevie
Control-M has been part of the landscape at Sky for over 16 years, so we thought that it would be useful to spend some time looking at the Control-M journey that Sky and BMC has had together during that time.
Stevie
Take cues from slide
1998 –
Control-M was introduced at Sky in 1998.
This pre dates BMC as Control-M was owned by New Dimension Software, as this was prior to BMC’s acquisition of New Dimension in 1999.
At the time Sky was venturing into open systems infrastructure
Control-M was introduced to manage batch processing on our emerging Solaris estate on our Customer Management Platform.
Small estate ~250 batch jobs across 10 clients, using Control-M 224 and ECS 500.
In 1999 we implemented Control-M on our mainframe. Prior to this we were keying in jobs manually at the console! Control-M made life a lot easier for the Ops Team.
2004 –
Solaris estate had grown to a few hundred servers with around 2000 jobs being managed daily.
Decided to take the plunge and upgrade to Control-M 6.1.01.
Jack – Step in and ask what the drivers were for upgrading to 6.1.01 – other than support issues
Stevie –
Around the time of the upgrade Sky was working on a CRM programme which saw a transformation from fairly linear, simple workload automation solutions, to complex, interdependent automation workflows across multiple applications.
This was truly the beginning of an new era for us in workload automation at Sky and BMC Control-M was the perfect tool for the job.
2006 –
Implemented our first plug in – Advanced File Transfer.
This was a revelation for our Technology dev ops teams as up until that point they had the burden of scripted solutions for file transfers.
This time also coincided with increasing move towards secure transfer protocols such as SFTP. AFT handled this with great ease.
Today AFT is used to manage all financial data transfers in and out of Sky, as well as a huge range of other data transfers.
Our auditors love this as they have confidence in AFT with its built in security, its robustness and logging.
2011 –
By 2011 we were running around 11k jobs per day with around 40k submissions.
Our 6.1.01 platform was beginning to show signs of age, with the hardware reaching capacity and end of life so we decided to upgrade to V7.0.
Intermediate upgrade to Control-M 6.2 required before the migration to V7.0.
We learned a lot of lessons with the V7.0 migration as there were many bespoke solutions such as in-house API’s that required to be rewritten.
We also moved from Sybase to the Postgress DB solution which required a rewrite of many of our Control-M housekeeping and reporting scripts.
2013 –
In 2013 we changed our pricing model with BMC which enabled us to implement Control-M Forecast and it also opened up a new world of plugins, such as Business Objects, which we implemented to great effect. We’ll go into more detail about Business Objects later in one of our case studies.
2014 –
This brings us on to 2014, which has undoubtedly been the most exciting year in our Control-M journey at Sky.
First of all we implemented V8.0 in our training and sandbox environments, this has allowed us to have familiarization sessions with users, allowing them to get a real feel for the new look GUI . The feedback has been overwhelmingly positive.
2014 –
2014 also saw the implementation of Self Service and Batch Impact Manager.
We will talk more about this shortly but suffice to say we consider this to be the most eexciting development in our Control-M journey at Sky in terms of customer engagement and satisfaction.
2014 –
This year we have also seen the implementation of the Hadoop plugin and the Business Process Integration Suite (latterly called Control-M for Web Services, Java and Messaging) which has allowed our MQ developers to make message calls directly from Control-M.
This has been of great benefit to our Lightweight Services Team who is leading the way with the biggest transformation of our customer management platform at Sky.
Stevie - take cues from slide.
Let me give you a little insight into what Control-M looks like today at Sky
1)
Today we manage 6 Control-M servers, 3 Enterprise Managers and just over 1.5k agents.
Covers production, pre-prod, Dev/Test, Sandbox and training envs
Our production Control-M servers runs V7.0 fix pack 4.
EM prod is V7.0 fix pack 6
Training/sandbox run V8.0, fix pack 2
The Production platform runs on physical LINUX Red Hat servers, spread across two datacentres in Edinburgh, Scotland.
HP DL380 G7. 24GB memory. processing power two physical CPU sockets per physical machine each with 8 cores.
High availability in production is managed by VCS
In non-prod we use the Control-M DB mirroring functionality
In the last year our uptime in production was 99.96% with planned failovers so quick and seamless that none of our customers even notice
Our agent server estate is predominantly a mixture of Solaris and LINUX, with a small number of Window Servers.
2)
Control-M manages workload automation services on > 70 applications across a wide range of business functions.
The next slide will look at some examples of the business processes and technologies.
3)
There are approx. 180k job submissions each day across all environments.
4)
Managed by a team of 20 multi-skilled people who provide 24*7 onsite cover and are responsible for everything from first line support of workload automation services, to scheduling to configuration and admin of the Control-M platform. The team also delivers infrastructure monitoring services, backup services, system failovers, code deployments and system maintenance as well as the out of hours ITSD service.
Jack
This slide demonstrates some of the business functions and technologies that we use Control-M to manage at Sky. It covers a very diverse range of business areas and technologies, touching on almost all aspect of our business at Sky.
I’ll pick out a few of the key business functions –
Executive level reporting runs every night at through Control-M at Sky with the output going to our Exec every Friday morning. At key points in the financial year the data from these reports gets fed into Sky’s financial reports, which ultimately determine the value of the company.
All financial data transfers in and out of Sky to Banks and other financial institutions run through Control-M (using the Control-M AFT module)
Broadcast platform delivers our content with processes on our backend Broadcast Support System being managed through Control-M.
Control-M is used to manage workload automation across our entire Customer Management platform.
Stevie
Some of the Technologies (read from left to right on slide)
A lot of our legacy applications predate Control-M plugins and rely on wrapper scripts to call processes such as java, Oracle and Netbackup.
Over the past few years we have seen an increasing use of the Control-M plugins to manage Business Objects, AFT and MQ processes. There is more to come with a proof of concept of Oracle DB queries using the Control-M DB plugin.
Big Data has become a big part of the Technology landscape over the past couple of years. There is a big data project ongoing at Sky, which is currently implementing the Control-M plugin for Hadoop, which we will discuss in more detail later on.
Have heard workload automation referred to as “the connective tissue” that binds Technology together.
Jack – lead in to section
At this point we’d like to share with you the details of what we consider to be the most exciting development in our Control-M journey so far at Sky
The implementation of Control-M Self Service has transformed the way that our customers interact with workload automation services and has consequently brought us closer to our customers.
The rollout of Self Service enthused and invigorated our customers; this has greatly enhanced the workload automation piece and the reputation of the Operations Team at Sky.
Stevie - take cues from slide
1)
Over the past few years our customers were increasingly asking for visibility of their batch services
2)
Control-M was a “black box” – customers seen inputs and outputs but not the “bit in the middle” which was the preserve of the Operations Team (refer to graphic on slide – expand on input/outputs)
3)
Customers spent a lot of time contacting the Operations Team in order to get real time and historical information about their batch services
Control-M Reporting Facility was good for repeatable queries but ad-hoc queries had to be performed manually.
4)
Bespoke in-house solutions - dashboards were good but each area had unique requirements and this had a significant dev/support overhead. We also had an issue when a DB query from a bespoke solution hung the Control-M production DB during the new day procedure. This led to a delay in the delivery of Exec reporting at quarter end.
GUI client access for customers – suitable in some cases but had training overhead. Dashboard/service view that customers wanted is not available natively.
Jack
Implemented Control-M Self Service at Sky in April 2014.
Also implemented Control-M Batch Impact Manager at the same time
We decided to spend a 6 week period, up the end of May, to bed the solution in and get to know its capabilities before rolling out to customers.
This was an invaluable experience as it allowed us to set up BIM and service views and to run them without the visibility of the wider community.
By the time we went out to our customers we already had most of their services set up and bedded in and we had a great understanding of the nuances of the tools.
We chose one of our closest customers in Technology for a pilot scheme.
They were able to use Self Service from day 1, the feedback was phenomenal. Once they were satisfied we embarked on the rollout of phase I.
Phase I was the rollout to areas where batch services were already running in Control-M.
This covered around 30 business and Technology teams at Sky.
It was a very easy sell with the feedback overwhelmingly positive, people wanted Self Service.
Phase I was complete by the end of August 2014. We then embarked on phase II.
This involves breaking into business areas at Sky that currently do not use Control-M.
Stevie
Key points
Control-M not in Dev/Test
FDD issue
Self Service addressing
Self Service allowing move to new areas of business
Migration from Chronicle, Jenkins etc
Over the past few years we have encountered issues at Sky due to the fact Control-M is only used on some applications in the dev/test environment.
For many applications the first time that we run new or amended processes through Control-M is at the non-functional test stage of our code release cycle.
On occasion this has led to problems for our customers when we have encountered issues running jobs in Control-M that have defects due to the Control-M environment.
A recent example involved a Control-M API which called the ctmcreate utility. By the time we ran the process through Control-M in NFT we did not have time to fix the defect before the go-live date. The end result was that implementation of the solution had to be delayed and our customers in Finance were understandably not impressed.
This is an area that Self Service is helping to address. As part of Phase II we are implementing Control-M across many areas of dev/test and setting up views in Self Service for the dev and test teams to manage their services using the “Order Service” function in Self Service.
Feedback from the test teams has been overwhelmingly positive – they love Self Service!
Self Service has also allowed us to move out to areas of the business at Sky who are using scheduling tools other that Control-M.
Later on we will look at a couple of these initiatives in closer detail in our case studies section, but to summarise these experiences, we have won them over and we are in the process of migrating schedules from cron, Windows Task Manager, Jenkins and Chronicle into Control-M.
Jack
1)
A wide range of our customers, from IT Analysts, Business Analysts to Directors are using Self Service on a daily basis to stay informed about the status of their batch services.
There has been a large uptake in customers using the iOS and Android Apps while away from the office.
Examples of this are –
Our Director of Finance and Directors of the Customer Business Platforms use the Control-M Self Service app on their iPads / iphones to check the status of the customer billing service. This is particularly important at quarter and year end processing. Prior to Self Service they used to rely on hourly Billing / SLA status report sent out by Ops.
Some of our Technology Support teams are using the iOS and Android app for Self Service to check the status of their overnight batch runs when they get out of bed in the morning. With Self Service they know exactly what is going on when they arrive at the office and their boss is asking for an update. If the service is running late they also use the info from the BIM predicted end time. Previously this information would be obtained via early morning telephone calls to the Ops Team. Prior to the BIM implementation predicting end times for a service was a manual process which was time consuming and very often inaccurate. We eliminated the guess work.
Stevie
Take cues from slide
2)
There has also been a significant reduction in the number of low level queries from customers to our Ops team as customers are now able to retrieve the information for themselves.
Whether that is from the active environment or using the “History View” in Self Service.
3)
Feedback from our customers shows that they love the look and feel of Self Service, with its Simple, intuitive, modern user interface.
There is no scheduling tool knowledge required which means that it can be rolled out to anyone and can be of value immediately.
Self Service has also introduced a common language between the business and IT. The service names are in easy to understand terms and the views are shared.
The notes field, which is updated by Ops, is also a great tool when there are exceptions to the norm.
4)
Once the users and services are set up Self Service can be used straight away, therefore it is a very quick time to provide value for our customers.
Jack lead in then Stevie come in – you can see these quotes from aspects of the business are simple but extremely effective.
Particularly like the “quite simply, simple” quote.
Jack
As previously mentioned we implemented Batch Impact Manager in April 2014, at the same time as Self Service.
This has been of great benefit to our Operations Team as it means that BIM is proactively monitoring critical batch at a service level, as opposed to a job level on the GUI.
We call BIM our virtual Computer Operator! – BIM enables workload automation 'monitoring' from a 'virtual but for a real' service level – removing human error.
Some of our batch services have very little slack time and prior to the BIM implementation, we had a bit constant challenge of identifying when jobs would run longer than their average runtime.
Stevie
Key points
Long running jobs report
BIM highlights immediately
BIM estimated end times
We have an hourly report which lists any job that is executing for >30mins over the average.
However, this could have just missed one of the reports and you would have to wait an hour on the next report to highlight this. BIM gets us straight to the potential issue, as and when it occurs.
This allows Ops to take action immediately.
Another useful feature of BIM is when services are running late, say after system maintenance or code deployment outages. The estimated end times are very accurate and this information is made available to our out customers, full in the knowledge that it is accurate.
Prior to BIM we had to make a guess, based on manually reviewing the Control-M stats. This could become complicated, time consuming and prone to inaccuracies when there were hundreds or even thousands of jobs in a service.
Jack
What we want to show here is a few examples of how we have used Control-M to implement solutions to solve some problems that were impacting our business and customers.
Stevie
Replacement of in-house scheduling tool –
Key points
Big data project
Developed in-house scheduling tool
Unsupportable in prod
SLA and regulatory breaches
Control-M implemented
In 2012, a business requirement existed for a big data solution to capture and analyse Sky customer behaviours, preferences and habits so that Sky could optimise advertising revenue by adopting a targeted advertising solution for our customers.
Our Business Intelligence team were tasked with developing an application to underpin the solution.
As part of this process the team decided to develop an in-house scheduling package to manage workload automation.
Unfortunately at the time, the team were not aware of the capabilities of the Control-M service at Sky.
A lot of time and effort was deployed in the design, build and testing of the solution.
However, once in production the solution was quickly found to unsuportable
The status of the service could only be gained via complex sql queries.
There was a lot of dead time in the schedule but no easy way of identifying where this existed.
The requirement for a daily run was not being achieved as a daily run took anything between 24 – 40 hours.
Even the most basic scheduling functionality had to be coded in a very cumbersome fashion.
The solution caused a lot of pain for our Technology teams but more importantly our customers. There were very tight deadlines on the data which was not delivered to our customers on time on most days. In this case the customers were Sky’s exec, Government agencies, and other broadcasters such as the BBC.
This left us in danger of breaching our contractual obligations and thus liable to financial or legal penalties.
Then they saw the light and called in the Control-M Cavalry!!
We got involved and were able to analyse each process and quite simply utilise functionality native to Control-M to deliver a new solution
This service is now managed via Control-M, monitored by our Ops Team, It is also a BIM service so any exceptions are instantly escalated to Ops and resolved before they become customer impacting.
Since implementation we have gone from failing the SLA on almost every day to achieving the SLA every day.
Huge financial savings on the overhead of the original solution.
Jack
Sky Bet Online Gambling
Rapidly growing part of the business
Pace of growth has outstripped capabilities of Technology, including current scheduling tools
Increasing challenges to code around requirements using current tools – Chronicle, cron and Jenkins
POC of Control-M successful in June 2014
Migration to Control-M ongoing currently - TPOC
Support teams also have full visibility via Control-M Self Service
Scalability in both functionality and capacity fit for purpose going forward.
Mature working model for changes adopted.
License and support cost savings through the decommission of Chronicle.
Stevie
Implementation of Business Objects plugin for Control-M –
Key points
Exec reporting run through Control-M
Handover to SkyIQ
BO processing not through Control-M
Control-M implemented in two weeks
Fully visible plus SLA met every day
The extract of the data for our executive reporting suite at Sky is managed through Control-M.
The data contains vital information about Sky’s performance and is used to produce reports that are published on the London Stock Exchange, ultimately affecting the value of the company.
The batch window was very tight – 6 hours overnight to deliver a schedule whose runtime was around 5 hours 30mins.
There was a handover of data to one of our downstream business units who had an SLA of 09:00 to deliver the data to the exec.
The data was passed through the Business Objects application.
There was no easy way to control the flow through BO, therefore someone was onsite from 06:00 to manage this through
120 reports were launched concurrently. Most of which failed due to lack of system resource on the BO app.
SLA failure on most days, leading to massive impact to Sky
The Control-M cavalry was once again called in!
Business Objects plugin for Control-M was identified as a solution to this problem.
It was implemented tested and delivered within two weeks.
Priorities of the various processes were identified and standard Control-M resource and priority functionality was adopted.
Process was optimised over a period of a few weeks.,
SLA now met on almost every day.
Stevie
Hadoop implementation –
The Control-M for Hadoop plugin was implemented in June 2014 as part of the big data project at Sky.
The batch solution is currently in the design phase.
Hadoop agent installed/configured.
POC delivered
First Hadoop job set up.
First phase should hit production by the spring of next year.
All of the usual functionality of Control-M
.
Notes on Hadoop -
Hadoop is an open-source software framework for storage and large-scale processing of data-sets on clusters of commodity hardware.
Hadoop is a way of storing enormous data sets across distributed clusters of servers and then running "distributed" analysis applications in each cluster.
If you remember nothing else about Hadoop, keep this in mind: It has two main parts - a data processing framework and a distributed filesystem for data storage.
Integrate with Pig, Hive, Sqoop, MapReduce, and Hadoop Distributed File System (HDFS).
Stevie
Business Process Integration Suite (or Control-M for Web Services, Java and Messaging as it has recently been renamed)
Our Lightweight Services Team are implementing a solution that uses the MSG component of the plugin to manage MQ messaging.
Insanely simple to set up – JRE and message queue name in the account
Job calls the account
Scope for much more use going forward also Java plugin.
Jack
Expansion of Control-M into new business areas
We spoke about SkyBet online gambling in the last slide.
There are a couple of POC’s ongoing, looking into the benefits of migrating existing processes into Control-M - Similar benefits to SkyBet
Exploring opportunities in the Broadcast, Netbackup, Auto server Builds and big data.
Stevie
Control-M V8.0 or V9.0 Upgrade
As we mentioned earlier we implemented Control-M V8.0 in our sandbox and training envs last year. This has allowed us to get a feel for the new front end in V8.0
It has been really well received, with the team loving this and wanting it now. If only it were that straight forward!!
Recently we had a session with the Control-M R&D Team to review the functionality of Control-M V9.0.
Like a lot of Control-M customers we have found the agent upgrades to be the most time consuming part of Control-M upgrades.
This functionality arrived with V8.0 but goes a step further in V9.0 with the ability to package up Control-M agent deployments through the CCM (some space considerations – default 750Mb up to 1.5Gb)
As I understand it if we migrate from our current V7.0 to V9.0 then we can do this with no agent downtime (need to verify this with Martin)
Stevie
Control-M Workload Change Manager Trial
We are looking to trial this towards the end of this year
It will open up the world of the scheduling space in the same that that Self Service has done for the Control-M active environment.
A small number of Dev Ops Teams, SkyBet being a good example, want to maintain some level of control over the scheduling
Proposed benefits include better collaboration tools between schedulers and Dev Ops Teams, enable the implementation of site standards,
Jack
Over the past 16 years BMC Control-M has proven to be critical to the success of Sky’s business
Control-M is “woven into the fabric” of what we do at Sky
Control-M Self Service has “changed our customers world” by transforming interactions with workload automation services
Sky & BMC's relationship is a happening place - watch this space, the future is exciting!