It isn’t just about workstations and laptops any more.
If you haven’t read the 25 point plan, you should follow the link and read it.Of special interest to this discussion are 1 – FDCCI and 3 – Cloud firstFDCCI is a huge effort to greatly reduce the number of Federal data centers which has very high level visibility from both the President and Congress.
“When a college student is sitting in a classroom with a MacBook Air and a WiFi network, they’re Tweeting and Facebook connecting with friends, getting daily Groupon emails — that’s a lifestyle that the government doesn’t currently cater to,”
VA has developed a fairly radical plan to be completely out of the data center operations business before 2020.We are leveraging common goals with DoD to begin this process and we will start early next year.
No sane person recommends “forklifting” old technology into new data centers. FDCCI should be seen as an opportunity to standardize and more fully integrate the systems that have grown organically over the years.
Meritalk:•82% of the private sector* know their PUE vs. 23% of Federal agencies •94% of the private sector* know their average load across their data centers vs. 31% of Federal agencies•Fewer than half (42%) of Federal IT decision makers agree that their departments have an incentive to achieve data center savings, including savings that will be realized by budgets outside of ITVirtualization is the current FDCCI “buzz metric.” Not everything can be virtualized.Moving from a physical server is called P-to-V, which simply stands for Physical (platform) to Virtual. It takes time to virtualize the first oneReplication implies identical systems & versionsIt still takes staff to deploy and manage itIt still requires software licenseThis process can be time consuming, more so if the application being virtualized is going to be a tenant along with other applications. While the system administrator to sever ration is higher with virtualized vs. physical systems, there still comes a point when additional workload requires additional staff.This advantage is also compromised when a variety of non-standard software or versions is involved.Do you notice a theme developing here? Standardization aids virtualization and reduces support costs.
Like many other agencies, we have started by developing a private Cloud. I don’t think we’ll stay at this level very long, or at least not exclusively in the private Cloud business.We are already in the public cloud with the Chapter 33 Education system. We have about 1500 claims processors on the education system$15B of benefits have been paid
While you may read articles saying that there is confusion as to what Cloud really means, from a Federal standpoint, there really isn’t. We use the NIST definition.
On-demand self-service: A consumer can unilaterally provision computing capabilities automatically without requiring human interaction with each service’s provider.Broad network access: Capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms.Resource pooling: The provider’s computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. The customer generally has no control or knowledge over the exact location of the provided resources.Rapid elasticity: Capabilities can be rapidly and elastically provisioned, in some cases automatically, to quickly scale out and in.Measured Service: Cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported providing transparency for both the provider and consumer of the utilized service.
Again, VA uses the NIST definition, which is government-wide. Vendors often cite one or more of these features, which tends to muddy the definition of what Cloud really means or what feature set it provides.Agility: rapid provisioning, both in an IT and business office senseCost Savings: not entirely proven, does shift purchases from massive infrastructure outlays to services which require much less up front costsDevice Independence: web browser basedMulti-tenancy: better utilizes infrastructureImproved Availability: data centers more reliable than computer rooms/closets. Better/newer infrastructure, monitoring, 24 x 7 staffing.Scalability: Assumes excess capacity in place, provides what is often called “the illusion of unlimited/infinite capacity”Security: Physical security at data centers, but this is a controversial issue. There are concerns about loss of control over sensitive data. Security becomes much more complex to administer. There are concerns about how easily data could be moved from one vendor to another. Most Federal agencies are taking a “go slow” approach for systems with PII and are utilizing private clouds for those systems.Ease of Maintenance: Cloud deployments are typically replicated systems. It is easier to patch once and replicate for both hardware and software.
A few other observations based on the NIST definition of Cloudand software.
An example of a Private Cloud would be VLER or MyHealtheVet. The systems are housed in VA managed data centers, the systems are administered by VA personnel, network connectivity to the data centers are controlled and monitored by VA.An example of a Public Cloud would be Chapter 33 which is hosted at a third party data center. In both cases, VA would manage the application, but in Public Cloud hosting, the servers, storage and network connectivity are typically managed by vendor personnel.
Cloud isn’t just about technology. The underpinning operations of IT and the business will be impacted as well.
There is an assumption made here that going into a new project, these key components are known. In our current state, most are typically not.
Barring some unforeseen changes in Federal law, competing with private sector will be very difficult in the Cloud services arena. Cloud assumes speed, government process are not amenable to rapidity of execution.Ultimately, construction dollars are an even more limiting factor than staffing and purchasing constraints.
The term “Legacy system” was coined by vendors eager to sell shiny new systems with the implication that older systems were dowdy, outmoded and difficult to maintain. By definition however, Legacy systems are systems work and are in place because they provide a necessary service and replacement would come at some risk in additional to financial outlay. Replacing them with new systems is a complex decision that likely will be made on a case-by-case basis. Much of our code base, as is the case in most other private and public sector IT operations, is not immediately or easily virtualized. Which isn’t to say this can’t be or shouldn’t be done, but likely will be over time. We’ll discuss that more when we get to the VA specific portion of this presentation.Legacy systems are sometimes cited, as in a recent GCN article, as the likeliest target for virtualization and Cloud. In reality, they’re probably the least likely in most cases.
But at this point in time, we’re all over the map regarding “standards.” for waivers to them.As we move toward multi-tenancy on server and storage farms, we also need to get serious about release management and versioning controls. This can be a hard sell to business customers who really don’t understand why we want to update to new versions of operating systems or application software when what they have is running fine. We need to examine not only the cost to upgrade, but the increased maintenance costs – both in terms of hardware and skill sets – needed to stand still. We need to stop talking and start doing as far as new projects are concerned. We need to insist at minimum that all new projects adhere to VA standards rather than applying for waivers. If we don’t start somewhere, we’ll never get there.
Most projects don’t budget for regression testing and as a result, under schedule pressures, the push is to simply slap the same old stuff out there with vague promises that the next version will include upgrades. They almost never do.In the future, SD&E will route hosting to where it makes the most sense to host. Start designing with network latency in mind.Communication is key. It does little good to make a high level presentation if the information isn’t disseminated down through the ranks. If we have project team A working on a Cloud First approach and project team B blissfully programming away like it was 1999.
John rucker keynote presentation 11-01-2011
John Rucker Data Center Operations FDCCI representativeDepartment of Veterans Affairs
December 2012 isn’t that far off… FDCCI Virtualization 25 Point Plan Budget Woes Legacy systems Cloud Mobile Devices
Deliver business outcomes and customer experience withoutregard to infrastructureDeploy a hospital tracking system like you would “Angry Birds”on an iPhone.Leave your expectations at the door; we need a new way tomanage IT. Vivek Kundra, Federal CIO NIST Cloud Computing Workshop, April 2011One Trillion Devices Connected to the Internet By 2013 Padma Warrior, Cisco Chief Technology Officer March, 2010
1 Complete detailed implementation plans to consolidate 800 data centers by 20152 Create a government-wide marketplace for data center availability3 Shift to a “Cloud First” policy4 Stand-up contract vehicles for secure IaaS solutions5 Stand-up contract vehicles for “commodity” services6 Develop a strategy for shared services7 Design a formal IT program management career path8 Scale IT program management career path9 Require Integrated Program Teams10 Launch a best practices collaboration platform11 Launch technology fellows program12 Enable IT program manager mobility across government and industryhttp://www.cio.gov/documents/25-Point-Implementation-Plan-to-Reform-Federal%20IT.pdf
13 Design and develop cadre of specialized IT acquisition professionals14 Identify IT acquisition best practices and adopt government-wide15 Issue contracting guidance and templates to support modular development16 Reduce barriers to entry for small innovative technology companies17 Work with Congress to create IT budget models that align with modular development·18 Develop supporting materials and guidance for flexible IT budget models19 Work with Congress to scale flexible IT budget models more broadly20 Work with Congress to consolidate Commodity IT spending under Agency CIO21 Reform and strengthen Investment Review Boards ·22 Redefine role of Agency CIOs and Federal CIO Council23 Rollout “TechStat” model at bureau-level ·24 Launch “myth-busters” education campaign25 Launch an interactive platform for pre-RFP agency-industry collaboration
Weed out inefficiencies,Recognize cost savingsChange a fiefdom culture to a sharing cultureDarwinian pressure when it comes toinnovationCreating a 21st century workplace
Geographical Locations VA Strategic Goals REGION 2 All VA enterprise and mission REGION 1 critical systems consolidated by 2018 in 4 or fewer National Data Centers (NDCs) REGION 4 1 20 23 12 No VA Owned Data Centers by 2 19 11 ITC Location end of 2018 21 10 4 VA will continue to administer 3 applications and systems inGuam 15 5 NDC’s 9 6 VA will co-locate all Region 2 21 22 16Philippines 18 7 VistA systems into St. Louis 21 REGION 3 DECC and Region 3 VistA 17 systems into Warner Robins DECC beginning Q2FY12Alaska 20 Hawaii 21 8 CRDC will move into CRRC Puerto Rico 2012. 8 VA will co-locate all Region 1 and Region 4 VistA Systems into VISN Location DISA DECCs in FY13 DECC Location R2 & R3 (2011 - 2013) ITCs will consolidate into NDCs DECC Location R1 & R4 (2013 - 2015) by 2018
Planning Beats FlounderingDevelop a Communications PlanDon’t forget the Risk Register101: Bite Sized PiecesCharter Your ProjectScope Creep Kills More Projects Than Any Other FactorIdentify Stakeholders There are more of them than you think
Can’t Have Too MuchWell, actually you can…Manage Your Stakeholder ListHave FAQ on handHave the “Short Version” availableWeb Site vs. SharePointSomeone Won’t Get the MessageMarket SuccessAdmit What You Don’t Know
FDCCI is going to be painful and expensive if we don’t virtualize & standardize. We need to start nowVA will complete consolidations within this decade
MoneyTimeTurfMetrics Can’t Measure Some Things Measuring the Wrong Things Cost Savings Difficult to Estimate
VA has only one CIO. One. And we are the2nd largest Federal DepartmentUp to 100,000 Tablets/Smart Phones in next18 monthsPMASAgile DevelopmentCloud: Private & Public
Cloud computing is a model for enabling highly-available, convenient, on-demand access to ashared pool of configurable computing resources(e.g., networks, servers, storage, applications,and services) that can be rapidly provisioned andreleased with minimal management effort orservice provider interaction.“NIST definition of Cloud Computing” National Institute of Standards andTechnology Special Publication 800-145 (Draft); January 2011
On-demand self-serviceBroad network accessResource poolingRapid elasticityMeasured Service
AgilityCost SavingsDevice IndependenceMulti-tenancyImproved AvailabilityScalabilitySecurityEase of Maintenance
Minimal management effort or service providerinteraction: Reduce number of iterations throughthe approval process; provide more self-service;allow easier cost estimating by customer.Location independence: Workload can runanywhere; move aroundQuickly Scale Out; Quickly Scale In: Scaling tools inthe hands of customer; automated scaling; theillusion of unlimited capacity“Your” platform could be anywhere, at any giventime and shared with others.Server huggers don’t like Cloud.
Private: Run within the organization. Fullorganizational control (although control can beshared between organizational elements)Public: Non-VA owned. Usually non-governmentowned. Control can vary, but inevitably somesecurity controls run by the vendor, ofteneverything but the application is vendor managedVarious Mixtures: Not us, not yet
FundingCost ModelsApproval ProcessFinancial TransparencySecurityStaffingNetworkInfrastructure As if you don’t know this already
All IT business processes will integrate moving-to-the-cloud initiative by factoring it into thedecision making. Not just included in the development lifecycleof programs/projects but factored into alldecision-making and implementation for new andongoing business processes
• Cost• Privacy• Compliance• Application• Bandwidth/latency• Schedule There is NO default assumption!
This assumes we have a transparent apples-to-apples cost comparison – currently we do not This assumes we have fairly tight requirements for applications – often we do not• Sea change For VA IT operations - our role will become managing infrastructure, not owning assets Mirrors Changes in Software Production – move from in-house to outsourced
Long term, Federal IT Operations are ill-equipped andill prepared to compete with public CloudI don’t see this situation improvingThe end of Federal leadership in IT operations is athandThis doesn’t mean we won’t continue to exertleadership in other IT areas
They offer far more flexibility than we can realistically hopeto achieve, no matter how agile and hard working ourorganizations are: • We can’t hire fast enough • We can’t buy fast enough • We can’t build/upgrade facilities fast enough • We can’t decide fast enough In short, we simply don’t move at “Cloud Speed”
• Legacy systems need not be outmoded or inefficient.• Legacy systems typically follow a predictable capacity planning profile – rapid provisioning not needed• Already amortized• Costly to move to Cloud• There is much to be said for stable, reliable systems!• Not everything can or should be moved to Cloud• …but let’s not overplay that “security” card
We all need to get serious about standardization Standard Tools Standard Platforms Standard VersionsWe all need to get serious about releasemanagementCloud isn’t going awayVirtualization isn’t a fadWe’re not as special as we think we areStop Talking and Start Doing
Adopt Platform and Software Standards Implement Release Management Build in Regression Testing Now Data Center Independence Vendor Independence Device Independence Develop a costing model Establish Governance Train Staff• Communicate. Communicate. Communicate.
As we move into a “post-operational” world, we needto develop different skill sets & staffing modelsThe next gen Federal IT will have to think more likesystems aggregators and system architectsWill also need to consider IT from a businessperspectiveGreat opportunity for new workers, or workerslooking for a career changeThis transition will not be easy for existing staffHuge management challengeWe still have an obligation to protect Federal data