National Instruments built a DevOps team to rapidly deliver new cloud-based software products using cloud hosting platforms and model-driven automation. With this approach, the small DevOps team has quickly delivered multiple major products to market with low costs. The team uses agile processes, cloud infrastructure from Amazon Web Services and Microsoft Azure, and a custom system called PIE for infrastructure automation. This has allowed National Instruments to innovate faster while maintaining reliability.

1. National Instruments

2. DevOps and Cloud at NI
Ernest Mueller
Cloud Systems Architect, LabVIEW R&D
ernest.mueller@ni.com @ernestmueller http://theagileadmin.com

3. The Short Form
We built a DevOps team to rapidly deliver new SaaS
products and product functionality using cloud
hosting and services (IaaS, PaaS, SaaS) as the
platform and operations, using model driven
automation, as a key differentiating element.
With this approach we have delivered multiple major
products to market quickly with a very small staffing
and financial outlay.
3

4. National Instruments (aka “NI”)
• 30 years old; 5000+ employees around the
world, half in Austin, mostly engineers; $873M in
2010
• Hardware and software for data
• acquisition, our graphical dataflow
LabVIEW is embedded design, instrument
control, and test
programming language used by
scientists and engineers in many
fields
4

5. From toys to black holes
5

6. Genesis
• Our hardware and software product strategy started
to spawn software-as-a-service ideas – some from
customer demand, some from internal drivers
• There were existing product to Web integration points
but these were uncoordinated and poorly maintained
• LabVIEW R&D greenfielded an internal group in 2009
to serve as an internal ISV for hosted services, the
dotCom team, seeded by a couple key folks from IT
6

7. Challenges
• Lack of close R&D/IT relationship
• Traditional siloed IT department (programmers split by
business unit, infrastructure split by technology)
• Low organizational agility – 6 weeks to get a server
• Uptime problems from complexity and silos
• On premise data centers at power/cooling capacity
• R&D primarily experienced in desktop software and
specialized, dedicated hardware, not server/Web scale
• Consensus driven environment
7

8. More Challenges
• IT software releases painful – quarterly for ERP, monthly for
Web – Web often having 80+ release items, many manual.
• Production issues (availability, performance, security) would
persist for months on average awaiting resolution
• Waterfall system development process created friction
against dev teams uptaking agile methods
• Software often not designed with any service management
goals in mind
• Dev teams frustrated with low throughput of
changes/requests from systems team
8

9. Now, Discover Your Strengths
• Strong base of “best and brightest,” motivated employees
• Culture of innovation and “do it yourself”
• Large Web presence (ni.com) with extensive in house
programming and operational experience
• Entrepreneurial internal environment
• Significant reinvention/retooling effort going on in R&D
• Increasing focus on system sales and quality (performance,
reliability, security) over yet-more-features
9

10. DevOps Ideation
• We had developed a Systems Development
Framework we used with dev teams, but it was an
inherently waterfall model
• Velocity conference began to spread Ops best
practices and catalyze innovation in 2008
• OpsCamp Austin (Jan 2010) introduced us to the
Visible Ops book and the “DevOps” term. “Hey, so
that’s what we’ve been trying to do!”
• Able to move forward quickly with more confidence
10

11. Sidebar: What Is DevOps?
• When I say DevOps, I mean:
 Developers and operations staff both included as part of
the project team – NOT developers throwing over a wall to
QA who throws over a wall to operations
 Developers and operations staff collaborating on service
design, development, testing, release, and management –
availability, capacity, security, etc. all need work at design
time and have obvious operational requirements
11

12. Starting Fresh - Blessing and Curse
Everything was new, so we had to simultaneously develop:
• Products
• Team
• Process
• Systems
• Code
• Operations
• System Automation
12

13. The Products (“Hosted Services”)
• Customer facing:
 LabVIEW Web UI Builder (2009)
 LabVIEW FPGA Compile Cloud (2010)
 Technical Data Cloud (2011)
 More in progress!
• Internal facing (NI product teams are the customer):
 LabVIEW.com Cloud Framework
 Cloud Hosting
 Operations
13

14. LabVIEW Web UI Builder
• Write a LabVIEW(ish) app, save it to the cloud and run it there.
• Build and deploy it to an embedded target and hook it up to
Web services to give it a sweet UI
• Also, an experimental testbed for LabVIEW changes
• Freemium model – use it for free, packaging and deploying
your app to a target requires a license (compiles run in the
cloud) – try it at ni.com/uibuilder
• Silverlight RIA, back end on Amazon Web Services – EC2, S3,
SimpleDB; Java/Linux/Apache/Tomcat and .NET/IIS/Windows
14

15. 15

16. 16

17. LabVIEW Web UI Builder Cloudlet
WebLV
WebLV
Compiler
Browser Services
Security
Web Server
Project
Services
Data
Internal Auth Db
Routing
Services
License Db
PIE
LDAP
Load Install Mgmt File
DNS Gateway
Balancer Services Server Server
17

18. LabVIEW FPGA Compile Cloud
• LabVIEW FPGA compiles take hours and consume
extensive system resources; compilers are getting
larger and more complex
• Implemented on Amazon - EC2,
Java/Linux,C#/.NET/Windows,
and LabVIEW FPGA
• Also an on premise product,
the “Compile Farm”
18

19. LabVIEW FPGA Compile Cloud
NI Hosted Compile Service
User Login
& Rights management
Links to user account
& support
19

20. Technical Data Cloud
• “I just want to upload my sensor data directly to the
cloud, man.”
• REST and LabVIEW API that lets you upload and
retrieve discrete and waveform data
• Welcome to the Internet of Things
• Being built on Microsoft Azure – specific bits TBD, all
.NET and LabVIEW
21

21. LabVIEW.com Cloud Framework
• Platform that makes the magic happen by providing base
plumbing for developers of SaaS apps
• Core Services - reusable Web services and facilities
• ILLS (internal login & licensing services) – distributed user
repo and licensing, complete with feed from Oracle and self
service user portal; Java/Tomcat, OpenDS LDAP, mySQL
• PIE (Programmable Infrastructure Environment) – sets up
systems for you, autoscales, deploys code; uses an XML
model and runtime registry; Java – more on this later!
• Building out a core platform? Didn’t that slow velocity? No.
22

22. The Team
• DevOps!
 Application architect
 Systems architect (me)
 2 developers
 1 system automation developer
 Operations lead
 2 follow-the-sun operations staff in Malaysia
• Work with other R&D product development teams
 Different orgs (LabVIEW, non-LV software, hardware)
 Geographically distributed (Austin, Aachen, Bangalore, Singapore)
24

23. The Process
• Agile!
• All systems work used the “developer” tools and systems as
part of DevOps collaboration philosophy
 Revision control (Perforce)
 Bug tracking (HP Quality Center)
 Specs and reviews (Atlassian Confluence wiki)
 Task tracking and burndown (JIRA/Greenhopper -> TFS)
• All members collaborate on all aspects of the product
• Test driven development – well, ideally
25

24. The Systems
• Cloud!
• After a quick cost assessment and experimentation, decided
on Amazon EC2 as our initial hosting platform, added Azure
later due to business relationship
• Needed control and agility we wouldn’t be able to get
internally – dynamic requirements, fast scaling
• Needed Linux and Windows both for software support
• Using multiple point SaaS providers for functionality (If it’s not
core, outsource it!)
• Agility and time to market far outweighed cost efficiency
26

25. Code
• REST!
• All REST-based Web services
• Multiple tech stacks - cloud and systems mgmt code mostly
in Java, product code mostly in C#/.NET
• Key cloud app architecture concerns – multitenant, parallel,
asynchronous, loosely coupled, APIed, instrumented,
resilient in dynamic/ephemeral environment
• Developers deliver tests, monitoring, system model with their
service
27

26. Operations
• The “secret sauce”!
• Not just ticket handling or “keep the lights on.” Focus on
delivering value to the customer and developer.
• Provide performance management, availability, systems
management, incident handling, security, log management,
monitoring, rapid deployment
• Inspirations: O’Reilly “Secret Sauce” paper, Velocity
conference, Visible Ops book, Transparent Uptime blog
28

27. Sidebar: What Is Operations?
• Operations is a domain of expertise within IT
systems management that involves the planning,
design, deployment, operation, security,
maintenance, monitoring, tuning, and repair of
applications and systems.
• Includes: systems engineers, systems administrators,
release managers, support techs
• It is a set of knowledge used across lifecycle phases
29

28. System Automation
• PIE!
• The “Programmable Infrastructure Environment”
• XML system model defines systems, services, code installs,
runtime interaction
• Runtime registry for systems info and eventing
• PIE autobuilds the runtime system from the model –
provisioning, software installs, monitoring integration
• Perform orchestration and control on many instances of
dynamic environments
30

29. 31

30. Results
• Win!
• A continuous pipeline of products delivered quickly
• LabVIEW Web UI Builder went beta in 2009, 1.0 in
2010
• FPGA Compile Cloud went beta in 2010, 1.0 in 2011
• Technical Data Cloud going beta in 2011
• Unqualified happiness with cloud, DevOps approach
• Not innovation vs. reliability – new approach gets both!
32

31. Bottom Line
• Operations is no longer a basic facilities-like “gate” –
many modern applications (our cloud applications, for
sure) require extensive systems engineering for
functionality, customer satisfaction, and profitability
33

32. Residual Challenges/Lessons Learned
• Culture change – building collaboration, mutual respect, and
trust among globally distributed dev teams, ops, and others
(QA, security, etc.)
• Educating developers on operational issues and vice versa
• General agile challenges (large teams, global teams)
• New agile infrastructure challenges (what does unit testing
mean for my environment build, and how do I do that?)
• Maintaining vision through rapid change
• Cloud-compatible tooling still emerging
• Selling the products; product manager collaboration
34

33. Where do we go from here?
• Continue to evolve PIE to add better orchestration, treat data
as a first class citizen
• Move to full continuous integration and deploy-on-demand,
necessitating intense investment in testing
• Uptake of Microsoft Azure (mostly complete) as an additional
cloud provider
• Private cloud integration; OpenStack/virtualization
• More SaaS products!
35

34. Discussion!
36