Earlier this year, ICONIQ Growth¹ performed an in-depth study of the developer technology stack to help us better understanding emerging trends, most commonly adopted tools, and key questions assessed during decision making processes.
Beyond DevOps: Finding Value through RequirementsGail Murphy
DevOps practices have enabled faster delivery of software features. However, there remains a gap in consistently tracking how features connect to customer and organizational value. Requirements engineering needs to play a key role in identifying and linking features to value, as well as tracking value delivery and reassessing features over time. This will allow organizations to focus on delivering value rather than just features through their software development processes.
The (Un) Expected Impact of Tools in Software EvolutionGail Murphy
The document discusses how software architecture and tool architecture impact a continual flow of value in software development. It argues that software architecture needs to evolve gracefully over time to enable value delivery, and that tools should support human and tool interactions to facilitate appropriate architectural decisions. However, tool architecture is often ignored despite likely interactions with software architecture. More research is needed to understand these relationships and how tool architecture can support software architecture evolution.
In this talk, I explore what productivity means to software developers, how we might track the value that is delivered in software produced by developers and how we might begin to think about measuring the productive delivery of effective software.
Keynote at International Conference on Performance Engineering (ICPE) 2020.
How to extend the shelf life of software and enable long-lived, adaptable software architectures.
Herzliya - July 2015 @
ILTAM - Israeli Users' Association of Advanced Technologies in Hi-Tec Integrated Systems
IASA - International Association of Software Architects
This document discusses the software development lifecycle (SDLC) and DevOps. It provides an overview of the SDLC phases and Agile Scrum framework. It describes the need for DevOps by explaining problems that can occur when development and operations teams are separated. It proposes DevOps as a solution to automate software delivery and infrastructure changes through a cross-functional team and toolchain. The document outlines various tools used in a DevOps toolchain for version control, IDEs, project management, continuous integration, testing, security, collaboration and more. It concludes by discussing future plans to implement OpenStack, Docker and gain experience with Amazon Web Services.
Achieving Secure DevOps: Overcoming the Risks of Modern Service DeliveryPerforce
DevOps and Continuous Delivery practices are attracting the attention of many organizations looking to increase the speed of their application delivery, yet doing so the wrong way can risk both quality and security. In this webinar, Forrester analysts Kurt Bittner and Rick Holland will share their insights on how DevOps and Security teams can work better together to meet these challenges, along with best practices for bringing greater security to product development and delivery.
DevOps continues to be a buzzword in the software development and operations world, but is it really a paradigm shift? It depends on what lens you view it through.
Roman Garber, an active software security engineering and software team lead thinks so. Ed Adams, Security Innovation CEO, a 20-year software quality veteran and former mechanical engineer, curmudgeonly disagrees.
Beyond DevOps: Finding Value through RequirementsGail Murphy
DevOps practices have enabled faster delivery of software features. However, there remains a gap in consistently tracking how features connect to customer and organizational value. Requirements engineering needs to play a key role in identifying and linking features to value, as well as tracking value delivery and reassessing features over time. This will allow organizations to focus on delivering value rather than just features through their software development processes.
The (Un) Expected Impact of Tools in Software EvolutionGail Murphy
The document discusses how software architecture and tool architecture impact a continual flow of value in software development. It argues that software architecture needs to evolve gracefully over time to enable value delivery, and that tools should support human and tool interactions to facilitate appropriate architectural decisions. However, tool architecture is often ignored despite likely interactions with software architecture. More research is needed to understand these relationships and how tool architecture can support software architecture evolution.
In this talk, I explore what productivity means to software developers, how we might track the value that is delivered in software produced by developers and how we might begin to think about measuring the productive delivery of effective software.
Keynote at International Conference on Performance Engineering (ICPE) 2020.
How to extend the shelf life of software and enable long-lived, adaptable software architectures.
Herzliya - July 2015 @
ILTAM - Israeli Users' Association of Advanced Technologies in Hi-Tec Integrated Systems
IASA - International Association of Software Architects
This document discusses the software development lifecycle (SDLC) and DevOps. It provides an overview of the SDLC phases and Agile Scrum framework. It describes the need for DevOps by explaining problems that can occur when development and operations teams are separated. It proposes DevOps as a solution to automate software delivery and infrastructure changes through a cross-functional team and toolchain. The document outlines various tools used in a DevOps toolchain for version control, IDEs, project management, continuous integration, testing, security, collaboration and more. It concludes by discussing future plans to implement OpenStack, Docker and gain experience with Amazon Web Services.
Achieving Secure DevOps: Overcoming the Risks of Modern Service DeliveryPerforce
DevOps and Continuous Delivery practices are attracting the attention of many organizations looking to increase the speed of their application delivery, yet doing so the wrong way can risk both quality and security. In this webinar, Forrester analysts Kurt Bittner and Rick Holland will share their insights on how DevOps and Security teams can work better together to meet these challenges, along with best practices for bringing greater security to product development and delivery.
DevOps continues to be a buzzword in the software development and operations world, but is it really a paradigm shift? It depends on what lens you view it through.
Roman Garber, an active software security engineering and software team lead thinks so. Ed Adams, Security Innovation CEO, a 20-year software quality veteran and former mechanical engineer, curmudgeonly disagrees.
Day 1: ICT Strategic Planning, Mr. Soufiane Ben Moussa, CTO, House of Commons...wepc2016
The challenges parliaments face are not simply ones of technology adoption; many are strategic and need to be addressed at a systemic level. To resolve this challenge, there needs to be a stronger focus on articulating, addressing and resolving the strategic barriers.
This document is a master's thesis that examines best practices for managing agile software development projects. It discusses traditional and agile development methodologies like Scrum and Extreme Programming (XP). It also analyzes three case studies of agile projects and evaluates tools to support agile development. The thesis concludes that adopting a methodology fully and using communication tools are keys to agile success.
An Introduction to Agile Software DevelopmentSerena Software
Agile software development stresses rapid iterations, small and frequent releases, and evolving requirements facilitated by direct user involvement in the development process. Serena’s application lifecycle management tools provide a framework to visualize scope, orchestrate mundane and repetitive development tasks, and enforce process. Unlike agile-specific products offered by agile-only vendors, Serena products are methodology neutral and can be applied equally well to agile as well as more traditional serial development processes, so they can support all the development activities within an enterprise.
Abstract:
Cybercrime in its various forms is expected to cost the world more than US$6 trillion per year by 2021. There are nearly 1 Million Viruses and Malware created daily. With the increased usage of open source and third-party components, it becomes challenging to ensure these externally developed components do not introduce security vulnerabilities into the final product. While adoption of Agile practices leads to continuous software release but security checks get pushed towards the end of the release cycle. This more often than not leads to uncomfortable situations. Many times it leads to delays also. With higher code velocity comes the challenge of making sure every change is secure.
Security can no longer remain an after-thought, it has to be integrated at every stage of the software delivery life-cycle (design for security, secure coding, security testing, penetration testing in staging, and security monitoring in production). These controls can be tightly integrated in DevOps pipeline and become operational much like monitoring tools. Engineering teams have to continuously test for security at Development, QA and Staging phases. This session will explore how to integrate the ecosystem of technologies to build security checks in all phases of software development like Architecture, Design and Implementation in order to create a true DevSecOps practice.
Key Takeaways:
1. What is the impact of Lean and Agile practices on Security verification?
2. How does adoption of Opensource and third party software increase the challenges of keeping our products secure?
3. How can you perform Security testing continuously in different phases of Agile software development?
4. How can adoption of DevSecOps practices lead to a culture of Continuous Security testing?
5. How to integrate tools and technologies to perform security checks in all phases of software development?
How can we communicate the effectiveness of DevOps to technical and business people?
What metaphors and examples help?
What kind of people should we hire?
This presentation was given as an Ignite talk at DevOps Days Europe 2010 in Hamburg.
Agile development and open development practices share a great deal of features. But the distributed nature of open development can make some common Agile practices difficult, or even impossible to adopt. This presentation is an initial exploration of how the two may mesh together.
In this lecture I will present a unified Agile process and techniques that allow for a seamless transition from the system engineering level to the SW engineering level in an iterative and evolutionary way. I will also show the benefits the unifying the processes of the two levels and of the resulting component based architecture. I will also talk on the architect’s role and this role evolves over time and will conclude with presenting a small but real life project example.
Extending Jenkins to the Mainframe. A Simpler Approach.DevOps.com
Wouldn’t it be great to use familiar tools when you’re needing to integrate mainframe applications into your DevOps pipeline?
Mainframe applications have historically required specialized tools and knowledge to build, maintain and integrate with distributed systems. Existing tool integrations are designed for vendor-specific tool chains and they require a great deal of specialized knowledge and expertise to set up.
CA Technologies engineering decided to break with tradition and utilize advancements on the zOS platform to provide a Command Line Interface that turns Mainframe into “just” another deployment target for the DevOps pipeline.
In this webinar the presenters will show how to integrate, build and test COBOL applications into a modern DevOps pipeline managed by Jenkins Continuous Integration software.
This document outlines an approach for integrating security into the software development lifecycle (SDLC) using DevSecOps principles. It discusses how security can shift left by being incorporated into various phases of product development and delivery, including product management, design, development, deployment, defect management, and monitoring. It provides examples of how to integrate security practices and tools at each stage. The goal is to establish security as a critical product feature rather than an afterthought, and foster collaboration between security and development teams through a DevSecOps model and maturity criteria.
CodeValue Architecture Next 2018 - Executive track dilemmas and solutions in...Erez PEDRO
Moderen Software projects are challenging to develop. Eran Stiller, Ronen Rubinfeld, and Erez Pedro from CodeValue show a method for conducting multidisciplinary product discovery.
This document provides an introduction to software engineering. It defines software as computer programs, documentation, and data structures. Software can be generic, developed for a general market, or bespoke (custom), developed for a single customer. The document also discusses what software engineering is, the difference between software engineering and computer science, the costs of software engineering, software engineering methods, CASE tools, attributes of good software, types of software applications, and characteristics of web applications.
The document discusses software security remediation and provides data on how long it takes to fix common vulnerabilities. It finds that setup, testing fixes, and deployment take significant time. Cross-site scripting fixes average 9.6 minutes for stored and 16.2 for reflected XSS. Confirming fixes and testing can take more time than the fixes. The data provides a starting point for planning but has limitations from being one company's projects. Understanding all phases is key to minimizng remediation costs.
Software Architecture for Agile DevelopmentHayim Makabee
Slides of a workshop given at Herzliya on June/2017, organized by ILTAM and IASA Israel. This workshop was dedicated to the topic of Software Architecture in the context of Agile Development. We answered the question: “How much Design Up Front should be done in an Agile project?” Hayim presented his approach of Adaptable Design Up Front (ADUF), describing its rationale, applications in practice and comparison to other approaches such as Emergent Design. He explained why adaptability is essential for the development of complex software systems using Agile methods. The concepts were illustrated through practical software architecture approaches such as micro-services and examples of real software systems that were developed in the past. The workshop also included an exercise on the definition and evolution of the design of an interesting system.
Mainframe DevOps: A Zowe CLI-enabled RoadmapDevOps.com
The Zowe open source framework, hosted by the Linux Foundation's Open Mainframe Project, is often referred to as a Swiss Army knife for mainframe modernization, but where to begin? This session, which is based on findings from numerous Design Thinking workshops, will help DevOps champions and mainframe leaders jumpstart their modernization journeys.
We’ll explore a few high-value use cases like plugging into enterprise CI/CD pipelines and incorporating off-platform tools like code quality. And by addressing practical considerations like Zowe installation, set-up and support, this session will equip attendees with the information they need to become mainframe DevOps mobilizers.
Jay Lyman 451 ResearchBrent Beer GitHubSteven Anderson Sendachi talk about these topics:
Cloud, DevOps, agile development capability and adoption of containers are all important in both perception and reality.
Enterprise adoption of cloud computing, DevOps, agile development and containers are all growing, including production use.
Modernizing applications to SaaS & migrating them to the cloud are equally important as net-new, so-called ‘cloud-native’ applications.
Advantages and benefits of these technologies and methodologies center on: flexibility and speed, cost reduction, improvements in resiliency and reliability and fitness for new/emerging applications.
Barriers center on: lack of internal skills, immaturity, lack of familiarity, satisfaction with current technology, cost and security.
The document discusses five principles for securing DevOps:
1) Automate security testing by integrating it into CI/CD pipelines.
2) Integrate security testing early to "fail quickly" and avoid issues late in the process.
3) Avoid false positives from security tests to prevent blocking critical updates.
4) Build security expertise among developers by training them in secure coding practices.
5) Maintain visibility into application security for deployed software to enable quick responses to attacks.
How to achieve security, reliability, and productivity in less timeRogue Wave Software
This introductory session lays the foundation for boosting the effectiveness of mission-critical systems testing by covering industry best practices for code security, software reliability, and team productivity. For each area, you will learn how to mitigate the top issues by seeing real examples and understanding the tools and techniques to overcome them. This includes: The value of different testing methods; The importance of standards compliance; and understanding how DevOps and continuous integration fit in.
SWE-401 - 2. Software Development life cycle (SDLC)ghayour abbas
The document describes the Software Development Life Cycle (SDLC) which is a structured process for developing software through stages including communication, requirement gathering, feasibility study, system analysis, software design, coding, testing, integration, implementation, and operation and maintenance. It also discusses several software development paradigms that guide the development process, such as waterfall, iterative, spiral, V-model, and big bang models.
- Software engineering is extremely complex and expensive work, with large software systems costing more than buildings and often having high failure rates.
- The two main factors that cause "runaway" software projects that exceed budgets and schedules are poor estimation done too early and unstable requirements that change frequently.
- Programmers are often given impossible tasks with too much work and not enough time, leading them to produce workarounds and quick fixes rather than well-designed solutions.
The document discusses several security-related topics including promoting the OWASP Orange Saft tool, outcomes from a security guidance stakeholder meeting, feedback for improving security guidance in IDEs, topics to cover in a new CISO guide, questions to include in the guide, securing GitHub integration, an incident response playbook, and a CISO round table discussion. It also summarizes outcomes from several breakout groups at an OWASP event on threat modeling, application security curriculum design, and infosec warranties and guarantees.
Day 1: ICT Strategic Planning, Mr. Soufiane Ben Moussa, CTO, House of Commons...wepc2016
The challenges parliaments face are not simply ones of technology adoption; many are strategic and need to be addressed at a systemic level. To resolve this challenge, there needs to be a stronger focus on articulating, addressing and resolving the strategic barriers.
This document is a master's thesis that examines best practices for managing agile software development projects. It discusses traditional and agile development methodologies like Scrum and Extreme Programming (XP). It also analyzes three case studies of agile projects and evaluates tools to support agile development. The thesis concludes that adopting a methodology fully and using communication tools are keys to agile success.
An Introduction to Agile Software DevelopmentSerena Software
Agile software development stresses rapid iterations, small and frequent releases, and evolving requirements facilitated by direct user involvement in the development process. Serena’s application lifecycle management tools provide a framework to visualize scope, orchestrate mundane and repetitive development tasks, and enforce process. Unlike agile-specific products offered by agile-only vendors, Serena products are methodology neutral and can be applied equally well to agile as well as more traditional serial development processes, so they can support all the development activities within an enterprise.
Abstract:
Cybercrime in its various forms is expected to cost the world more than US$6 trillion per year by 2021. There are nearly 1 Million Viruses and Malware created daily. With the increased usage of open source and third-party components, it becomes challenging to ensure these externally developed components do not introduce security vulnerabilities into the final product. While adoption of Agile practices leads to continuous software release but security checks get pushed towards the end of the release cycle. This more often than not leads to uncomfortable situations. Many times it leads to delays also. With higher code velocity comes the challenge of making sure every change is secure.
Security can no longer remain an after-thought, it has to be integrated at every stage of the software delivery life-cycle (design for security, secure coding, security testing, penetration testing in staging, and security monitoring in production). These controls can be tightly integrated in DevOps pipeline and become operational much like monitoring tools. Engineering teams have to continuously test for security at Development, QA and Staging phases. This session will explore how to integrate the ecosystem of technologies to build security checks in all phases of software development like Architecture, Design and Implementation in order to create a true DevSecOps practice.
Key Takeaways:
1. What is the impact of Lean and Agile practices on Security verification?
2. How does adoption of Opensource and third party software increase the challenges of keeping our products secure?
3. How can you perform Security testing continuously in different phases of Agile software development?
4. How can adoption of DevSecOps practices lead to a culture of Continuous Security testing?
5. How to integrate tools and technologies to perform security checks in all phases of software development?
How can we communicate the effectiveness of DevOps to technical and business people?
What metaphors and examples help?
What kind of people should we hire?
This presentation was given as an Ignite talk at DevOps Days Europe 2010 in Hamburg.
Agile development and open development practices share a great deal of features. But the distributed nature of open development can make some common Agile practices difficult, or even impossible to adopt. This presentation is an initial exploration of how the two may mesh together.
In this lecture I will present a unified Agile process and techniques that allow for a seamless transition from the system engineering level to the SW engineering level in an iterative and evolutionary way. I will also show the benefits the unifying the processes of the two levels and of the resulting component based architecture. I will also talk on the architect’s role and this role evolves over time and will conclude with presenting a small but real life project example.
Extending Jenkins to the Mainframe. A Simpler Approach.DevOps.com
Wouldn’t it be great to use familiar tools when you’re needing to integrate mainframe applications into your DevOps pipeline?
Mainframe applications have historically required specialized tools and knowledge to build, maintain and integrate with distributed systems. Existing tool integrations are designed for vendor-specific tool chains and they require a great deal of specialized knowledge and expertise to set up.
CA Technologies engineering decided to break with tradition and utilize advancements on the zOS platform to provide a Command Line Interface that turns Mainframe into “just” another deployment target for the DevOps pipeline.
In this webinar the presenters will show how to integrate, build and test COBOL applications into a modern DevOps pipeline managed by Jenkins Continuous Integration software.
This document outlines an approach for integrating security into the software development lifecycle (SDLC) using DevSecOps principles. It discusses how security can shift left by being incorporated into various phases of product development and delivery, including product management, design, development, deployment, defect management, and monitoring. It provides examples of how to integrate security practices and tools at each stage. The goal is to establish security as a critical product feature rather than an afterthought, and foster collaboration between security and development teams through a DevSecOps model and maturity criteria.
CodeValue Architecture Next 2018 - Executive track dilemmas and solutions in...Erez PEDRO
Moderen Software projects are challenging to develop. Eran Stiller, Ronen Rubinfeld, and Erez Pedro from CodeValue show a method for conducting multidisciplinary product discovery.
This document provides an introduction to software engineering. It defines software as computer programs, documentation, and data structures. Software can be generic, developed for a general market, or bespoke (custom), developed for a single customer. The document also discusses what software engineering is, the difference between software engineering and computer science, the costs of software engineering, software engineering methods, CASE tools, attributes of good software, types of software applications, and characteristics of web applications.
The document discusses software security remediation and provides data on how long it takes to fix common vulnerabilities. It finds that setup, testing fixes, and deployment take significant time. Cross-site scripting fixes average 9.6 minutes for stored and 16.2 for reflected XSS. Confirming fixes and testing can take more time than the fixes. The data provides a starting point for planning but has limitations from being one company's projects. Understanding all phases is key to minimizng remediation costs.
Software Architecture for Agile DevelopmentHayim Makabee
Slides of a workshop given at Herzliya on June/2017, organized by ILTAM and IASA Israel. This workshop was dedicated to the topic of Software Architecture in the context of Agile Development. We answered the question: “How much Design Up Front should be done in an Agile project?” Hayim presented his approach of Adaptable Design Up Front (ADUF), describing its rationale, applications in practice and comparison to other approaches such as Emergent Design. He explained why adaptability is essential for the development of complex software systems using Agile methods. The concepts were illustrated through practical software architecture approaches such as micro-services and examples of real software systems that were developed in the past. The workshop also included an exercise on the definition and evolution of the design of an interesting system.
Mainframe DevOps: A Zowe CLI-enabled RoadmapDevOps.com
The Zowe open source framework, hosted by the Linux Foundation's Open Mainframe Project, is often referred to as a Swiss Army knife for mainframe modernization, but where to begin? This session, which is based on findings from numerous Design Thinking workshops, will help DevOps champions and mainframe leaders jumpstart their modernization journeys.
We’ll explore a few high-value use cases like plugging into enterprise CI/CD pipelines and incorporating off-platform tools like code quality. And by addressing practical considerations like Zowe installation, set-up and support, this session will equip attendees with the information they need to become mainframe DevOps mobilizers.
Jay Lyman 451 ResearchBrent Beer GitHubSteven Anderson Sendachi talk about these topics:
Cloud, DevOps, agile development capability and adoption of containers are all important in both perception and reality.
Enterprise adoption of cloud computing, DevOps, agile development and containers are all growing, including production use.
Modernizing applications to SaaS & migrating them to the cloud are equally important as net-new, so-called ‘cloud-native’ applications.
Advantages and benefits of these technologies and methodologies center on: flexibility and speed, cost reduction, improvements in resiliency and reliability and fitness for new/emerging applications.
Barriers center on: lack of internal skills, immaturity, lack of familiarity, satisfaction with current technology, cost and security.
The document discusses five principles for securing DevOps:
1) Automate security testing by integrating it into CI/CD pipelines.
2) Integrate security testing early to "fail quickly" and avoid issues late in the process.
3) Avoid false positives from security tests to prevent blocking critical updates.
4) Build security expertise among developers by training them in secure coding practices.
5) Maintain visibility into application security for deployed software to enable quick responses to attacks.
How to achieve security, reliability, and productivity in less timeRogue Wave Software
This introductory session lays the foundation for boosting the effectiveness of mission-critical systems testing by covering industry best practices for code security, software reliability, and team productivity. For each area, you will learn how to mitigate the top issues by seeing real examples and understanding the tools and techniques to overcome them. This includes: The value of different testing methods; The importance of standards compliance; and understanding how DevOps and continuous integration fit in.
SWE-401 - 2. Software Development life cycle (SDLC)ghayour abbas
The document describes the Software Development Life Cycle (SDLC) which is a structured process for developing software through stages including communication, requirement gathering, feasibility study, system analysis, software design, coding, testing, integration, implementation, and operation and maintenance. It also discusses several software development paradigms that guide the development process, such as waterfall, iterative, spiral, V-model, and big bang models.
- Software engineering is extremely complex and expensive work, with large software systems costing more than buildings and often having high failure rates.
- The two main factors that cause "runaway" software projects that exceed budgets and schedules are poor estimation done too early and unstable requirements that change frequently.
- Programmers are often given impossible tasks with too much work and not enough time, leading them to produce workarounds and quick fixes rather than well-designed solutions.
The document discusses several security-related topics including promoting the OWASP Orange Saft tool, outcomes from a security guidance stakeholder meeting, feedback for improving security guidance in IDEs, topics to cover in a new CISO guide, questions to include in the guide, securing GitHub integration, an incident response playbook, and a CISO round table discussion. It also summarizes outcomes from several breakout groups at an OWASP event on threat modeling, application security curriculum design, and infosec warranties and guarantees.
The document provides an overview of the Software Development Life Cycle (SDLC) and popular software development methodologies. It describes the SDLC model which includes requirements analysis, design, coding, testing, and maintenance. It also summarizes three other models: the prototyping model which uses iterative prototyping and customer feedback; the Rapid Application Development (RAD) model which emphasizes short development cycles and component reuse; and the component assembly model which develops software from reusable components.
Unlocking Engineering Observability with advanced IT analyticssource{d}
In this webinar, source{d} CEO Eiso Kant will introduce source{d} Enterprise Edition (EE), the data platform for the software development life cycle (SDLC), With built-in visualization, management capabilities and advanced analytic functions, source{d} EE provide IT executives with visibility into their software portfolio, engineering processes and workforce.
Learn how source{d} EE can help everyone in the IT organization to quickly get access to customizable analytic solutions for IT modernization and software compliance, cloud-native and DevOps transformation, engineering effectiveness, and talent management.
Git into the Flow, with the Ultimate Continuous Delivery Workflow on HerokuSalesforce Developers
Any suspicion that Linus Torvalds was a Linux one-hit-wonder was dispelled when he released the Git distributed versioning system. Git is a popular source code management tool, and sophisticated software delivery flows are now built around its powerful branching model. Join us to learn how to leverage Git and GitHub for maximum delivery velocity, and for an introduction to how Heroku GitHub Integration, Review Apps, and Pipelines let you deliver software with ease and confidence.
- Stefan Streichsbier is the CEO of GuardRails and a professional white-hat hacker who has identified severe shortcomings in security processes and technologies, leading him to create GuardRails.
- The document discusses the evolution of DevOps and increasing complexity, the state of security and how it needs to fit within modern development workflows, and introduces the concept of DevSecOps to address shortcomings and better integrate security.
- Key aspects of DevSecOps discussed include how to create, test, and monitor secure applications and empower development teams to build security in from the start rather than see it as a separate function. Automated security tools and the need to reduce noise and improve usability for developers is also
How to Decide Technology Stack for Your Next Software Development Project?Polyxer Systems
Selecting the Perfect Tech Stack: Avoid costly mistakes & optimize performance. Learn how to pick the ideal technology stack for your project with our expert insights.
Creating a DevOps Practice for Analytics -- Strata Data, September 28, 2017Caserta
Over the past eight or nine years, applying DevOps practices to various areas of technology within business has grown in popularity and produced demonstrable results. These principles are particularly fruitful when applied to a data analytics environment. Bob Eilbacher explains how to implement a strong DevOps practice for data analysis, starting with the necessary cultural changes that must be made at the executive level and ending with an overview of potential DevOps toolchains. Bob also outlines why DevOps and disruption management go hand in hand.
Topics include:
- The benefits of a DevOps approach, with an emphasis on improving quality and efficiency of data analytics
- Why the push for a DevOps practice needs to come from the C-suite and how it can be integrated into all levels of business
- An overview of the best tools for developers, data analysts, and everyone in between, based on the business’s existing data ecosystem
- The challenges that come with transforming into an analytics-driven company and how to overcome them
- Practical use cases from Caserta clients
This presentation was originally given by Bob at the 2017 Strata Data Conference in New York City.
This DevOps CTO Masterclass covers DevOps tools, methodologies, and principles. The presentation introduces DevOps and its history, then discusses when DevOps is needed through a case study of a company that implemented DevOps to improve their development process. The remainder of the presentation covers DevOps practices for various stages including planning, coding, building, testing, deploying, operating, and monitoring. Key takeaways are to plan and communicate, automate processes, and continuously improve.
Maruti Gollapudi has over 17 years of experience as a principal architect, specializing in digital customer experience. Some of his significant contributions include developing a data aggregation and analytics platform hosted on AWS that enables capabilities like social analytics, text analytics using NLP and machine learning, and enterprise search. He has experience building solutions leveraging technologies such as Java, JBoss, Kafka, MongoDB, Solr, Watson, and various analytics and social APIs. Recent projects include developing a headless CMS for page building and dynamic content modification for CNBC, and architecting a middleware for CNBC's integration with Uber to dynamically serve ride-related content.
The document discusses the role of the modern software architect. It provides definitions of a software architect as someone who makes high-level design choices and dictates technical standards. The main responsibilities of an architect are to limit development choices by choosing standards and frameworks and communicating designs to developers. The document also discusses how the role of the architect changes with the size of the organization and types of architectures like enterprise, solution, and application architects. It notes challenges with more agile development where architecture may not receive focus and issues like technical debt can increase over time.
Integrated Analysis of Traditional Requirements Engineering Process with Agil...zillesubhan
In the past few years, agile software development approach has emerged as a most attractive software development approach. A typical CASE environment consists of a number of CASE tools operating on a common hardware and software platform and note that there are a number of different classes of users of a CASE environment. In fact, some users such as software developers and managers wish to make use of CASE tools to support them in developing application systems and monitoring the progress of a project. This development approach has quickly caught the attention of a large number of software development firms. However, this approach particularly pays attention to development side of software development project while neglects critical aspects of requirements engineering process. In fact, there is no standard requirement engineering process in this approach and requirements engineering activities vary from situation to situation. As a result, there emerge a large number of problems which can lead the software development projects to failure. One of major drawbacks of agile approach is that it is suitable for small size projects with limited team size. Hence, it cannot be adopted for large size projects. We claim that this approach can be used for large size projects if traditional requirements engineering approach is combined with agile manifesto. In fact, the combination of traditional requirements engineering process and agile manifesto can also help resolve a large number of problems exist in agile development methodologies. As in software development the most important thing is to know the clear customer’s requirements and also through modeling (data modeling, functional modeling, behavior modeling). Using UML we are able to build efficient system starting from scratch towards the desired goal. Through UML we start from abstract model and develop the required system through going in details with different UML diagrams. Each UML diagram serves different goal towards implementing a whole project.
Large scale agile development practicesSkills Matter
This document describes the experience of a large team developing and maintaining a large scale C# and SQL application over 15+ years using agile practices. Key aspects included:
- A team of 60 developers maintaining over 10 million lines of code
- Strict consistency enforced through architecture, naming conventions, patterns, and over 343,000 automated tests
- Continuous integration and distributed testing on developer machines to run the large test suite regularly
- A focus on reducing technical debt and making improvements that benefit all developers
Software Analytics:Towards Software Mining that Matters (2014)Tao Xie
This document discusses software analytics and summarizes several related papers and projects. It introduces Software Analytics, which aims to enable software practitioners to perform data exploration and analysis to obtain useful insights. It then summarizes papers on techniques for performance debugging by mining stack traces, scalable code clone analysis, incident management for online services, and using games to teach programming.
Evolution of the Software Development Lifecycle and DevOps ToolsCatalyst Investors
The document discusses the evolution of the software development lifecycle (SDLC) and DevOps tools. It describes how rapid increases in processing power necessitated more agile development approaches like iterative development and continuous integration. This prompted a proliferation of DevOps tools to facilitate new processes. Factors like cloud adoption, security integration, and diverse use cases are driving continued growth in the DevOps tools market. The document analyzes the competitive landscape and advises evaluating tools based on integration capabilities, ease of adoption, ROI demonstration, and cross-functionality.
This document introduces a DevSecOps maturity model to help organizations assess their current DevSecOps practices and plan their journey to more advanced practices. The model outlines four stages of maturity across six competency areas related to the development lifecycle. It also describes an online self-assessment tool that organizations can use to determine their current maturity level and identify areas for improvement. The model is intended to help leaders answer three key questions: where their organization is now, where they want it to be, and how to get there to advance their DevSecOps capabilities.
How to Choose the Right Tech Stack for Your Development Project?VlinkInfo
VLink’s dedicated developers can help you analyze, assess, choose, and execute your project’s best possible tech stack, putting you on the path to a successful software product launch.
This document provides an overview of agile software development and extreme programming (XP). It discusses how agile methods aim to rapidly develop and deliver working software through an iterative process with customer collaboration. Key aspects of XP are described, including planning with user stories, small incremental releases, test-driven development, pair programming, collective code ownership, and continuous integration. The document contrasts plan-driven and agile development approaches and outlines some principles and practices of XP such as simple design, refactoring, and sustainable pace of work.
Ludmila Orlova HOW USE OF AGILE METHODOLOGY IN SOFTWARE DEVELO.docxsmile790243
Ludmila Orlova
HOW USE OF AGILE METHODOLOGY IN SOFTWARE DEVELOPMENT INFLUENCE AGILITY OF THE BUSINESS
Agile methodology is widely distributed tool for software development. Presented article explore research data about use of these tools, its influence to quality of the end product and performance of development and overall agility of business and companies.
KEYWORDS:
Agile, software development, agile business
CONTENT
1 INTRODUCTION
2 AGILE SOFTWARE DEVELOPMENT
3 SCALING AGILE
4 AGILE BUSINESS
5 CONCLUSION
REFERENCES
1 INTRODUCTION
Fast pace of science progress in solid state electronics led to incredible progress of computer devices that on its turn demanded software to control and manage the power of computer calculations and usage.
Software engineering emerged in the beginning of 20th century and by the end of it became separate state of art science, activity and the profession for millions. There are about 18.2 million software developers worldwide, a number that is due to rise to 26.4 million by 2019, a 45% increase, says Evans Data Corp. in its latest Global Developer Population and Demographic Study (P. Thibodeau, 2013). Along with growing number of software developers (software development firms, projects and people involved), increased the need for effective management of software development process. This demanded new approach and methodology from business researchers and managers. In the last several decades there was huge number of research, both in IT field and business management dedicated to this area.
Popularity of agile software development methods started about decade ago and at present these methods are employed by many big, medium size and small companies. Still growing attention to agile methods from software development specialists confirm these methods filled the lack of management techniques for software development that emerged and developed extremely fast along with speedy advancement of hardware in IT area. Great number of research done in areas such as changes in performance of software development using agile methods or scaling agile for large companies and teams. Also one of modern trends is an attempt to apply agile methodology for project management, marketing, sales and other activities. Goal of this article is to explore influence of application agile methods in software development to agility of whole company and business. Presented work based on secondary data taken from a multiple sources, the work performed as an exploratory study and a review of existing research in the area.
2 AGILE SOFTWARE DEVELOPMENT
Definition of an adjective agile in English is: able to move quickly and easily or able to think and understand quickly (Oxford Dictionary, 2015). The most often contemporary use presented by the following sentence: Relating to or denoting a method of project management, used especially for software development, that is characterized by the division of tasks into ...
Similar to ICONIQ Analytics: The Modern Developer Technology Stack (20)
State of Artificial intelligence Report 2023kuntobimo2016
Artificial intelligence (AI) is a multidisciplinary field of science and engineering whose goal is to create intelligent machines.
We believe that AI will be a force multiplier on technological progress in our increasingly digital, data-driven world. This is because everything around us today, ranging from culture to consumer products, is a product of intelligence.
The State of AI Report is now in its sixth year. Consider this report as a compilation of the most interesting things we’ve seen with a goal of triggering an informed conversation about the state of AI and its implication for the future.
We consider the following key dimensions in our report:
Research: Technology breakthroughs and their capabilities.
Industry: Areas of commercial application for AI and its business impact.
Politics: Regulation of AI, its economic implications and the evolving geopolitics of AI.
Safety: Identifying and mitigating catastrophic risks that highly-capable future AI systems could pose to us.
Predictions: What we believe will happen in the next 12 months and a 2022 performance review to keep us honest.
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeWalaa Eldin Moustafa
Dynamic policy enforcement is becoming an increasingly important topic in today’s world where data privacy and compliance is a top priority for companies, individuals, and regulators alike. In these slides, we discuss how LinkedIn implements a powerful dynamic policy enforcement engine, called ViewShift, and integrates it within its data lake. We show the query engine architecture and how catalog implementations can automatically route table resolutions to compliance-enforcing SQL views. Such views have a set of very interesting properties: (1) They are auto-generated from declarative data annotations. (2) They respect user-level consent and preferences (3) They are context-aware, encoding a different set of transformations for different use cases (4) They are portable; while the SQL logic is only implemented in one SQL dialect, it is accessible in all engines.
#SQL #Views #Privacy #Compliance #DataLake
Global Situational Awareness of A.I. and where its headedvikram sood
You can see the future first in San Francisco.
Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. Every six months another zero is added to the boardroom plans. Behind the scenes, there’s a fierce scramble to secure every power contract still available for the rest of the decade, every voltage transformer that can possibly be procured. American big business is gearing up to pour trillions of dollars into a long-unseen mobilization of American industrial might. By the end of the decade, American electricity production will have grown tens of percent; from the shale fields of Pennsylvania to the solar farms of Nevada, hundreds of millions of GPUs will hum.
The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word. Along the way, national security forces not seen in half a century will be un-leashed, and before long, The Project will be on. If we’re lucky, we’ll be in an all-out race with the CCP; if we’re unlucky, an all-out war.
Everyone is now talking about AI, but few have the faintest glimmer of what is about to hit them. Nvidia analysts still think 2024 might be close to the peak. Mainstream pundits are stuck on the wilful blindness of “it’s just predicting the next word”. They see only hype and business-as-usual; at most they entertain another internet-scale technological change.
Before long, the world will wake up. But right now, there are perhaps a few hundred people, most of them in San Francisco and the AI labs, that have situational awareness. Through whatever peculiar forces of fate, I have found myself amongst them. A few years ago, these people were derided as crazy—but they trusted the trendlines, which allowed them to correctly predict the AI advances of the past few years. Whether these people are also right about the next few years remains to be seen. But these are very smart people—the smartest people I have ever met—and they are the ones building this technology. Perhaps they will be an odd footnote in history, or perhaps they will go down in history like Szilard and Oppenheimer and Teller. If they are seeing the future even close to correctly, we are in for a wild ride.
Let me tell you what we see.
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...Social Samosa
The Modern Marketing Reckoner (MMR) is a comprehensive resource packed with POVs from 60+ industry leaders on how AI is transforming the 4 key pillars of marketing – product, place, price and promotions.
The Building Blocks of QuestDB, a Time Series Databasejavier ramirez
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Discussion on Vector Databases, Unstructured Data and AI
https://www.meetup.com/unstructured-data-meetup-new-york/
This meetup is for people working in unstructured data. Speakers will come present about related topics such as vector databases, LLMs, and managing data at scale. The intended audience of this group includes roles like machine learning engineers, data scientists, data engineers, software engineers, and PMs.This meetup was formerly Milvus Meetup, and is sponsored by Zilliz maintainers of Milvus.
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...Aggregage
This webinar will explore cutting-edge, less familiar but powerful experimentation methodologies which address well-known limitations of standard A/B Testing. Designed for data and product leaders, this session aims to inspire the embrace of innovative approaches and provide insights into the frontiers of experimentation!
3. 3
Executive Summary – Project Overview (1 of 2)
The following is an in-depth study of the modern developer tech stack, in which we break down the DevOps
lifecycle into six distinct phases, each with its own set of tools
DevOps Lifecycle: Seven Tool Categories Explored in Depth
Example
Tools
Project Management
Tools used to track and manage project flow within and across teams
Code
Management
CI/CD Monitoring Defense
Development Review Monitoring & Security
Tools that enable the
writing, design and
building of software
Tools that help with the
review and testing of
code
Source code
management tools
Tools used to deploy
code; CI/CD
Tools that monitor
performance
Tools that let software
teams discover, triage
and fix errors and threats
Deployment
Development Verification
4. 4
Executive Summary – Project Overview (2 of 2)
For each of these lifecycle stages, we will examine high-level trends, followed by a drill-down on specific tools and
associated key top-of-mind questions
DevOps Lifecycle:
Key Questions & Concepts Explored by Lifecycle Stage
Machine Learning
Cloud & Containerization
Monitoring & Security
EngagementPurchase Retention
Familiarity of tools
Selection criteria
Tiers & Spend
Purchase
Usage frequency
Usage depth / penetration
Change in adoption
Price Elasticity
NPS
Mission criticality
Stickiness
Churn likelihood
Tool Decision Making Process and Usage: Concepts tested in survey
At End of Contract:
Renew vs. Switch?
+ Other Themes &
Trends Explored
5. 5
Executive Summary – Project Methodology
We’ve used 3 data sources for this project along with the guidance and perspectives of our Technical Advisory
Board - each providing a unique lens through which we can answer key questions
External survey fielded through a
panel of software companies, with
~200 respondents
Detailed survey teasing out high-level trends
related to the developer tech stack decision-
making process, followed by a drill-down on
specific tools and several open-ended topics
related to broader tech strategy
Note that this survey questionnaire was dynamic based on which tools
respondents used – n-sizes will vary by question and are noted as
relevant across slides
Survey and interviews across ICONIQ
Growth companies for additional
insights + validation
Focused on 3 key dimensions – spend,
satisfaction & other noteworthy trends
Additionally, gathered context around tool
selection, focusing on prior and existing pain
points
StackShare: Dataset with 250K+
registered developers reporting their
companies’ technology stacks
Comprehensive data on tech stacks across
hundreds of companies, allowing us to
gather industry-wide themes as well as
insights related to individual tools
performance / prevalence
+ G2Crowd for reviews and other various
related reports
ICONIQ Growth
Portfolio Survey
External Survey
Secondary
Research
StackShare, G2Crowd, Other
Perspectives from ICONIQ Growth Technical Advisory Board
Nate Walkingshaw
Pluralsight, CXO
(Tanner Labs, Stryker)
Jeff Rothschild
Facebook, Former VP
Infrastructure
(Mpath, Veritas)
Matt Eccleston
Dropbox, Former
VP Growth
(VMWare)
AdityaAgarwal
Dropbox, CTO
(Cove, Facebook)
AnanthaKancherla
Lyft, VP Engineering
(Dropbox, Facebook)
Keith Adams
Slack, Chief Architect
(Facebook, VMWare)
Nate Walkingshaw
Pluralsight, CXO
(Tanner Labs, Stryker)
Jeff Rothschild
Facebook, Former VP
Infrastructure
(Mpath, Veritas)
Matt Eccleston
Dropbox, Former
VP Growth
(VMWare)
AdityaAgarwal
Dropbox, CTO
(Cove, Facebook)
AnanthaKancherla
Lyft, VP Engineering
(Dropbox, Facebook)
Keith Adams
Slack, Chief Architect
(Facebook, VMWare)
Nate Walkingshaw
Pluralsight, CXO
(Tanner Labs, Stryker)
Jeff Rothschild
Facebook, Former VP
Infrastructure
(Mpath, Veritas)
Matt Eccleston
Dropbox, Former
VP Growth
(VMWare)
AdityaAgarwal
Dropbox, CTO
(Cove, Facebook)
AnanthaKancherla
Lyft, VP Engineering
(Dropbox, Facebook)
Keith Adams
Slack, Chief Architect
(Facebook, VMWare)
Nate Walkingshaw
Pluralsight, CXO
(Tanner Labs, Stryker)
Jeff Rothschild
Facebook, Former VP
Infrastructure
(Mpath, Veritas)
Matt Eccleston
Dropbox, Former
VP Growth
(VMWare)
AdityaAgarwal
Dropbox, CTO
(Cove, Facebook)
AnanthaKancherla
Lyft, VP Engineering
(Dropbox, Facebook)
Keith Adams
Slack, Chief Architect
(Facebook, VMWare)
Nate Walkingshaw
Pluralsight, CXO
(Tanner Labs, Stryker)
Jeff Rothschild
Facebook, Former VP
Infrastructure
(Mpath, Veritas)
Matt Eccleston
Dropbox, Former
VP Growth
(VMWare)
AdityaAgarwal
Dropbox, CTO
(Cove, Facebook)
AnanthaKancherla
Lyft, VP Engineering
(Dropbox, Facebook)
Keith Adams
Slack, Chief Architect
(Facebook, VMWare)
Nate Walkingshaw
Pluralsight, CXO
(Tanner Labs, Stryker)
Jeff Rothschild
Facebook, Former VP
Infrastructure
(Mpath, Veritas)
Matt Eccleston
Dropbox, Former
VP Growth
(VMWare)
AdityaAgarwal
Dropbox, CTO
(Cove, Facebook)
AnanthaKancherla
Lyft, VP Engineering
(Dropbox, Facebook)
Keith Adams
Slack, Chief Architect
(Facebook, VMWare)
Former CTO at Dropbox
(Cove, Facebook)
Chief Architect at Slack
(Facebook, VMWare)
Former VP Growth at
Dropbox (VMWare)
VP Engineering at Lyft
(Dropbox, Facebook)
Former VP Infrastructure at
Facebook (Mpath, Veritas)
CXO at Pluralsight
(Tanner Labs, Stryker)
6. 6
Executive Summary – Key Findings
In response to bottoms-up adoption and a proliferation of tools, we have seen a growing focus on organizational-
level security and integration, with code development and project mgmt. tools being central to architecture design
Key Project Findings: The Dev Stack Decision Making Process
The number of tools used by
developers has proliferated...
...And, project management and
code development tools have become
“anchor-points”
Resulting in a focus on security &
integration...
While tool adoption is generally driven
in a bottoms-up fashion, final selection
criteria continues to be defined by top-down
decision makers
Additionally, challenges and
organizational decisions related to
machine learning resource allocation
are top-of-mind
1
3
2
4
5
▪ As the number of available tools related to the code development process continues to explode, the focal point in
the design and assembly of technology stacks has evolved
▪ Emerging technology companies (potentially by nature of the agility required in high-growth stages) have evolved to
encourage the experimentation of new tools, with early adoption largely driven by a bottoms-up motion
▪ In particular, project management & code development are key tool categories around which the rest of the stack is built
▪ Project management tools are a critical conduit between product and business teams
▪ Development tools or IDEs (Integrated Development Environments), are intuitively central to any technology architecture as
they are typically where developers spend most time
▪ Because there are now many tools in the average stack with multiple potential points of failure, security has
become a top priority
▪ Concurrently, integration capabilities have become critical in order to effectively manage the overarching
architecture across disparate tools
▪ Integration capabilities are top-of-mind in the selection of project management tools while reliability is a close
second
▪ Although price is sometimes important, ROI time horizon is often more so, indicating an appreciation for the value that
can be driven by even some of the more expensive tools
▪ Machine Learning resource allocation is top-of-mind in a world where demand for this skillset has outpaced
growth in the necessary talent pool
▪ Most companies currently have in-house ML teams and capabilities, while a smaller subset outsource machine
learning needs on an ad-hoc basis as their primary approach
7. 7
Executive Summary – Tools Proliferation
Given low experimentation costs, companies have started to include an increasing number of tools in their
developer stacks; however, more mature companies tend to have consolidated stacks
1
Number of Tools Used by DevOps Team
StackShare | Example Companies | Includes Business Tools
“I think it’s a huge mistake to try and use a one-size-fits-all tool
across different use cases, even within departments… each of
our tools are great at the highly specific thing we use them
for, but wouldn’t be as effective if we tried to stretch them
across secondary capabilities.”
– Decision maker at enterprise SaaS company
“We’re at the cutting edge of a rapidly evolving space,
which means a lot of experimentation.”
– Director of Engineering at HyperScience
“We are constantly evaluating the best tools for the job
to make sure our tech stack allows us to maintain
a great product for our users.”
– Sr. Engineer at OkCupid
9
19
20
23
27
28
28
33
35
35
35
36
37
38
40
40
45
53
55
59
59
74
Apple
Stripe
Google
Docker
Datadog
Zendesk
Microsoft
Amazon
Shopify
Sentry
Pluralsight
Facebook
Heroku
Chime
Square
GitLab
LaunchDarkly
Instacart
Airbnb
Asana
Uber
Delivery Hero
“Legacy” tech giants have relatively fewer tools in
their developer stacks – potentially driven by stack
construction at a time with more limited options and also
likely the result of an NIH culture and having the
resources to build more solutions and tools in-house
Note: We explored an average
of 10+ tools per respondent in-
depth in our external survey
“There is a balance, of course, but I still strongly believe in a
best-in-breed approach – The cost of having to change your
tool stack and the underlying data down the road is just
too high not to choose the best one from the get-go.”
– Decision maker at enterprise SaaS company
Deep Dive Into Select Companies’ Stacks on Page 63
Sources: StackShare, ICONIQ Analytics & Research
8. 8
Executive Summary – Stack Design Considerations
As a result, security has overwhelmingly become the most important consideration; not only was it most
important, respondents indicated willingness to compromise efficiency and scalability to ensure security
45%
39%
38%
34%
33%
32%
30%
25%
24%
Security / Compliance
Integration with other solutions
Speed of deployment
Maintainability
Team expertise
Cross-platform compatibility
Scalability
Cost
Type of project
Most Important Design Criteria Most Prevalent Themes From
Trends Noted
8
4
2 2
1 1 1 1
Security/
Compliance
Speed/
Efficiency
Artificial
Intelligence/
ML
LowCode
DevTools
Analytics
CI/CD
Design
IOT
“We now focus first on security because
of previous threats but also focus on
maintaining efficiency as well when possible.”
“Security first and foremost, efficiency later
even at the expense of increased cost.”
“Security takes priority over efficiency as it
will be easier to find vulnerabilities now rather
than in the field.”
Select Survey Quotes
2
“Our top concern is keeping our system
updated to new threats without disrupting our
end users' work. To address this, we run as
much security as we can as deep in the
background as we can, schedule patches
and updates during times of low use in our
system, and support our users when there
is a conflict.”
In addition to the proliferation of tools, another driver
of the increased focus on secure tools may be
shortage in cybersecurity talent
-- IBM cybersecurity study
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200, primary interviews + secondary research
9. 9
% Respondents
With Tools
% Rating as
Central to
Architecture
Primary tool around
which the stack
was designed
Mission
Criticality
Average of tools rated
“high mission critical”
Executive Summary – Overall Stack
Project management & code development tools are central to stack architecture in a landscape experiencing
rapid growth in tools; while code deployment and security tools are largely ubiquitous, they tend to be secondary
from a decision-making standpoint
Project
Management
Code
Development
Code Review Code
Deployment
Code Security
78% 82% 84% 79% 86%
33% 33%
17% 10% 7%
60% 60% 50% 40% 50%
21
3
Project Management & Code
Development tools serve as anchor
points in a landscape that has
experienced rapid proliferation of tools.
Management tools in particular are critical
as the conduit between product and
business teams.
Development tools - where the engineering
team spends the bulk of their time - are
also a key decision point.
1
Code deployment tools, by virtue of
being used last in the code building
process, seem to take the back seat in
terms of the decision-making process.
With CI/CD methods, teams also tend to
retro-fit existing toolsets vs. proactively
design stacks around deployment tools.
2
While security across tools remain a key
priority, code security tools as a
category are often selected once the
other key pieces are in place, likely
driven by the fact that this is generally the
last phase of the development cycle.
3
3
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
10. 10
Executive Summary – Strategic Considerations: Machine Learning Resources
Generally, companies increasing deploy small teams of Machine Learning engineers dedicated to each function
as they scale; outsourcing is a helpful way for some companies to supplement resources on an ad-hoc basis
How are companies thinking about their
broader machine learning strategy?
➢ Most companies are currently using insourced
machine learning capabilities
➢ Some teams (~5-10%) use outsourced ML talent
as their primary resource
➢ Between $10M and $250M, companies
increasing deploy small teams of ML engineers
for each function; much larger companies
($250M+), however, switch to have dedicated teams
that operate cross-functionally
Machine Learning Resource Organization
Which of the following best describes your organization's strategy as it pertains to machine
learning? Select the one that best fits your situation.
In-House Teams by Function
Maintaining a small team of machine
learning engineers for each function
Shared In-House Team
Maintaining a dedicated team of machine
learning engineers that operate across all
facets of the organization
Machine Learning Resource Allocation
5
29N 33N 23N 23N 55N
50%
25%
44%
69%
39%
44%
65%
44%
25%
52%
6% 10% 11%
6% 9%
$2 to $10M $10 to $50M $50 to $100M $100 to $250M $250M+
Ad-Hoc Outsourcing
Using outsourced machine learning
engineers on an Ad Hoc basis
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
By Company Scale (Revenue)
11. 11
Executive Summary – By Tool Type
Decisions around tools selection are often driven by factors idiosyncratic to the tool category; sometimes these
are driven by scale, by coding language or by codebase environment
DevOps Lifecycle: Key Themes & Findings
Top Tools -
External Survey
(methodology on next slide)
Project Management
Code Management CI/CD Monitoring Defense
Development Review Monitoring & SecurityDeployment
Development Verification
Companies generally start with generic project management tools while processes are amorphous, but add more developer-specific tools at scale
A
B C D E F G
Traditional, free tools
remain universal,
while some teams
are starting to use
low code tools –
choice largely driven
by programming
language
Given importance of
integration at this
phase, tools that
span multiple pieces
of the DevOps
lifecycle fare
particularly well
Tried-and-True
names and multi-
purpose tools that
span the DevOps
lifecycle are stronger
options
Integration drive
success of
ecosystem players
and multi-purpose
tools
Ability to aggregate
data from various
sources in a
distributed
infrastructure is key
Selection generally
dictated by codebase
architecture given
decision often comes
later in process; 2+
tools common to
ensure all needs
served
Further detail by tool category on subsequent slides
4
Most Commonly
Used Tool - ICONIQ
Growth Portfolio
Varied
Top Tools External Survey
(methodology on next slide)
Most Commonly Used Tool -
ICONIQ Growth Portfolio
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200; ICONIQ Growth Portfolio Survey, n = 20
12. 12
Executive Summary – Tools Scoring Methodology
In order to distill an objective view of ‘top tools’ by category, we used a combination of various metrics to calculate
a composite score, including brand awareness, adoption, satisfaction, engagement and retention likelihood
Churn
Seat Penetration
% Daily Users
NPS (Scores 9 or 10)
Prevalence – Usage
Familiarity
We used surveyed metrics along the
purchase lifecycle (purchase, engage,
retain) to derive a composite score for
each tool:
✓ Brand awareness | Familiarity
✓ Adoption | Prevalence
✓ Satisfaction | NPS
✓ Engagement | % Daily & Seat Penetration
✓ Retention likelihood | Churn propensity
Tool Scoring Methodology
ICONIQ Analytics External Dev. Stack Survey Rankings
ICONIQ
Composite Score
By tool category by vendor
Score based on average ranking
across metrics from ICONIQ
External Dev. Stack Survey
4
13. 13
Executive Summary – Project Management
Companies typically start with generic project management tools with built-in flexibility (e.g., Smartsheet) but
transition to more robust tools specific to the developer process and team as they scale (e.g., Jira)
Top Tools
& Metrics
Top
Selection
Criteria
Project Management
Integration Capabilities & ROI Time Horizon
4
Differences by Company Scale
Smartsheet is more popular among smaller companies
given amorphous processes at earlier stages easier to handle
in generic, self-defined tool
Larger companies exhibit deeper engagement vs. other
scale buckets – in terms of % daily users
Key Takeaways
▪ Integration capabilities are of key importance as project
management tools serve as an anchor point in a
landscape that has experienced rapid proliferation of tools
▪ Although price is sometimes important, ROI time horizon
is often more important, indicating buyers appreciate
the value these tools can bring, even at higher price
points
Familiarity Prevalence NPS
9 or 10
% Daily
Seat
Penetration
Avg.
Ranking
% Respondents
Project Management tools tend to be more critical component
of tech stack for larger scale vs. smaller scale
39% 31% 77% 50% 54%
46% 32% 63% 47% 49%
41% 27% 67% 40% 48%
41% 27% 55% 52% 53%
A
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
14. 14
Executive Summary – Development
Coding language drives selection of code development tools (vs. company maturity); traditional, free development
tools remain universal, while low code / no code tools continue to emerge as a parallel category
Top Tools
& Metrics
Top
Selection
Criteria
Development Tools
Integration Capabilities & ROI Time Horizon
4
Avg.
Ranking
36% 26% 48% 60% 57%
46% 35% 61% 35% 49%
28% 22% 75% 42% 51%
Differences by Scale
Key Takeaways
Choice of development tool more likely to be driven by
programming language (Mobile, C++, etc.)
While integration remains top of mind for companies in
their selection of development tools, pricing structure &
contract flexibility start to matter as companies scale
Free traditional tools such as Notepad++ and
Visual Studio Code remain universally the
most prevalent code development tools
Some teams are starting to implement low code
/ no code tools such as Appian and Appsheet
to improve deployment speed
Betty Blocks, a fully “enterprise-grade” no-code
tool somewhat more popular among smaller
companies
Familiarity Prevalence NPS
9 or 10
% Daily
Seat
Penetration
B
% Respondents
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
15. 15
Executive Summary – Verification
While Codacy ranks highest in this category, multi-dimensional tools like GitLab also fare well given the
importance of integration for code verification tools
Top Tools
& Metrics
Top
Selection
Criteria
Verification
Integration Capabilities & ROI Time Horizon
4
Differences by Scale
Key Takeaways
Given importance of integration capabilities for code
verification tools, companies are willing to use multi-purpose
tools such as GitLab despite shortcomings in certain areas
Avg.
Ranking
While integration remains top of mind for companies in
their selection of verification tools, pricing structure &
contract flexibility start to matter as companies scale
31% 24% 75% 48% 59%
40% 32% 62% 36% 49%
24% 15% 58% 50% 49%
Codacy and GitLab are within the top 3 tools
for most scale buckets in terms of % of
respondents using; however GitLab generally
has lower engagement scores amongst $10-$50M
bucket, but remains a top tool overall
GitLab has the highest overall prevalence in
developer stacks, propelled by those who
prioritize reliability and integration – criteria
across which GitLab ranks particularly well
Familiarity Prevalence NPS
9 or 10
% Daily
Seat
Penetration
C
% Respondents
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
16. 16
Executive Summary – Code Management
Given code management space is generally more consolidated than other tool groups, awareness is a key driver
with GitHub being the top used and known tool; GitLab follows closely given consolidation synergies
Top Tools
& Metrics
Top
Selection
Criteria
Code Management
Integration Capabilities & ROI Time Horizon
4
Avg.
Ranking
35% 27% 60% 40% 47%
20% 14% 55% 50% 50%
19% 10% 40% 20% 41%
Differences by Scale
Key Takeaways
Given less fragmentation in the code management
ecosystem, top vendors fare much better than those less
well known
▪ Pricing structure & contract flexibility are more important for
smaller companies
▪ Bitbucket ranked higher for smaller companies vs. larger
companies
GitHub, most well-known in the code
management space, has both the most users
and good NPS / G2 review scores...
...However, GitLab comes close due to
consolidation synergies with other parts of the
DevOps lifecycle
Bitbucket was paired with the most other code
management tools – implying functionality gaps
that need other tools to supplement
Familiarity Prevalence NPS
9 or 10
% Daily
Seat
Penetration
D
% Respondents
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
17. 17
Executive Summary – CI/CD
In the CI/CD space, integration drives strength of both ecosystem players, such as AWS, and tools that serve
multiple functions across the DevOps lifecycle such as GitLab
Top Tools
& Metrics
Top
Selection
Criteria
CI / CD
Integration Capabilities & ROI Time Horizon
4
Avg.
Ranking
Differences by Scale
Key Takeaways
Integration drives strength of both ecosystem players,
such as AWS, and tools that serve multiple functions
within the DevOps lifecycle
▪ Pricing structure & contract flexibility are more important for
smaller companies
▪ Generally, larger companies have a higher base of serious daily
users; indicating decision makers are probably more intentional in
their purchases / sign ups
AWS CodeDeploy has the highest overall
prevalence in developer stacks, propelled by a
high % of companies using AWS as their cloud
provider – cohort that has over-indexed affinity
toward AWS CodeDeploy for CI/CD
However, GitLab comes close due to
consolidation synergies with other parts of the
DevOps lifecycle
38% 27% 60% 37% 49%
20% 14% 55% 50% 50%
24% 15% 52% 39% 47%
Familiarity Prevalence NPS
9 or 10
% Daily
Seat
Penetration
E
% Respondents
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
18. 18
Executive Summary – Monitoring
For monitoring tools, ability to unify logs, metrics, and traces from across one’s distributed infrastructure is key;
this makes Datadog – a tool with 200+ integrations – the preferred choice
Top Tools
& Metrics
Top
Selection
Criteria
Monitoring
Integration Capabilities & ROI Time Horizon
4
Avg.
Ranking
Differences by Scale
Key Takeaways
For monitoring tools, the ability to unify logs, metrics, and
traces from across one’s distributed infrastructure is key
▪ Scale does not seem to drive selection of code monitoring tools
in a meaningful way, despite slightly different selection criteria
▪ Customer service especially important for larger companies while
integration capabilities & ROI more important for smaller scale
buckets
Code monitoring tools generally have low
engagement scores in terms of % using daily;
somewhat expected given predominantly passive
nature of involvement
Nonetheless, Datadog’s engagement metric is
2x that of the next best (Kibana)
“Monitoring for many apps is the top reason
developers like Datadog” – StackShare Community
Familiarity Prevalence NPS
9 or 10
% Daily
Seat
Penetration
31% 25% 59% 41% 48%
17% 12% 61% 22% 39%
16% 11% 53% 18% 54%
F
% Respondents
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
19. 19
Executive Summary – Defense
Selection of defense tools likely driven by the particular codebase environment the company operates in; most
companies have more than one defense tool given the disparate functions served (e.g., SIM vs. Code scanning)
Top Tools
& Metrics
Top
Selection
Criteria
Defense
Integration Capabilities
4
Avg.
Ranking
Differences by Scale
Key Takeaways
Selection of defense tools likely primarily driven by the particular
codebase environment the company operates in
Most companies, will have 2+ defense tools for both redundancy
and given slightly different capabilities across
this ‘best-of-breed’ group (e.g., SIM vs. Code Scanning)
Scale does not seem to drive selection of defense
tools in a meaningful way
PagerDuty and Lacework have the highest
proportion of daily users despite lagging in overall
prevalence
Familiarity Prevalence NPS
9 or 10
% Daily
Seat
Penetration
G
28% 24% 83% 53% 55%
30% 19% 68% 43% 47%
20% 12% 72% 44% 44%
Checkmarx is the most popular defense tool
across most companies; it is especially prevalent
amongst those that prioritize ROI time horizon
% Respondents
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
21. 21
Project Management Tools – Usage
JIRA and Asana are the most widely used project management tools within larger teams while Smartsheet is
disproportionately used by smaller companies
EngagementPurchase Retention
32% 31%
27% 27%
19% 19% 18% 17% 17% 17% 16% 15% 15% 14% 14%
12% 10%
2%
Smartsheet
Airtable
JIRA
Asana
Front
Workzone
Clubhouse
Miro
Monday.com
PivotalTracker
Trello
Hiver
Pipefy
VersionOne
Productboard
Planview
Targetprocess
Wrike
$2 to $10M (24N) 38% 29% 8% 4% 13% 21% 8% 13% 13% 4% 8% 17% 17% 25% 0% 13% 13% 4%
$10 to $50M (28N) 50% 14% 25% 29% 18% 11% 21% 14% 11% 18% 25% 14% 18% 7% 14% 11% 14% 0%
$50 to $100M (21N) 38% 52% 24% 29% 19% 24% 33% 29% 33% 29% 24% 19% 14% 19% 19% 24% 14% 10%
$100 to $250M (25N) 20% 36% 40% 32% 16% 24% 24% 20% 36% 28% 4% 16% 20% 12% 20% 8% 8% 0%
$250M+ (57N) 23% 30% 32% 33% 25% 18% 12% 16% 7% 12% 18% 14% 11% 11% 14% 9% 7% 0%
Usage by Company Scale (Annual Revenue) – Top 10 tools
Which of the following project / workflow management tools does your organization currently use? Base: 155
c
Ranked by prevalence within full cohort / all respondents
StackShare top 3 tool of 2019
within “Collaboration Tools”
StackShare top 3 tool of 2019
within “Collaboration Tools”
# of StackShare
companies using:
~1,700
# of StackShare
companies using:
~500
# of StackShare
companies using:
~1,400
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
22. 22
Project Management Tools – Selection Criteria & Top Tools
Integration capabilities are top-of-mind in the selection of project management tools while reliability is a close
second; price is often contextualized relative to value / productivity gains
EngagementPurchase Retention
Selection Criteria & Top Tools by Company Scale (Annual Revenue)
Top
Selection
Criteria
(Criteria
Selected as
Top 3 |
Above
Median
Shown)
Top
Tools
(Ranked)
$2 to $10M $10 to $50M $50 to $100M $100 to $250M $250+
Integration
Capabilities (67%)
Customer Reviews
(46%)
Price
(33%)
Smartsheet
Airtable
VersionOne
Integration
Capabilities (64%)
Reliability
(54%)
ROI Time Horizon
(39%)
Smartsheet
Asana
JIRA
ROI Time Horizon
(62%)
Integration
Capabilities (57%)
Customer Service
(38%)
Airtable
Smartsheet
Clubhouse &
Monday.com (tied)
ROI Time Horizon
(64%)
Integration
Capabilities (40%)
Reliability
(40%)
JIRA
Airtable
Monday.com
Integration
Capabilities (63%)
Reliability
(51%)
ROI Time Horizon
(40%)
Asana
JIRA
Airtable
Details on
following page
Key Takeaways
▪ Integration capabilities
are top-of-mind in the
selection of project
management tools while
reliability is a close second
▪ Although, price is
sometimes important, ROI
time horizon is often more
important, indicating
buyers contextualize
price relative to value /
productivity gains
▪ Smartsheet is popular
among smaller teams
while JIRA and Airtable
are most popular for
larger teams
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
23. 23
Ability to integrate 30% 32% 28% 28% 20% 20% 16% 18% 17% 17%
Pricing structure &
contract flexibility
28% 33% 17% 22% 15% 15% 15% 15% 13% 17%
Reliability 31% 21% 31% 33% 15% 13% 16% 13% 12% 13%
Return on investment
(ROI) time horizon
21% 39% 29% 30% 17% 19% 20% 19% 21% 24%
Project Management Tools – Selection Criteria & Top Tools
Airtable ranks very highly amongst respondents that prioritize integration, pricing structure & contract flexibility
and ROI time horizon; however, lacks in reliability; Asana, more than other tools, stood out in its reliability score
Top 3 Criteria Smartsheet Airtable JIRA Asana Front Workzone Clubhouse Miro Monday.com
Pivotal
Tracker
Overall % Using 32% 31% 27% 27% 19% 19% 18% 17% 17% 17%
Top Tools (% Respondent Using) by Selection Criteria
While Asana ranks decently in all
4 top criteria, it stands out in it’s
reliability
Airtable ranks very highly amongst respondents
that prioritize integration, pricing structure and ROI
time horizon; however, lacks in reliability
These 4 criteria (out of 9) are
uniformly the most important
selection criteria with a sharp drop
off after
EngagementPurchase Retention
Other Selection Criteria: Time to implement, customer service, customer reviews, price, user-friendliness
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
24. 24
Clubhouse has provided the best overall tool for
ticketing and project management without a steep
learning curve.
[Front] Integrations are on point and simple to setup
and manage. At Bento we have tried out numerous
integrations - from softphones to centralized helpdocs -
these have had varying degrees of success, but the part
that stays consistent is the ease of setup within Front.
I like that Jira combines multiple features to create
unique tools that help me to start a new project. The
Kanban boards are EXTREMELY helpful and the design
Jira provides is very aesthetic and easy to understand.
Project Management Tools – Tools Scorecard
In line with usage prevalence, Smartsheet and Airtable also score highly in engagement and satisfaction metrics
including positive reviews on G2
EngagementPurchase Retention
% Overall Users % Daily Users NPS1 (9 or 10) G22 Score (out of 5)
Smartsheet 32% 47% 81% 4.2
Airtable 31% 50% 78% 4.6
JIRA 27% 52% 77% 4.1
Asana 27% 40% 76% 4.3
Front 19% 30% 69% 4.6
Workzone 19% 34% 68% 4.3
Clubhouse 18% 32% 67% 4.4
Miro 17% 37% 67% 4.7
Monday.com 17% 38% 73% 4.5
Pivotal Tracker 17% 42% 69% 4.0
[Airtable] A software that undoubtedly enhances the
business's goals and strengthens its operation.
[Smartsheet] I like that it's a built in widget on Microsoft
Teams, which is our primary collaboration software in the
company.
Engagement & Satisfaction Metrics
N = 155 Respondents
User Reviews [G2]
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
Note: (1) NPS is calculated as the % respondents ranking likelihood to recommend tool as 9 or 10 (“promoters”) less the %
respondents ranking same question as a 6 or lower (“detractors”) x 100 (2) G2 score is average rank across reviewers on scale
of 1 (worst) to 5 (best)
25. 25
Project Management Tools – Usage Frequency
Productboard, JIRA and Airtable have the highest proportion of daily users – with 50%+ using daily; larger
companies have a higher base of serious daily users vs. those at smaller scales
EngagementPurchase Retention
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Productboard
JIRA
Airtable
Smartsheet
Pivotal Tracker
Asana
Trello
Monday.com
Targetprocess
Miro
Pipefy
Workzone
Planview
Hiver
Wrike
Clubhouse
Front
VersionOne
Daily Weekly Monthly Few times a year Ad Hoc
$2 to $10M $10 to $50M $50 to $100M $100 to $250M
N/A 25% 25% 100%
100% 29% 80% 70%
0% 25% 55% 89%
56% 50% 38% 60%
100% 20% 17% 71%
0% 25% 17% 50%
0% 14% 60% 100%
33% 0% 14% 67%
33% 0% 17% 60%
0% 20% 67% 60%
60% 33% 20% 33%
0% 25% 50% 0%
0% 33% 29% 33%
67% 0% 25% 50%
Usage Frequency
Usage Frequency (% Daily)
by Company Scale
Generally, larger
companies have a
higher base of serious
daily users; indicating
smaller teams may be
experimenting with
several tools
N
21
42
48
49
26
42
25
26
16
27
23
29
18
24
3
28
30
21
Small sample size
Small sample size
Small sample size
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
26. 26
NPS Distribution Scores 1 or 2 Scores 9 or 10
5% 81%
4% 78%
4% 77%
5% 76%
4% 73%
4% 69%
13% 69%
0% 68%
4% 67%
2% 67%
0% 67%
0% 63%
4% 63%
3% 62%
0% 60%
7% 60%
0% 56%
0% 55%
Project Management Tools – Satisfaction
VersionOne, Pipefy, Airtable, Productboard and Pivotal Tracker have the highest % of users rating 9 or 10 on the
NPS scale; Targetprocess stood out in its polarization of user base – with 13% of respondents rating 1 or 2
NPS Score 10
NPS Score 9
NPS Scores 3-8
NPS Scores 1 or 2
Ranked by
NPS Scores by Tool – All Responses
On a scale of one to ten, how likely are you to recommend the following software to a colleague or someone in your network? 10 being most likely.
EngagementPurchase Retention
VersionOne
Pipefy
Airtable
Productboard
Pivotal Tracker
Monday.com
Targetprocess
Clubhouse
Hiver
Asana
Wrike
Smartsheet
Miro
Workzone
Trello
Front
Planview
JIRA
Small sample size
N
21
23
48
21
26
26
16
28
24
42
3
49
27
29
25
30
18
42
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
Note: NPS is calculated as the % respondents ranking likelihood to recommend tool as 9 or 10 (“promoters”) less the %
respondents ranking same question as a 6 or lower (“detractors”) x 100
27. 27
Project Management Tools – Churn Propensity
Clubhouse, Workzone and Productboard have 10%+ of its users indicating that they’d churn in 3-5 years; smaller
companies are unlikely to reduce their use of project management software significantly
Churn Propensity by Tool
How is your organization's adoption of the following project management software likely to change 3-5 years from now? – Decrease significantly or stop entirely
EngagementPurchase Retention
ALL $2 to $10M $10 to $50M $50 to $100M $100 to $250M $250M+
Clubhouse 11% 0% 0% 14% 17% 14%
Workzone 10% 0% 0% 17% 10%
Productboard 10% 0% 0% 0% 25%
Pivotal Tracker 8% 0% 0% 0% 14% 14%
Targetprocess 6%
VersionOne 5% 0% 0% 0% 33% 0%
Airtable 4% 0% 0% 0% 0% 12%
Hiver 4% 0% 0% 0% 0% 13%
JIRA 2% 0% 0% 0% 10% 0%
Smartsheet 2% 0% 0% 13% 0% 0%
Monday.com 0% 0% 0% 0% 0% 0%
Front 0% 0% 0% 0% 0% 0%
Miro 0% 0% 0% 0% 0% 0%
Asana 0%
Planview 0% 0% 0% 0% 0% 0%
Trello 0% 0% 0% 0% 0% 0%
Wrike 0%
Pipefy 0% 0% 0% 0% 0% 0%
Within the groups where a
significant number of users
plan to reduce use adoption
significantly, the patterns are
relatively similar across tools
Smaller companies are
unlikely to reduce their
use of project mgmt.
software significantly
Note: Data points with very small sample size removed where applicable
Small sample size
Small sample size
Small sample size
N
28
29
21
26
16
21
48
24
42
49
26
30
27
42
18
25
3
23
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
30. 30
Code Development Tools – Usage
Notepad++, Visual Studio Code and AQTime are the most widely used code development tools; Appsheet and
Appian are among the most prevalent low code / no code tools
EngagementPurchase Retention
$2 to $10M (29N)
$10 to $50M (33N)
$50 to $100M (23N)
$100 to $250M (23N)
$250M+ (55N)
Usage by Company Scale (Annual Revenue) – Top 10 tools
Which of the following code development tools does your organization currently use? Base: 163N
StackShare rated top 3 tool of 2019
within “Build, Test & Deploy Tools”
Low Code /
No Code
35%
26%
22%
17%
14%
12%
8% 7%
5%
2%
33%
28%
17%
6%
Notepad++
VisualStudioCode
AQTimePro
Atom
Collaborator
Emacs
SwaggerHub
SublimeText
Repl.it
Vim
Appsheet
Appian
BettyBlocks
Skuid
48% 21% 24% 10% 10% 7% 10% 10% 10% 3% 21% 24% 17% 0%
33% 27% 18% 15% 9% 9% 9% 6% 0% 0% 27% 21% 12% 9%
48% 17% 26% 39% 22% 22% 9% 13% 13% 4% 39% 26% 26% 9%
4% 30% 30% 17% 4% 13% 9% 4% 0% 0% 57% 43% 26% 9%
36% 29% 18% 13% 20% 11% 5% 5% 4% 4% 31% 27% 11% 4%
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
31. 31
Code Development Tools – Selection Criteria & Top Tools
Integration capabilities are top-of-mind in the selection of code development tools; pricing structure, contract
flexibility and customer reviews and service matter more for these tools than they do for project management ones
EngagementPurchase Retention
Selection Criteria & Top Tools by Company Scale (Annual Revenue)
Top
Selection
Criteria
(Criteria
Selected as
Top 3 |
Above
Median
Shown)
Top
Tools
(Ranked)
$2 to $10M $10 to $50M $50 to $100M $100 to $250M $250M+
Integration
Capabilities (55%)
Price (41%)
ROI Time Horizon
(38%)
Notepad++
Appian
AQTIme Pro
Reliability (52%)
Integration
Capabilities (49%)
Customer Service
(39%)
Notepad++
Appsheet
Visual Studio Code
ROI Time Horizon
(52%)
Integration
Capabilities (48%)
Customer Reviews
(35%)
Notepad++
Appsheet
Atom
Integration
Capabilities (57%)
Reliability (39%)
Pricing Structure &
Contract Flexibility
(39%)
Appsheet
Appian
Visual Studio Code
AQTime Pro
Integration
Capabilities (55%)
Pricing Structure &
Contract Flexibility
(45%)
Reliability
(45%)
Notepad++
Appsheet
Visual Studio Code
Details on
following page
Key Takeaways
▪ Integration capabilities
are top-of-mind in the
selection of code
development tools
▪ Although, price is
sometimes important, ROI
time horizon is often as
important, indicating
buyers contextualize
price relative to value /
productivity gains
▪ Notepad++ is the top tool
for most scale buckets;
Appsheet and Appian are
close contenders across
the board
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
32. 32
Pricing structure &
contract flexibility
Reliability
Ability to integrate
Return on investment
(ROI) time horizon
Code Development Tools – Selection Criteria & Top Tools
Notepad++ ranks very highly amongst respondents that prioritize integration; however, ranks 3rd place in terms of
reliability – where Visual Studio Code and some low code applications rank better
Top 3 Criteria Notepad++
Visual Studio
Code
AQTime Pro Atom Collaborator Emacs SwaggerHub
Overall % Using 35% 26% 22% 17% 14% 12% 8%
Top Tools (% Respondent Using) by Selection Criteria
EngagementPurchase Retention
Other Selection Criteria: Time to implement, customer service, customer reviews, price, user-friendliness
36% 25% 25% 22% 17% 14% 11%
28% 30% 13% 12% 9% 9% 6%
38% 28% 22% 19% 14% 9% 5%
38% 23% 30% 14% 17% 13% 11%
Notepad++ ranks very highly amongst
respondents that prioritize integration;
however, ranks 3rd place in terms of
reliability
While AQTime Pro generally does
not rank well, it does better
among respondents that prioritize
ROI time horizon
Appsheet Appian BettyBlocks
33% 28% 17%
34% 27%
30% 25%
29% 30%
36% 31%
16%
7%
21%
22%
Low Code /
No Code
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
33. 33
“The only negative point [about Atom] is that it takes too
much time to start. So if it were to use less resources for its
operation it would be really nice.”
“I like Appian so that you can integrate into many other
applications to ensure a continuous process flow. Appian
has the capability of providing real-time data on all
connected systems.”
Code Development Tools – Tools Scorecard
While Notepad++ has the most users, Visual Studio Code users engage with their tool most frequently; low code
tool Appsheet rates highly in terms of satisfaction in our survey as well as G2 Crowd
EngagementPurchase Retention
% Overall Users % Daily Users NPS1 (9 or 10) G2 Score2 (out of 5)
Notepad++ 35% 35% 61% 4.6
Visual Studio Code 26% 60% 48% 4.6
AQTime Pro 22% 42% 75% 4.0
Atom 17% 29% 50% 4.4
Collaborator 14% 17% 70% 4.0
Emacs 12% 21% 42% 4.5
SwaggerHub 8% 46% 62% 4.0
“Notepad is free and open source software. Notepad
provides far better functionality rather than just an editor. It
supports multiple languages like XML, Java, HTML etc.
Notepad displays the content with respect to the language
you are using and also gives you the formatted pattern.”
Engagement & Satisfaction Metrics User Reviews [G2]
Appsheet 33% 48% 69% 4.8
Appian 28% 51% 73% 4.4
BettyBlocks 17% 30% 56% 4.5
Low Code / No Code Tools
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
Note: (1) NPS is calculated as the % respondents ranking likelihood to recommend tool as 9 or 10 (“promoters”) less the %
respondents ranking same question as a 6 or lower (“detractors”) x 100 (2) G2 score is average rank across reviewers on scale
of 1 (worst) to 5 (best)
34. 34
Code Development Tools – Usage Frequency
Visual Studio Code, Sublime Text and SwaggerHub have the highest proportion of daily users – with 50%+ using
daily; barring top tools, larger companies have a higher base of serious daily users vs. those at smaller scales
EngagementPurchase Retention
$2 to $10M $10 to $50M $50 to $100M $100 to $250M
Usage Frequency Usage Frequency (% Daily) by Company Scale
83.3% 77.8% 50.0% 57.1%
66.7% 50.0% 33.3% 0.0%
66.7% 66.7% 50.0% 50.0%
28.6% 16.7% 50.0% 71.4%
50.0% 18.2% 27.3% 100.0%
42.9% 28.6% 16.7% 80.0%
50.0% 33.3% 33.3% 76.9%
Low Code / No Code Tools
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Visual Studio Code
Sublime Text
SwaggerHub
AQTime Pro
Notepad++
Atom
Repl.it
Vim
Emacs
Collaborator
Appian
Appsheet
Skuid
BettyBlocks
N/A 33.3% 50.0% 50.0%
Generally, larger
companies have a
higher base of serious
daily users; indicating
smaller teams may be
experimenting with
several tools
N
57
42
28
36
23
12
19
13
4
8
54
45
27
9
Small sample size
Small sample size
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
35. 35
NPS Distribution Scores 1 or 2 Scores 9 or 10
0% 100%
0% 88%
8% 83%
0% 75%
0% 70%
8% 62%
0% 61%
4% 50%
0% 48%
0% 42%
Code Development Tools – Satisfaction
Vim has standout NPS scores, while Repl.it and Sublime are somewhat close; Sublime stood out in its
polarization of user base – with 8% of respondents rating 1 or 2 out of 10 despite have large share of high scores
NPS Score 10
NPS Score 9
NPS Scores 3-8
NPS Scores 1 or 2
Ranked by
NPS Scores by Tool – All Responses
On a scale of one to ten, how likely are you to recommend the following software to a colleague or someone in your network? 10 being most likely.
EngagementPurchase Retention
Vim
Repl.it
Sublime Text
AQTime Pro
Collaborator
SwaggerHub
Notepad++
Atom
Visual Studio Code
Emacs
0% 78%
2% 73%
4% 69%
0% 56%
Skuid
Appian
Appsheet
BettyBlocks
Low Code / No
Code Tools
Small sample size
Small sample size
N
13
19
42
36
8
28
23
12
57
4
27
54
45
9 Small sample size
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
Note: NPS is calculated as the % respondents ranking likelihood to recommend tool as 9 or 10 (“promoters”) less the %
respondents ranking same question as a 6 or lower (“detractors”) x 100
36. 36
Code Development Tools – Overall Tool Ranking
When we consider all key surveyed metrics along the purchase lifecycle, the universally used free code editors
Visual Studio Code and Notepad++ came out top; Appian ranked top amongst low code tools
Familiarity Prevalence NPS (9 or 10) % Daily
Seat
Penetration
Churn Familiarity Prevalence NPS (9 or 10) % Daily
Seat
Penetration
Churn Avg.
Visual Studio Code 36% 26% 48% 60% 57% 2% 3 4 9 1 1 5 3.8
Notepad++ 46% 35% 61% 35% 49% 4% 1 1 6 6 6 6 4.3
AQTime Pro 28% 22% 75% 42% 51% 6% 6 5 1 5 4 8 4.8
SwaggerHub 13% 8% 62% 46% 53% 0% 10 10 5 4 3 2 5.7
Atom 31% 17% 50% 29% 41% 0% 5 6 8 8 10 2 6.5
Collaborator 21% 14% 70% 17% 48% 9% 8 8 3 10 7 9 7.5
Appian 34% 28% 73% 51% 53% 0% 4 3 2 2 2 2 2.5
Appsheet 42% 33% 69% 48% 50% 2% 2 2 4 3 5 4 3.3
BettyBlocks 25% 17% 56% 30% 46% 4% 7 7 7 7 8 7 7.2
Composite “Score” by Tool
% of Respondents Ranking by Criteria Avg. Ranking
Lower score = better
Measure of brand
awareness + adoption
Measure of
engagement
Satisfaction
Likely to keep
using product
Low Code / No Code Tools
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
38. 38
Code Verification Tools – Usage
GitLab and Codacy are popular within all scale buckets; however, GitLab customers skew large while Selenium
customers skew toward the smaller scale buckets
EngagementPurchase Retention
$2 to $10M (22N)
$10 to $50M (32N)
$50 to $100M (27N)
$100 to $250M (25N)
$250M+ (62N)
Usage by Company Scale (Annual Revenue) – Top 10 tools
Which of the following code review tools does your organization currently use? Base: 168N
32% 18% 23% 23% 14% 5% 14% 18% 9% 5% 9% 5% 5% 5% 5% 0%
13% 22% 19% 13% 13% 6% 9% 9% 9% 6% 6% 9% 0% 3% 3% 3%
33% 30% 22% 7% 30% 33% 22% 11% 0% 11% 11% 7% 4% 7% 4% 4%
24% 32% 12% 24% 20% 20% 16% 12% 12% 12% 4% 8% 0% 0% 12% 4%
44% 21% 11% 15% 6% 10% 6% 10% 10% 6% 5% 3% 11% 8% 5% 2%
32%
24%
16% 15% 14% 14% 12% 11%
8% 8% 7% 6% 5% 5% 5%
2%
GitLab
Codacy
Selenium
ESLint
Katalon
Mabl
Ranorex
TestComplete
Zephyr
Testim
TimeShiftX
TestingWhiz
SahiPro
SonarQube
Webking
Watir
# of StackShare
companies using:
~2,800
# of StackShare
companies using:
~14,000
# of StackShare
companies using:
~3,600
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
39. 39
Code Verification Tools – Selection Criteria & Top Tools
Integration capabilities and ROI time horizon are top-of-mind in the selection of code review tools; Codacy and
GitLab are within the top 3 tools for all scale buckets in terms of % of respondents using
EngagementPurchase Retention
Selection Criteria & Top Tools by Company Scale (Annual Revenue)
Top
Selection
Criteria
(Criteria
Selected as
Top 3 |
Above
Median
Shown)
Top
Tools
(Ranked)
$2 to $10M $10 to $50M $50 to $100M $100 to $250M $250M+
Integration
Capabilities (68%)
Price (55%)
Pricing Structure &
Contract Flexibility
(36%)
GitLab
Selenium
ESLInt
Integration
Capabilities (59%)
ROI Time Horizon
(47%)
Pricing Structure &
Contract Flexibility
(34%)
Codacy
Selenium
GitLab
ROI Time Horizon
(52%)
Pricing Structure &
Contract Flexibility
(48%)
Customer Service
(41%)
GitLab
Mabl
Codacy
ROI Time Horizon
(60%)
Pricing Structure &
Contract Flexibility
(56%)
Integration
Capabilities (52%)
Codacy
GitLab
ESLint
Integration
Capabilities (63%)
ROI Time Horizon
(40%)
Reliability
(36%)
GitLab
Codacy
ESLint
Details on
following page
Key Takeaways
▪ Integration capabilities
and ROI Time Horizon
are top-of-mind in the
selection of code review
tools
▪ Pricing structure &
contract flexibility is
more important for code
review tools than it is
project management and
code development tools
▪ Codacy and GitLab are
within the top 3 tools for
all scale buckets
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
40. 40
26% 32% 18% 17% 15% 17% 22% 9% 9% 11%
41% 17% 13% 2% 6% 19% 13% 9% 15% 2%
33% 25% 15% 20% 18% 13% 13% 11% 7% 8%
25% 27% 12% 15% 15% 11% 10% 8% 7% 11%
Pricing structure &
contract flexibility
Reliability
Ability to integrate
Return on investment
(ROI) time horizon
Code Verification Tools – Selection Criteria & Top Tools
GitLab has the highest overall prevalence in developer stacks, propelled by those who prioritize reliability and
integration – criteria along which GitLab ranks particularly well
Top 3 Criteria GitLab Codacy Selenium ESLint Katalon Mabl Ranorex TestComplete Zephyr Testim
Overall % Using 32% 24% 16% 15% 14% 14% 12% 11% 8% 8%
Top Tools (% Respondent Using) by Selection Criteria
EngagementPurchase Retention
Other Selection Criteria: Time to implement, customer service, customer reviews, price, user-friendliness
GitLab has the highest overall prevalence in
developer stacks, propelled by those who prioritize
reliability and integration – criteria along which
GitLab ranks particularly well
Although Codacy is popular amongst the
overall group, it is not as much amongst
companies that prioritize reliability
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
41. 41
“Ranorex helps speed up the automation process - it’s
easy to learn and pick up and has the ability to enhance
functionality by manually adding user codes”
“MABL was extremely easy to use and get up to speed
on. By switching to MABL from open source JS libraries
we have eliminated a huge barrier to entry into automation,
increased productivity with non-SDETS contributing robust
tests, and minimized maintenance with the ML used to
identify objects and comparing to previous baselines.”
Code Verification Tools – Tools Scorecard
GitLab has both the most users and high G2 review scores with Codacy following closely
EngagementPurchase Retention
% Overall Users % Daily Users NPS1 (9 or 10) G2 Score2 (out of 5)
GitLab 32% 36% 62% 4.4
Codacy 24% 48% 75% 4.4
Selenium 16% 30% 44% 4.1
ESLint 15% 50% 58%
Katalon 14% 38% 54% 4.1
Mabl 14% 48% 52% 4.3
Ranorex 12% 40% 55% 4.2
TestComplete 11% 21% 37% 4.3
Zephyr 8% 43% 50% 4.0
Testim 8% 46% 62% 4.6
“Codacy checks our scala code for dumb (and not-so-
dumb) mistakes. The score (4 out 5), while not perfect, is
still an excellent indicator.”
“In my work as a freelancer, GitLab is an excellent tool
to keep track of each project that I carry out - each
feature of this software is very important for all users,
because we can import and export a project at the time we
want and it will always be updated to the most recent
version that we have endorsed. In particular, the ability to
create branches of the same project is key.”
Engagement & Satisfaction Metrics User Reviews [G2]
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
Note: (1) NPS is calculated as the % respondents ranking likelihood to recommend tool as 9 or 10 (“promoters”) less the %
respondents ranking same question as a 6 or lower (“detractors”) x 100 (2) G2 score is average rank across reviewers on scale
of 1 (worst) to 5 (best)
42. 42
Code Verification Tools – Usage Frequency
ESLint, Mabl and Codacy have the highest proportion of daily users – with ~50% using daily; generally, larger
companies have a higher base of serious daily users vs. those at smaller scales
EngagementPurchase Retention
Usage Frequency Usage Frequency (% Daily) by Company Scale
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
ESLint
Mabl
Codacy
Testim
Zephyr
Ranorex
Katalon
GitLab
Sahi Pro
SonarQube
Webking
Selenium
TimeShiftX
TestComplete
TestingWhiz
Watir
Daily Weekly Monthly Few Times a Year Ad Hoc
$2 to $10M $10 to $50M $50 to $100M
$100 to
$250M
$250M+
0% 50% 50% 83% 56%
0% 100% 44% 60% 33%
0% 29% 38% 63% 69%
100% 50% 33% 67% 25%
100% 67% 0% 33%
33% 33% 50% 50% 25%
33% 25% 38% 40% 50%
14% 0% 22% 83% 41%
0% 50% 33% 33% 29%
50% 0% 0% 33% 17%
Generally, larger
companies have a higher
base of serious daily
users, indicating decision
makers are probably more
intentional in their
purchases / sign ups
N
26
23
40
13
14
20
24
53
9
9
9
27
11
19
10
4
Small sample size
Small sample size
Small sample size
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
43. 43
NPS Distribution Scores 1 or 2 Scores 9 or 10
3% 75%
0% 73%
0% 67%
0% 67%
2% 62%
0% 62%
4% 58%
0% 56%
5% 55%
4% 54%
4% 52%
0% 50%
0% 50%
0% 44%
0% 37%
0% 25%
Code Verification Tools – Satisfaction
Codacy, TimeShiftX and Webking have the top NPS scores; unlike other tool categories, code review does not
have any that are greatly polarizing
NPS Score 10
NPS Score 9
NPS Scores 3-8
NPS Scores 1 or 2
Ranked by
NPS Scores by Tool – All Responses
On a scale of one to ten, how likely are you to recommend the following software to a colleague or someone in your network? 10 being most likely.
EngagementPurchase Retention
Codacy
TimeShiftX
Webking
SonarQube
GitLab
Testim
ESLint
Sahi Pro
Ranorex
Katalon
Mabl
Zephyr
TestingWhiz
Selenium
TestComplete
Watir
N
40
11
9
9
53
13
26
9
9
24
23
14
10
27
19
4 Small sample size
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
Note: NPS is calculated as the % respondents ranking likelihood to recommend tool as 9 or 10 (“promoters”) less the %
respondents ranking same question as a 6 or lower (“detractors”) x 100
44. 44
ALL $2 to $10M $10 to $50M $50 to $100M $100 to $250M $250M+
SonarQube 22%
Mabl 13% 0% 0% 11% 20% 17%
Katalon 13% 0% 13% 20% 0%
Webking 11% 0% 0% 0% 0% 33%
TestComplete 11% 0% 33% 0% 0%
TestingWhiz 10%
Testim 8% 0% 0% 0% 33% 0%
Ranorex 5% 0% 0% 0% 25% 0%
Selenium 4% 0% 0% 0% 0%
Codacy 0% 0% 0% 0% 0% 0%
TimeShiftX 0% 0% 0% 0% 0% 0%
GitLab 0% 0% 0% 0% 0% 0%
ESLint 0% 0% 0% 0% 0% 0%
Sahi Pro 0%
Zephyr 0% 0% 0% 0% 0%
Watir
Code Verification Tools – Churn Propensity
SonarQube, Mabl and Katalon have 12%+ of its users indicating that they’d churn in 3-5 years; smaller
companies are unlikely to reduce their use of code review software significantly
Churn Propensity by Tool
How is your organization's adoption of the following code review software likely to change 3-5 years from now? – Decrease significantly or stop entirely
EngagementPurchase Retention
Within the groups where a
significant number of users
plan to reduce use adoption
significantly, the patterns are
relatively similar across tools
Smaller companies
are unlikely to reduce
their use of code
development software
significantly
Note: Data points with very small sample size removed where applicable
Deep dive into high churn
products reveal correlation with
seamlessness; i.e., products
where termination is seamless
have the highest churn
probability despite high NPS
Small sample size
Small sample size
Small sample size
Small sample size
N
9
23
24
9
19
10
13
9
27
40
11
53
26
9
14
4
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
47. 47
Code Deployment Tools – Usage
GitHub is the most widely used code management tool; AWS CodeDeploy is the most commonly used CI/CD
while GitLab – a tool accomplishing both functions – ranks highly as well
EngagementPurchase Retention
$2 to $10M (26N)
$10 to $50M (30N)
$50 to $100M (24N)
$100 to $250M (21N)
$250M+ (56N)
Usage by Company Scale (Annual Revenue) – Top 10 tools
Which of the following code deployment tools does your organization currently use? Base: 157N
27%
14%
10%
1%
27%
15% 14%
8%
6% 5% 4% 3% 3% 2%
GitHub
GitLab
Bitbucket
HashiCorp
AWS
CodeDeploy
Ansible
GitLab
Jenkins
Netlify
CircleCI
TeamCity
Spinnaker
TravisCI
OctopusDeploy
Code Management CI/CD
23% 12% 8% 0% 12% 8% 12% 12% 8% 4% 4% 4% 0% 0%
37% 10% 7% 0% 23% 10% 10% 10% 7% 0% 7% 3% 0% 0%
33% 17% 25% 4% 38% 33% 17% 13% 13% 13% 4% 0% 4% 4%
29% 14% 5% 0% 29% 19% 14% 0% 5% 0% 10% 0% 5% 5%
21% 16% 7% 0% 32% 11% 16% 5% 2% 7% 2% 5% 4% 2%
# of StackShare
companies using:
~3,500
# of StackShare
companies using:
~2,800
# of StackShare
companies using:
~2,200
# of StackShare
companies using:
~2,200
# of StackShare
companies using:
~7,200
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
48. 48
Code Deployment Tools – Selection Criteria & Top Tools
Although integration capabilities are important for code deployment tools, more than other tool categories, price,
customer reviews and time to implement are top selection criteria
EngagementPurchase Retention
Selection Criteria & Top Tools by Company Scale (Annual Revenue)
Top
Selection
Criteria
(Criteria
Selected as
Top 3 |
Above
Median
Shown)
Top
Tools
(Ranked)
$2 to $10M $10 to $50M $50 to $100M $100 to $250M $250M+
Integration
Capabilities (42%)
ROI Time Horizon
(38%)
Pricing Structure &
Contract Flexibility
(37%)
GitHub
Alteryx
GitLab
Integration
Capabilities (58%)
ROI Time Horizon
(42%)
Pricing Structure &
Contract Flexibility
(39%)
GitHub
AWS CodeDeploy
Alteryx
Pricing Structure &
Contract Flexibility
(43%)
Time to Implement
(40%)
Integration
Capabilities (40%)
AWS CodeDeploy
GitHub
Ansible
Time to Implement
(42%)
Customer Reviews
(42%)
Price (42%)
Alertsite
GitHub
AWS CodeDeploy
Alteryx
ROI Time Horizon
(43%)
Customer Reviews
(38%)
Integration
Capabilities (38%)
AWS CodeDeploy
GitHub
AlertSite
Details on
following page
Key Takeaways
▪ Integration Capabilities,
Pricing, Time to
Implement and ROI Time
Horizon are important
▪ Pricing structure &
contract flexibility is
more important for code
deployment tools than it is
project management and
code development tools
▪ GitHub is within the top 3
tools for all scale buckets
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
49. 49
24% 10% 12% 16% 9% 10% 7%
36% 9% 21% 9% 8% 4% 2%
33% 12% 12% 9% 6% 11% 6%
29% 14% 14% 8% 7% 3% 5%
AWS
CodeDeploy
Ansible GitLab Bitbucket Jenkins Netlify CircleCI
27% 15% 14% 10% 8% 6% 5%
29% 12% 16%
32% 21% 9%
30% 12% 9%
27% 14% 8%
Pricing structure &
contract flexibility
Reliability
Ability to integrate
Return on investment
(ROI) time horizon
Code Deployment Tools – Selection Criteria & Top Tools
GitHub is the most prevalent code management tool, driven by its reliability; AWS CodeDeploy is the most
prevalent CI/CD, driven by an over-index from AWS cloud customers
Top 3 Criteria GitHub GitLab Bitbucket
Overall % Using 27% 14% 10%
Top Tools (% Respondent Using) by Selection Criteria
EngagementPurchase Retention
Other Selection Criteria: Time to implement, customer service, customer reviews, price, user-friendliness
AWS CodeDeploy has the highest overall prevalence in stacks,
propelled by those who prioritize reliability and integration
Code Management CI/CD
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
50. 50
Code Deployment Tools – Tools Scorecard
GitHub has both the most users and high G2 review scores; within CI/CD tools, AWS CodeDeploy has the highest
overall penetration but lags in daily engagement and G2 review scores vs. Ansible and GitLab
EngagementPurchase Retention
% Overall Users % Daily Users NPS1 (9 or 10) G2 Score2 (out of 5)
GitHub 27% 40% 60% 4.7
GitLab 14% 50% 55% 4.4
Bitbucket 10% 20% 40% 4.4
AWS CodeDeploy 27% 37% 60% 4.2
Ansible 15% 39% 52% 4.5
GitLab 14% 50% 55% 4.4
Jenkins 8% 0% 33% 4.3
Netlify 6% 33% 56% 4.5
CircleCI 5% 25% 88% 4.4
“GitHub also provides basic web hosting through GitHub
Pages, making it easy to create a custom web page for
your project/repo to share info, docs, download links,
etc. It has all distributed version control and source code
management functionalities of git.”
“It is very simple to use and is free for experimentation
for those who want to get the hang of deployment
pipelines without investing too much money. I used
CodeDeploy with Bitbucket and the Bitbucket CodeDeploy
plugin was very easy to setup. The deployment
configuration and groups are nice features.”
Engagement & Satisfaction Metrics User Reviews [G2]
Code Management
CI/CD
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
Note: (1) NPS is calculated as the % respondents ranking likelihood to recommend tool as 9 or 10 (“promoters”) less the %
respondents ranking same question as a 6 or lower (“detractors”) x 100 (2) G2 score is average rank across reviewers on scale
of 1 (worst) to 5 (best)
51. 51
Code Deployment Tools – Usage Frequency
Barring small user bases, GitLab has the highest proportion of daily users – with 50%+ using daily; generally,
larger companies have a higher base of serious daily users vs. those at smaller scales
EngagementPurchase Retention
Usage Frequency Usage Frequency (% Daily) by Company Scale
$2 to $10M $10 to $50M $50 to $100M
$100 to
$250M
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
GitLab
GitHub
Bitbucket
HashiCorp
Daily Weekly Monthly Few Times a Year Ad Hoc
67% 33% 50% 67%
17% 45% 63% 33%
50% 0% 17% 100%
Code Management
CI/CD
N
22
43
15
1
3
22
5
23
9
7
43
12
8
4
Small sample size
Octopus Deploy
GitLab
Spinnaker
Ansible
Netlify
TeamCity
AWS CodeDeploy
Jenkins
CircleCI
Travis CI
67% 33% 50% 67% 44%
0% 33% 25% 75% 50%
50% 0% 33% 100% 0%
33% 43% 22% 50% 39%
0% 0% 0% 0%
Small sample size
Small sample size
Small sample size
Small sample size
Generally, larger companies have
a higher base of serious daily
users; indicating decision makers
are probably more intentional in
their purchases / sign ups
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
52. 52
NPS Distribution Scores 1 or 2 Scores 9 or 10
0% 60%
0% 55%
7% 40%
Code Deployment Tools – Satisfaction
Amongst code management tools, GitHub and GitLab have the highest NPS; CircleCI, AWS CodeDeploy and
Netlify are amongst the top CI/CD tools in terms of user satisfaction
NPS Score 10
NPS Score 9
NPS Scores 3-8
NPS Scores 1 or 2
Ranked by
NPS Scores by Tool – All Responses
On a scale of one to ten, how likely are you to recommend the following software to a colleague or someone in your network? 10 being most likely.
EngagementPurchase Retention
0% 88%
0% 60%
0% 56%
0% 55%
4% 52%
0% 50%
0% 40%
0% 33%
29% 14%
GitHub
GitLab
Bitbucket
CircleCI
AWS CodeDeploy
Netlify
GitLab
Ansible
Travis CI
Spinnaker
Jenkins
TeamCity
Code Management
CI/CD
N
43
22
15
8
43
9
22
23
4
5
12
7
Small sample size
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
Note: NPS is calculated as the % respondents ranking likelihood to recommend tool as 9 or 10 (“promoters”) less the %
respondents ranking same question as a 6 or lower (“detractors”) x 100
53. 53
Code Deployment Tools – Overall Tool Ranking
When we consider all key surveyed metrics along the purchase lifecycle, GitHub leads the pack for code
management tools and AWS CodeDeploy for CI/CD; GitLab ranks #2 in both of those categories
Familiarity Prevalence NPS (9 or 10) % Daily
Seat
Penetration
Familiarity Prevalence NPS (9 or 10) % Daily
Seat
Penetration
Avg.
GitHub 35% 27% 60% 40% 47% 2 2 3 2 4 2.4
GitLab 20% 14% 55% 50% 50% 4 4 5 1 2 3.2
Bitbucket 19% 10% 40% 20% 41% 5 5 7 7 7 6.2
AWS CodeDeploy 38% 27% 60% 37% 49% 1 2 3 4 3 2.4
GitLab 20% 14% 55% 50% 50% 3 3 6 3 5 4.0
Ansible 24% 15% 52% 39% 47% 7 7 4 5 1 4.8
Netlify 13% 6% 56% 33% 51% 8 8 1 6 6 5.8
CircleCI 12% 5% 88% 25% 47% 6 6 8 8 8 7.2
Jenkins 13% 8% 33% 0% 38% 1 2 3 4 3 2.4
Composite “Score” by Tool
% of Respondents Ranking by Criteria Avg. Ranking
Lower score = better
Measure of brand
awareness + adoption
Measure of
engagement
Satisfaction
Likely to keep
using product
Code
Management
CI/CD
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
55. 55
Code Monitoring & Security Tools – Usage
Datadog is the most prevalent monitoring tool followed by Kibana and Prometheus; Checkmarx and Exabeam
lead the pack amongst defense tools
EngagementPurchase Retention
$2 to $10M (26N)
$10 to $50M (30N)
$50 to $100M (24N)
$100 to $250M (21N)
$250M+ (56N)
Usage by Company Scale (Annual Revenue) – Top 10 tools
Which of the following code security tools does your organization currently use? Base: 157N
25%
12% 11% 10%
8% 7% 6%
3% 3%
24%
19%
12% 11%
7%
3%
6%
3% 2% 2%
Datadog
Kibana
Prometheus
Grafana
Sentry
NewRelic
Logstash
TrackJS
Turbonomic
Checkmarx
Exabeam
Guardicore
SecureCode
Warrior
Lacework
PagerDuty
Tigera
Twistlock
TreatStack
Whitesource
18% 9% 5% 5% 9% 9% 9% 5% 0% 18% 14% 5% 14% 5% 0% 5% 5% 5% 5%
23% 10% 17% 7% 3% 7% 3% 3% 0% 17% 3% 17% 3% 10% 3% 7% 0% 0% 3%
26% 22% 13% 26% 17% 13% 17% 4% 4% 30% 35% 17% 30% 13% 9% 4% 4% 4% 0%
30% 9% 0% 13% 0% 9% 0% 9% 4% 30% 39% 4% 9% 9% 4% 4% 0% 0% 0%
25% 11% 15% 6% 9% 2% 4% 0% 6% 25% 13% 13% 6% 2% 2% 8% 4% 2% 2%
Monitoring Defense
StackShare rated top tool of 2019
within “Monitoring Tools”
# of StackShare
companies using:
~1,700
# of StackShare
companies using:
~1,600
# of StackShare
companies using:
~2,400
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
56. 56
Code Monitoring & Security Tools – Selection Criteria & Top Tools
Customer service is especially important for larger companies when it comes to code monitoring & security tools,
while integration capabilities are more important for smaller companies
EngagementPurchase Retention
Selection Criteria & Top Tools by Company Scale (Annual Revenue)
Top
Selection
Criteria
(Criteria
Selected as
Top 3 |
Above
Median
Shown)
Top
Tools
(Ranked)
$2 to $10M $10 to $50M $50 to $100M $100 to $250M $250M+
ROI Time Horizon
(50%)
Integration
Capabilities (41%)
Pricing Structure &
Contract Flexibility
(41%)
Datadog
Checkmarx
Exabeam
Integration
Capabilities (47%)
Reliability (43%)
Time to Implement
(40%)
Datadog
Checkmarx
Guardicore
Integration
Capabilities (52%)
ROI Time Horizon
(48%)
Customer Service
(43%)
Exabeam
Checkmarx
Secure Code
Warrior
Pricing Structure &
Contract Flexibility
(52%)
Integration
Capabilities (43%)
Customer Service
(39%)
Exabeam
Datadog
Checkmarx
Customer Service
(40%)
Pricing Structure &
Contract Flexibility
(40%)
ROI Time Horizon
(40%)
Datadog
Checkmarx
Prometheus
Details on
following page
Key Takeaways
▪ Customer service more
important criteria for
security tools than any
other tool category
▪ Customer service
especially important for
larger companies while
integration capabilities
more important for smaller
scale buckets
▪ Checkmarx only tool
within the top 3 tools for
all scale buckets
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
57. 57
26% 15% 8% 13% 5%
20% 10% 14% 4% 8%
27% 11% 13% 10% 8%
32% 14% 7% 2% 11%
Pricing structure &
contract flexibility
Reliability
Ability to integrate
Return on investment
(ROI) time horizon
Code Monitoring & Security Tools – Selection Criteria & Top Tools
Datadog has the highest overall prevalence amongst monitoring tools, and is the #1 choice for most companies;
Checkmarx and Exabeam are close on most selection metrics and rank highly amongst defense tools
Top 3 Criteria Datadog Kibana Prometheus Grafana Sentry
Overall % Using 25% 12% 11% 10% 8%
Top Tools (% Respondent Using) by Selection Criteria
EngagementPurchase Retention
Other Selection Criteria: Time to implement, customer service, customer reviews, price, user-friendliness
Datadog not only has the highest overall prevalence in
developer stacks, it is also the #1 choice for companies
prioritizing pricing, reliability, integration and ROI time horizon
21% 24% 15% 15% 10%
18% 14% 8% 10% 8%
21% 15% 11% 11% 6%
30% 25% 9% 11% 5%
Checkmarx Exabeam Guardicore
Secure Code
Warrior
Lacework
24% 19% 12% 11% 7%
Monitoring Defense
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
58. 58
[Kibana] “What I like the most is fact, that in a sea of logs,
you can easily search for special correlation ID or
something else. It's great tool for debugging, which is
what I use it for.”
[Checkamarx] “This is an excellent tool to write secure
code and follow best practices. I like that it gives a
detailed overview of the issue in your static code and also
provides ways to solve it. It attributes a risk profile to
each issue and this way you can solve the ones with
high priority first.”
Code Monitoring & Security Tools – Tools Scorecard
Datadog is not only the most prevalent monitoring tool, but also has much higher engagement (% daily users)
than competitors; within defense tools, Checkmarx has the highest engagement and user satisfaction
EngagementPurchase Retention
% Overall Users % Daily Users NPS1 (9 or 10) G2 Score2 (out of 5)
Datadog 25% 41% 59% 4.2
Kibana 12% 22% 61% 3.8
Prometheus 11% 18% 53% 4.3
Grafana 10% 33% 47% 4.4
Sentry 8% 17% 50% 4.5
Checkmarx 24% 53% 83% 4.1
Exabeam 19% 43% 68%
Guardicore 12% 44% 72% 4.2
Secure Code Warrior 11% 25% 56%
Lacework 7% 50% 60% 4.4
“With Datadog you can quickly get up and running.
May be the easiest option out there. Since it enables you to
put everything into one dashboard irrespective of their
zone, VPC or environment type we can have one
bookmarked place to look at for the first report. Once you
figure out how to set up Datadog agents then it's a one
point solution. Integrating it to IMs like Slack is really easy
and thresholds can be set individually to prioritize alerts.”
Engagement & Satisfaction Metrics User Reviews [G2]
Monitoring
Defense
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
Note: (1) NPS is calculated as the % respondents ranking likelihood to recommend tool as 9 or 10 (“promoters”) less the %
respondents ranking same question as a 6 or lower (“detractors”) x 100 (2) G2 score is average rank across reviewers on scale
of 1 (worst) to 5 (best)
59. 59
Code Monitoring & Security Tools – Usage Frequency
New Relic, Datadog and Grafana have the highest proportion of daily users within monitoring tools; PagerDuty
and Checkmarx have the highest within defense tools – with 50%+ users using daily
EngagementPurchase Retention
Usage Frequency Usage Frequency (% Daily) by Company Scale
0% 100% 67% 50%
100% 50% 0% 0%
0% 0% 67% 33%
0% 67% 40% 0%
0% 40% 33%
50% 0% 0%
$2 to $10M $10 to $50M $50 to $100M
$100 to
$250M
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
New Relic
Datadog
Grafana
Kibana
Turbonomic
Prometheus
Sentry
Logstash
TrackJS
Daily Weekly Monthly Few Times a Year Ad Hoc
PagerDuty
Checkmarx
Lacework
Guardicore
Tigera
Exabeam
Secure Code Warrior
Twistlock
Treat Stack
50% 20% 43% 71%
100% 0% 67% 100%
0% 40% 50% 100%
0% 0% 38% 44%
33% 100% 14% 50%
Monitoring
Defense
Monitoring
Defense
N
10
37
15
18
5
17
12
9
5
5
36
10
18
9
28
16
4
3
Small sample size
Small sample size
Small sample size
Small sample size
Small sample size
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
60. 60
NPS Distribution Scores 1 or 2 Scores 9 or 10
0% 78%
0% 61%
3% 59%
6% 53%
0% 50%
0% 47%
0% 40%
3% 83%
0% 72%
0% 68%
0% 60%
0% 60%
0% 56%
11% 56%
Code Monitoring & Security Tools – Satisfaction
Within monitoring tools, Logstash has the most satisfied users, followed closely by Kibana and Datadog; within
defense tools, Checkmarx has the biggest lead by a healthy margin
NPS Score 10
NPS Score 9
NPS Scores 3-8
NPS Scores 1 or 2
Ranked by
NPS Scores by Tool – All Responses
On a scale of one to ten, how likely are you to recommend the following software to a colleague or someone in your network? 10 being most likely.
EngagementPurchase Retention
Logstash
Kibana
Datadog
Prometheus
Sentry
Grafana
New Relic
Checkmarx
Guardicore
Exabeam
Lacework
PagerDuty
Secure Code Warrior
Tigera
Monitoring
Defense
N
9
18
37
17
12
15
10
36
18
28
10
5
16
9
Small sample size
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
Note: NPS is calculated as the % respondents ranking likelihood to recommend tool as 9 or 10 (“promoters”) less the %
respondents ranking same question as a 6 or lower (“detractors”) x 100
61. 61
Code Monitoring & Security Tools – Overall Tool Ranking
In both monitoring and security tools, the top choice has a significant lead vs. second choice – indicating tendency
to gravitate toward best in breed tools when it concerns security
Familiarity Prevalence NPS (9 or 10) % Daily
Seat
Penetration
Familiarity Prevalence NPS (9 or 10) % Daily
Seat
Penetration
Avg.
Datadog 31% 25% 59% 41% 48% 1 1 6 5 6 3.8
Kibana 17% 12% 61% 22% 39% 7 5 4 8 9 6.4
Prometheus 16% 11% 53% 18% 54% 8 6 8 9 2 6.6
Grafana 19% 10% 47% 33% 49% 5 8 10 6 5 6.8
Sentry 15% 8% 50% 17% 52% 9 9 9 10 3 8.0
Checkmarx 28% 24% 83% 53% 55% 3 2 1 1 1 1.6
Exabeam 30% 19% 68% 43% 47% 2 3 3 4 7 3.8
Guardicore 20% 12% 72% 44% 44% 4 5 2 3 8 4.3
Secure Code Warrior 17% 11% 56% 25% 52% 7 7 7 7 4 6.3
Lacework 11% 7% 60% 50% 38% 10 10 5 2 10 7.4
Composite “Score” by Tool
% of Respondents Ranking by Criteria Avg. Ranking
Lower score = better
Measure of brand
awareness + adoption
Measure of
engagement
Satisfaction
Likely to keep
using product
Monitoring
Defense
Sources: ICONIQ Analytics External Dev. Stack Survey, n=200
63. 63
Case Studies – Developer Stacks
A deeper look into specific companies’ developer stacks reveal newer companies to have a greater appetite for
tools experimentation; code verification is one where companies use fewer tools across the board
Code
Management
CI/CD Monitoring DefenseDevelopment Verification
Project
Management
Age: 22 Years
Revenue: ~$150B
Employees: ~100K
HQ: Bay Area
Age: 7 Years
Revenue: ~$25M
Employees: ~150
HQ: Bay Area
Age: 15 Years
Revenue: ~$1.5B
Employees: ~5K
HQ: Ottawa
Age: 6 Years
Revenue: ~$5M
Employees: ~50
HQ: Bay Area
Age: 5 Years
Revenue: ~$25M
Employees: ~1,500
HQ: London
AngularJS
Android
Studio
Bazel
EarlyGrey
GitHub
Git
Chef
Buildkite
New Relic
Asana
iDoneThis
Backbon.js
Apache
Thrift
Brunch Puppet Labs
Sentry
Prometheus
Graphite
HackerOne
Confluence
GitHub
HashiCorp
Ansible
CircleCI
Spinnaker
Armory
Graphite
Trello GitHub Codeship Sentry PagerDuty
Jira
Confluence
Gatsby
Visual
Studio
Cypress
ESLint
Git
BitBucket
Ansible
▪ Legacy giants, such as
Google, generally have
fewer tools, likely in part due
to NIH cultures
▪ Newer companies,
especially those experiencing
faster growth have a greater
appetite for
experimentation
▪ Company HQ location does
not seem to drive
meaningful differences in
number of tools
▪ Companies generally have
fewer verification tools
1
Age: 11 Years
Revenue: ~$3B
Employees: ~25K
HQ: Bay Area
Data not available
20
35
59
45
36
34
Prominent tool in external survey
Newer, fast growing companies have
more tools in their stack and a greater
number per tool category – indicating
appetite for experimentation
Sources: StackShare, ICONIQ Analytics & Research
64. 64
IMPORTANT DISCLOSURES
Important Disclosures:
These materials and the views expressed herein are intended for informational purposes only, and should not be used as or as a substitute for investment, financial, tax or
other forms of advice or for any other purpose. These materials are not intended to act as an offer or solicitation for the purchase or sale of any financial instrument.
Information referenced or used herein has been obtained from sources believed to be reliable but ICONIQ does not warrant the completeness or accuracy of any third-party
information. Past performance is not indicative of future results. Graphs and charts cannot be used in and of themselves to determine the advisability of investing in
securities or the timing of the purchase or sale of any security.
These materials contain opinions and/or estimates of ICONIQ and its personnel that constitute its and their best judgment as of the date of these materials and are subject
to change without notice. The opinions contained in these materials do not take into account individual circumstances, objectives, or needs and are not intended as
recommendations of particular securities, financial instruments, or strategies. The recipient of these materials must independently undertake its own diligence and
thereafter make its own decisions regarding the advisability of any securities or financial instruments referenced herein.
ICONIQ will not be responsible or have any liability for any injuries or damages of the recipient of these materials or any third party, including injuries or damages that may
result from (i) the reliance of any person upon any information or opinion provided herein or (ii) any errors, inaccuracies, omissions in, or any other failure of, the data
herein, from whatever cause. ICONIQ shall not be liable for direct, indirect or incidental special or consequential damages resulting from the information contained herein
and/or from any decision taken on the basis of such information, regardless of whether such damages were foreseeable.
Some of the companies selected for inclusion in this report are held by investment vehicles or clients managed by ICONIQ, which is an investment adviser registered with the
U.S. Securities and Exchange Commission. Registration as an investment adviser does not imply any level of skill or training.
To date, ICONIQ Capital Technology Investing, aka ICONIQ Strategic Partners (ICONIQ’s growth equity platform), has made investments in the following companies:
B2B Investments
Adyen, Alteryx, Apttus, BambooHR, BlackLine, Braze, Campaign Monitor, Collibra, Coupa, Crowdstrike, CyberGRX, Datadog, Dataiku, Dialpad, DocuSign, ezCater, Fastly,
FreeWill, GitLab, GreenSky, Hashicorp, HeadSpin, Heptagon, Highspot, IEX, Intercom, InVision, Lucid, Marqeta, Moveworks, Netskope, People.ai, Pluralsight, Procore,
Relativity, Restaurant365, Sendbird, ServiceTitan, Skuid, Smartling, Snowflake, Sprinklr, Truckstop, Turbonomic, Twistlock, Virtru, Zoom
B2C Investments
Age of Learning, Airbnb, Alibaba, Automattic, Chime, Epic Games, Flipkart, GoFundMe, Hippo Insurance, Honest Company, Houzz, Red Ventures, Robinhood, Tencent Music
Entertainment, Uber, Warby Parker, Wayfair, Wolt
The above investments are made by ICONIQ Strategic Partners Funds and does not include all investments made by ICONIQ Capital.