Supporting Digital Media Workflows in the Cloud with Perforce HelixPerforce
Walk through a distributed, non-destructive digital media workflow with graphics, audio and video media from start to finish. Learn the pain points and challenges of versioning increasingly large and varied formats, and see various strategies and best practices for configuring and managing depots in Perforce Helix that facilitate collaborative creative work while minimizing large data transfers. You’ll leave this session with the insights and skills needed to securely support automated digital media workflows in your organization using the Perforce Helix platform with the latest cloud services.
Perforce Helix Never Dies: DevOps at Bandai Namco StudiosPerforce
Traditionally at Bandai Namco Studios, there has been no unified version control system in place and teams could choose to use any VCS system for their game titles—Subversion, Git, AlienBrain, or none at all. I’ll talk about why Bandai Namco Studios chose to standardize on Perforce Helix, show how we develop LiveOps-type mobile applications using the Unity game engine, and the advantages we gain from centrally managing code and assets in Helix.
Accelerating Software Development with NetApp's P4flexPerforce
The challenge for developers who work with large volumes of data such as multimedia assets, video game art, and firmware designs, etc., is the ability to get a quick copy of source and build assets. By combining the technologies of Perforce and NetApp, a new Perforce workspace can be created in minutes instead of hours. Perforce in collaboration with NetApp has developed a p4 broker script written in Python that allows users to create workspaces quickly using NetApp FlexClone technology.
From ClearCase to Perforce Helix: Breakthroughs in Scalability at IntelPerforce
See how the Intel Security and Sensors Firmware team transitioned from IBM ClearCase to Perforce Helix with Microsoft TFS to enable robust and scalable ALM and CI with full traceability. Discover how Intel consolidated and converged 15 different development methodologies used to drive firmware projects to three single paths for all Intel platforms.
Software Testing in a Distributed EnvironmentPerforce
Distributed development across countries creates both challenges and opportunities for the production of high quality software. We’ll look at new ways of achieving automation for testing software in a continuous delivery context, using parallelization techniques and automated analysis fully integrated with a reliable and scalable SCM system. A new optimal method of testing common code in similar branches is presented along with the semantic merging of testing results.
Using Perforce Data in Development at TableauPerforce
Data plays a big role at Tableau—not just for our customers, but also throughout our company. Using our own products is not only one of our fundamental company values, but the analysis and discoveries we make are important to track as they shape our development processes and influence our day-to-day decisions. In this talk, we present and analyze a variety of data visualizations based on Perforce data from our development organization and share how it has influenced our infrastructure and development practices.
Gears of Perforce: AAA Game Development ChallengesPerforce
How does Vancouver-based Xbox team, The Coalition, use Perforce to build Gears of War? By pulling UE4 source from Epic Games, sharing source with other Microsoft Studios, supporting outsourcers—all while delivering 100GB/day inside the studio. Learn how and why we do what we do.
How to Combine Artifacts and Source in a Single ServerPerforce
See how to use Perforce Helix as an artifact manager by extending a Helix repository to store artifacts used for build and deployment. We’ll demo our proof of concept, Hive, and its core functions for configuring and adding new artifact repositories.
Supporting Digital Media Workflows in the Cloud with Perforce HelixPerforce
Walk through a distributed, non-destructive digital media workflow with graphics, audio and video media from start to finish. Learn the pain points and challenges of versioning increasingly large and varied formats, and see various strategies and best practices for configuring and managing depots in Perforce Helix that facilitate collaborative creative work while minimizing large data transfers. You’ll leave this session with the insights and skills needed to securely support automated digital media workflows in your organization using the Perforce Helix platform with the latest cloud services.
Perforce Helix Never Dies: DevOps at Bandai Namco StudiosPerforce
Traditionally at Bandai Namco Studios, there has been no unified version control system in place and teams could choose to use any VCS system for their game titles—Subversion, Git, AlienBrain, or none at all. I’ll talk about why Bandai Namco Studios chose to standardize on Perforce Helix, show how we develop LiveOps-type mobile applications using the Unity game engine, and the advantages we gain from centrally managing code and assets in Helix.
Accelerating Software Development with NetApp's P4flexPerforce
The challenge for developers who work with large volumes of data such as multimedia assets, video game art, and firmware designs, etc., is the ability to get a quick copy of source and build assets. By combining the technologies of Perforce and NetApp, a new Perforce workspace can be created in minutes instead of hours. Perforce in collaboration with NetApp has developed a p4 broker script written in Python that allows users to create workspaces quickly using NetApp FlexClone technology.
From ClearCase to Perforce Helix: Breakthroughs in Scalability at IntelPerforce
See how the Intel Security and Sensors Firmware team transitioned from IBM ClearCase to Perforce Helix with Microsoft TFS to enable robust and scalable ALM and CI with full traceability. Discover how Intel consolidated and converged 15 different development methodologies used to drive firmware projects to three single paths for all Intel platforms.
Software Testing in a Distributed EnvironmentPerforce
Distributed development across countries creates both challenges and opportunities for the production of high quality software. We’ll look at new ways of achieving automation for testing software in a continuous delivery context, using parallelization techniques and automated analysis fully integrated with a reliable and scalable SCM system. A new optimal method of testing common code in similar branches is presented along with the semantic merging of testing results.
Using Perforce Data in Development at TableauPerforce
Data plays a big role at Tableau—not just for our customers, but also throughout our company. Using our own products is not only one of our fundamental company values, but the analysis and discoveries we make are important to track as they shape our development processes and influence our day-to-day decisions. In this talk, we present and analyze a variety of data visualizations based on Perforce data from our development organization and share how it has influenced our infrastructure and development practices.
Gears of Perforce: AAA Game Development ChallengesPerforce
How does Vancouver-based Xbox team, The Coalition, use Perforce to build Gears of War? By pulling UE4 source from Epic Games, sharing source with other Microsoft Studios, supporting outsourcers—all while delivering 100GB/day inside the studio. Learn how and why we do what we do.
How to Combine Artifacts and Source in a Single ServerPerforce
See how to use Perforce Helix as an artifact manager by extending a Helix repository to store artifacts used for build and deployment. We’ll demo our proof of concept, Hive, and its core functions for configuring and adding new artifact repositories.
We’ve begun an initiative at Citrix to make software development inherently more secure. I’ll start with a few security anecdotes, give you a walkthrough of the security layers from data to physical, and highlight security features along the way. I’ll also discuss the Helix Versioning Engine protocol and show you why SSL encryption should be on by default.
Global Software Development powered by PerforcePerforce
From inception to sunset, hundreds of people from around the world are involved in the production and live operations of video games developed by Electronic Arts. An overview of how EA uses a variety of features in Perforce Helix to effectively utilize its world wide talent pool, develop software efficiently, and protect its intellectual property.
Committing to a company-wide software change is no small feat, but if you’re already sweating at the mere thought of checking code in and out, it’s time to plan your escape route.
So, break free and join Tom Tyler, Senior Consultant at Perforce and in-house ClearCase specialist to map out:
- Baseline-and-branch vs. detail history import strategies
- Porting
- Integrations for defect trackers, training, and tooling
- Cutover strategies
Alfresco Platform Update and Roadmap delivered by Gabriele Columbro, Senior Product Manager for Core Platform / API at Alfresco, with updates on the upcoming Alfresco 5.1 release, on Extreme Scalability (and Solr sharding), Share separation, the new API lifecycle and brand new Developer documentation, samples and tutorials. Mentions of the Upgrade Task Force and new Developer platform improvements like support for JAR modules and tracking / reporting of Share modules.
A Practical Guide to Selecting a Stream Processing Technology confluent
Presented by Michael Noll, Product Manager, Confluent.
Why are there so many stream processing frameworks that each define their own terminology? Are the components of each comparable? Why do you need to know about spouts or DStreams just to process a simple sequence of records? Depending on your application’s requirements, you may not need a full framework at all.
Processing and understanding your data to create business value is the ultimate goal of a stream data platform. In this talk we will survey the stream processing landscape, the dimensions along which to evaluate stream processing technologies, and how they integrate with Apache Kafka. Particularly, we will learn how Kafka Streams, the built-in stream processing engine of Apache Kafka, compares to other stream processing systems that require a separate processing infrastructure.
At Tuenti, we do two code pushes per week, sometimes modifying thousands of files and running thousands of automated tests and build operations before, to ensure not only that the code works but also that proper localization is applied, bundles are generated and files get deployed to hundreds of servers as fast and reliable as possible.
We use opensource tools like Mercurial, MySQL, Jenkins, Selenium, PHPUnit and Rsync among our own in-house ones, and have different development, testing, staging and production environments.
We had to fight with problems like statics bundling and versioning, syntax errors and of course the fact that we have +100 engineers working on the codebase, sometimes merging and releasing more than a dozen branches the same day. We also switched from Subversion to Mercurial to obtain more flexibility and faster branching operations.
With this talk we will explain the process of how code changes in ourcode repository end up in live code, detailing some practices and tips that we apply.
Opinionated containers and the future of game servers by Brendan FosberryDocker, Inc.
Combining my passions for automation and games, I will discuss the opportunity and challenge for automating and containerizing game servers. The necessity to prioritize scale and performance makes game servers a perfect candidate for the container revolution. However many aspects of game servers and apps make this pretty challenging. Starting from the perspective of a typical transition to in-house Docker based micro services at Shopkeep, I’ll take a deep dive into the problems with, benefits of and approaches to containerizing these “opinionated” applications in a wider setting.
URP? Excuse You! The Three Kafka Metrics You Need to KnowTodd Palino
What do you really know about how to monitor a Kafka cluster for problems? Is your most reliable monitoring your users telling you there’s something broken? Are you capturing more metrics than the actual data being produced? Sure, we all know how to monitor disk and network, but when it comes to the state of the brokers, many of us are still unsure of which metrics we should be watching, and what their patterns mean for the state of the cluster. Kafka has hundreds of measurements, from the high-level numbers that are often meaningless to the per-partition metrics that stack up by the thousands as our data grows.
We will thoroughly explore three key monitoring concepts in the broker, that will leave you an expert in identifying problems with the least amount of pain:
Under-replicated Partitions: The mother of all metrics
Request Latencies: Why your users complain
Thread pool utilization: How could 80% be a problem?
We will also discuss the necessity of availability monitoring and how to use it to get a true picture of what your users see, before they come beating down your door!
Uses the example of correct, high-througput, grouping and counting of streaming events as a backdrop for exploring the state-of-the art features of Apache Flink
URP? Excuse You! The Three Metrics You Have to Know confluent
(Todd Palino, LinkedIn) Kafka Summit SF 2018
What do you really know about how to monitor a Kafka cluster for problems? Is your most reliable monitoring your users telling you there’s something broken? Are you capturing more metrics than the actual data being produced? Sure, we all know how to monitor disk and network, but when it comes to the state of the brokers, many of us are still unsure of which metrics we should be watching, and what their patterns mean for the state of the cluster. Kafka has hundreds of measurements, from the high-level numbers that are often meaningless to the per-partition metrics that stack up by the thousands as our data grows.
We will thoroughly explore three key monitoring concepts in the broker, that will leave you an expert in identifying problems with the least amount of pain:
-Under-replicated Partitions: The mother of all metrics
-Request Latencies: Why your users complain
-Thread pool utilization: How could 80% be a problem?
We will also discuss the necessity of availability monitoring and how to use it to get a true picture of what your users see, before they come beating down your door!
Kafka Summit SF 2017 - Kafka and the Polyglot Programmerconfluent
An Overview of the Kafka clients ecosystem. APIs – wire protocol clients – higher level clients (Streams) – REST Languages (with simple snippets – full examples in GitHub) – the most developed clients – Java and C/C++ – the librdkafka wrappers node-rdkafka, python, GO, C# – why use wrappers Shell scripted Kafka ( e.g. custom health checks) kafkacat Platform gotchas (e.g. SASL on Win32)
Presented at Kafka Summit SF 2017 by Edoardo Comar and Andrew Schofield, IBM
How Samsung Engineers Do Pre-Commit Builds with Perforce Helix StreamsPerforce
Get an in-depth look at the life of a pre-commit build at Samsung using Perforce Helix Streams and Electric Cloud’s Electric Commander with Helix Swarm for code review.
Presentation about Code Review and Gerrit given at Confitura 2013 conference. Shows how to start with Gerrit, how configure Ecipse Mylyn Gerrit Connector plugin and how perform code reviews directly in the IDE
We’ve begun an initiative at Citrix to make software development inherently more secure. I’ll start with a few security anecdotes, give you a walkthrough of the security layers from data to physical, and highlight security features along the way. I’ll also discuss the Helix Versioning Engine protocol and show you why SSL encryption should be on by default.
Global Software Development powered by PerforcePerforce
From inception to sunset, hundreds of people from around the world are involved in the production and live operations of video games developed by Electronic Arts. An overview of how EA uses a variety of features in Perforce Helix to effectively utilize its world wide talent pool, develop software efficiently, and protect its intellectual property.
Committing to a company-wide software change is no small feat, but if you’re already sweating at the mere thought of checking code in and out, it’s time to plan your escape route.
So, break free and join Tom Tyler, Senior Consultant at Perforce and in-house ClearCase specialist to map out:
- Baseline-and-branch vs. detail history import strategies
- Porting
- Integrations for defect trackers, training, and tooling
- Cutover strategies
Alfresco Platform Update and Roadmap delivered by Gabriele Columbro, Senior Product Manager for Core Platform / API at Alfresco, with updates on the upcoming Alfresco 5.1 release, on Extreme Scalability (and Solr sharding), Share separation, the new API lifecycle and brand new Developer documentation, samples and tutorials. Mentions of the Upgrade Task Force and new Developer platform improvements like support for JAR modules and tracking / reporting of Share modules.
A Practical Guide to Selecting a Stream Processing Technology confluent
Presented by Michael Noll, Product Manager, Confluent.
Why are there so many stream processing frameworks that each define their own terminology? Are the components of each comparable? Why do you need to know about spouts or DStreams just to process a simple sequence of records? Depending on your application’s requirements, you may not need a full framework at all.
Processing and understanding your data to create business value is the ultimate goal of a stream data platform. In this talk we will survey the stream processing landscape, the dimensions along which to evaluate stream processing technologies, and how they integrate with Apache Kafka. Particularly, we will learn how Kafka Streams, the built-in stream processing engine of Apache Kafka, compares to other stream processing systems that require a separate processing infrastructure.
At Tuenti, we do two code pushes per week, sometimes modifying thousands of files and running thousands of automated tests and build operations before, to ensure not only that the code works but also that proper localization is applied, bundles are generated and files get deployed to hundreds of servers as fast and reliable as possible.
We use opensource tools like Mercurial, MySQL, Jenkins, Selenium, PHPUnit and Rsync among our own in-house ones, and have different development, testing, staging and production environments.
We had to fight with problems like statics bundling and versioning, syntax errors and of course the fact that we have +100 engineers working on the codebase, sometimes merging and releasing more than a dozen branches the same day. We also switched from Subversion to Mercurial to obtain more flexibility and faster branching operations.
With this talk we will explain the process of how code changes in ourcode repository end up in live code, detailing some practices and tips that we apply.
Opinionated containers and the future of game servers by Brendan FosberryDocker, Inc.
Combining my passions for automation and games, I will discuss the opportunity and challenge for automating and containerizing game servers. The necessity to prioritize scale and performance makes game servers a perfect candidate for the container revolution. However many aspects of game servers and apps make this pretty challenging. Starting from the perspective of a typical transition to in-house Docker based micro services at Shopkeep, I’ll take a deep dive into the problems with, benefits of and approaches to containerizing these “opinionated” applications in a wider setting.
URP? Excuse You! The Three Kafka Metrics You Need to KnowTodd Palino
What do you really know about how to monitor a Kafka cluster for problems? Is your most reliable monitoring your users telling you there’s something broken? Are you capturing more metrics than the actual data being produced? Sure, we all know how to monitor disk and network, but when it comes to the state of the brokers, many of us are still unsure of which metrics we should be watching, and what their patterns mean for the state of the cluster. Kafka has hundreds of measurements, from the high-level numbers that are often meaningless to the per-partition metrics that stack up by the thousands as our data grows.
We will thoroughly explore three key monitoring concepts in the broker, that will leave you an expert in identifying problems with the least amount of pain:
Under-replicated Partitions: The mother of all metrics
Request Latencies: Why your users complain
Thread pool utilization: How could 80% be a problem?
We will also discuss the necessity of availability monitoring and how to use it to get a true picture of what your users see, before they come beating down your door!
Uses the example of correct, high-througput, grouping and counting of streaming events as a backdrop for exploring the state-of-the art features of Apache Flink
URP? Excuse You! The Three Metrics You Have to Know confluent
(Todd Palino, LinkedIn) Kafka Summit SF 2018
What do you really know about how to monitor a Kafka cluster for problems? Is your most reliable monitoring your users telling you there’s something broken? Are you capturing more metrics than the actual data being produced? Sure, we all know how to monitor disk and network, but when it comes to the state of the brokers, many of us are still unsure of which metrics we should be watching, and what their patterns mean for the state of the cluster. Kafka has hundreds of measurements, from the high-level numbers that are often meaningless to the per-partition metrics that stack up by the thousands as our data grows.
We will thoroughly explore three key monitoring concepts in the broker, that will leave you an expert in identifying problems with the least amount of pain:
-Under-replicated Partitions: The mother of all metrics
-Request Latencies: Why your users complain
-Thread pool utilization: How could 80% be a problem?
We will also discuss the necessity of availability monitoring and how to use it to get a true picture of what your users see, before they come beating down your door!
Kafka Summit SF 2017 - Kafka and the Polyglot Programmerconfluent
An Overview of the Kafka clients ecosystem. APIs – wire protocol clients – higher level clients (Streams) – REST Languages (with simple snippets – full examples in GitHub) – the most developed clients – Java and C/C++ – the librdkafka wrappers node-rdkafka, python, GO, C# – why use wrappers Shell scripted Kafka ( e.g. custom health checks) kafkacat Platform gotchas (e.g. SASL on Win32)
Presented at Kafka Summit SF 2017 by Edoardo Comar and Andrew Schofield, IBM
How Samsung Engineers Do Pre-Commit Builds with Perforce Helix StreamsPerforce
Get an in-depth look at the life of a pre-commit build at Samsung using Perforce Helix Streams and Electric Cloud’s Electric Commander with Helix Swarm for code review.
Presentation about Code Review and Gerrit given at Confitura 2013 conference. Shows how to start with Gerrit, how configure Ecipse Mylyn Gerrit Connector plugin and how perform code reviews directly in the IDE
Microservices allow for extensible app architecture and a vendor-agnostic, scalable infrastructure. While microservices simplify app deployments, they come at a price: because they’re so fragmented, it is more difficult to track and manage all the independent, yet interconnected components of an app. All this information (requirements, code, test cases and results, build artifacts, and deployment blueprints) needs to live somewhere and most importantly be versioned. Using a real example and a live demonstration of Perforce Helix, Docker and Selenium, get best practices and tips for enabling a robust, scalable and extensible pipeline to support today’s modern app delivery.
[Webinar] The Changing Role of Release Engineering in a DevOps World with J. ...Perforce
The rise of DevOps is revitalizing age-old topics in release engineering and application lifecycle management, and aspects of software delivery that DevOps doesn’t magically solve. If you're responsible for the release engineering function in your organization, see what the new world looks like and which aspects of the industry it’s leaving behind.
Could you release off your mainline today? In our fast paced world well scheduled releases have become a thing of the past. Now more then ever you must maintain clean well tested code lines that can be shipped at any moment. At the last Merge we talked about how these increased demands pushed Xilinx to develop automation that validates every change before submission. In this talk we will continue that discussion covering the evolution of our tools over the past two years as we have battled with more developers, more products, and a faster code churn the ever before.
How Continuous Delivery Helped McKesson Create Award Winning ApplicationsPerforce
Healthcare has always had unique challenges, and as we move through the Affordable Care Act era, it requires new and stronger applications. Choosing the right tool to create and deploy these applications is critical. Hear how CI and CD (before we even knew the terms) contributed to the production of an award-winning electronic health record application, iKnowMed, and how those lessons learned continue to shape McKesson’s ongoing application development and deployment.
Granular Protections Management with TriggersPerforce
Managing the Perforce Helix protections table can be unwieldy at best. Learn how we implemented a trigger-based system that removes the need for an administrator to manually edit the protections table. By granting ownership of individual projects or codelines in the protections table, we can allow project managers to control permissions to a path without worrying about mistakes that could affect the entire company.
Data Summer Conf 2018, “Building unified Batch and Stream processing pipeline...Provectus
Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing pipelines, and also data ingestion and integration flows, supporting for both batch and streaming use cases. In presentation I will provide a general overview of Apache Beam and programming model comparison Apache Beam vs Apache Spark.
Headaches and Breakthroughs in Building Continuous ApplicationsDatabricks
At SpotX, we have built and maintained a portfolio of Spark Streaming applications -- all of which process records in the millions per minute. From pure data ingestion, to ETL, to real-time reporting, to live customer-facing products and features, continuous applications are in our DNA. Come along with us as we outline our journey from square one to present in the world of Spark Streaming. We'll detail what we've learned about efficient processing and monitoring, reliability and stability, and long term support of a streaming app. Come learn from our mistakes, and leave with some handy settings and designs you can implement in your own streaming apps.
Monoliths are so 2001 – What you need is ModularityGraham Charters
Presentation given at IBM InterConnect 2015 conference. Describes:
- the motivation for modularity
- issues with modularity in Java
- introduction to OSGi and WebSphere OSGi Applications
- strategy for adopting OSGi with existing Java EE applications, using a sample (AcmeAir) as a use case
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...Landon Robinson
At SpotX, we have built and maintained a portfolio of Spark Streaming applications -- all of which process records in the millions per minute. From pure data ingestion, to ETL, to real-time reporting, to live customer-facing products and features, continuous applications are in our DNA. Come along with us as we outline our journey from square one to present in the world of Spark Streaming. We'll detail what we've learned about efficient processing and monitoring, reliability and stability, and long term support of a streaming app. Come learn from our mistakes, and leave with some handy settings and designs you can implement in your own streaming apps.
Presented by Landon Robinson and Jack Chapa
Five Real-World Strategies for Perforce StreamsPerforce
Before you deploy Perforce Streams in your organization, you should have a plan in place. Get advice and hear the five strategies for using Streams and how to handle integration exceptions gracefully.
Digital Fabrication Studio.02 _Information @ Aalto Media FactoryMassimo Menichinelli
DIGITAL FABRICATION STUDIO (25438)
The course provides a general understanding on how to design and manufacture products and prototypes in a Fab Lab, using digital fabrication technologies and understanding their features and limits.
Students will learn how information shapes design, manufacturing and collaboration processes and artifacts in a Fab Lab. They will learn how to digitally fabricate a project or how to digitally modify an existing project; students will also learn how to manage, embed and retrieve information about a project. Projects and prototypes developed and manufactured in this course will not be interactive.
The course consists of lectures and a group project to be digitally fabricated, be it a project already designed but not yet realized or be it the modification of an existing project. Every lecture (3 hours) includes time for testing the technologies covered (1 hour) and for developing part of the group project and for receiving feedback about it (1 hour).
http://mlab.taik.fi/studies/courses/course?id=1963
Optimize DR and Cloning with Logical Hostnames in Oracle E-Business Suite (OA...Andrejs Prokopjevs
This presentation covers the idea of logical hostname feature and its possible use case with E-Business Suite, why it is a must-have configuration for DR, how it can improve your test/dev instance cloning and lifecycle processes, especially in a cloud deployment, support overview by 11i/R12.0/R12.1, and why it is a very hot topic right now for R12.2. Additionally, we will describe possible advanced configuration scenarios like container based virtualization. The content is based on real client environment implementation experience.
3 Ways to Improve Performance from a Storage PerspectivePerforce
In this session, get three takeaways about Perforce performance benchmarks and their results across varying storage protocols, using NetApp storage as an example. Learn how to use Perforce benchmarks and tools to validate the performance of your Perforce deployment; understand Perforce performance across different storage protocols; and get tips and tricks for deploying Perforce on varying storage technologies.
DBCC 2021 - FLiP Stack for Cloud Data LakesTimothy Spann
DBCC 2021 - FLiP Stack for Cloud Data Lakes
With Apache Pulsar, Apache NiFi, Apache Flink. The FLiP(N) Stack for Event processing and IoT. With StreamNative Cloud.
DBCC International – Friday 15.10.2021
Powered by Apache Pulsar, StreamNative provides a cloud-native, real-time messaging and streaming platform to support multi-cloud and hybrid cloud strategies.
Moving 1,000 Users & 100 Branches into StreamsPerforce
MathWorks has approximately 100 products derived from a single large code base, with over 1,000 developers contributing changes to almost one million source files. Their products are used to develop safety-critical systems, so managing a continuous influx of changes while guaranteeing quality and correctness is challenging. Learn how MathWorks, unable to use a simple model of “component as a directory,” created an elegant system using virtual streams and the Perforce broker in their ongoing efforts to modularize their code base.
Talk at RubyKaigi 2015.
Plugin architecture is known as a technique that brings extensibility to a program. Ruby has good language features for plugins. RubyGems.org is an excellent platform for plugin distribution. However, creating plugin architecture is not as easy as writing code without it: plugin loader, packaging, loosely-coupled API, and performance. Loading two versions of a gem is a unsolved challenge that is solved in Java on the other hand.
I have designed some open-source software such as Fluentd and Embulk. They provide most of functions by plugins. I will talk about their plugin-based architecture.
OpenLDAP has been replacing proprietary directory server offerings in the private sector, public sector and the financial sector at an increasing pace. This is largely due to its performance and scalability, dynamic configuration capabilities and flexible extensibility via bundled modules.
OpenLDAP would not have earned its place in these sectors without enterprise grade replication options.
In this talk, an overview of the latest production ready OpenLDAP 2.4 replication features will be discussed and the numerous best practice strategies will be presented covering the most common deployment configurations found in the wild.
Similar to How to Reduce Database Load with Sparse Branches (20)
How to Organize Game Developers With Different Planning NeedsPerforce
Different skills have different needs when it comes to planning. For a coder it may make perfect sense to plan work in two-week sprints, but for an artist, an asset may take longer than two weeks to complete.
How do you allow different skills to plan the way that works best for them? Some studios may choose to open up for flexibility – do whatever you like! But that tends to cause issues with alignment and siloes of data, resulting in loss of vision. Lost vision in the sense that it is difficult to understand, but also — and maybe more importantly — the risk of losing the vision of what the game will be.
With the right approach, however, you can avoid these obstacles. Join backlog expert Johan Karlsson to learn:
-The balance of team autonomy and alignment.
-How to use the product backlog to align the project vision.
-How to use tools to support the flexibility you need.
Looking for a planning and backlog tool? You can try Hansoft for free.
Regulatory Traceability: How to Maintain Compliance, Quality, and Cost Effic...Perforce
How do regulations impact your product requirements? How do you ensure that you identify all the needed requirements changes to meet these regulations?
Ideally, your regulations should live alongside your product requirements, so you can trace among each related item. Getting to that point can be quite an undertaking, however. Ultimately you want a process that:
-Saves money
-Ensures quality
-Avoids fines
If you want help achieving these goals, this webinar is for you. Watch Tom Totenberg, Senior Solutions Engineer for Helix ALM, show you:
-How to import a regulation document into Helix ALM.
-How to link to requirements.
-How to automate impact analysis from regulatory updates.
Efficient Security Development and Testing Using Dynamic and Static Code Anal...Perforce
Be sure to register for a demo, if you would like to see how Klocwork can help ensure that your code is secure, reliable, and compliant.
https://www.perforce.com/products/klocwork/live-demo
If it’s not documented, it didn’t happen.
When it comes to compliance, if you’re doing the work, you need to prove it. That means having well-documented SOPs (standard operating procedures) in place for all your regulated workflows.
It also means logging your efforts to enforce these SOPs. They show that you took appropriate action in any number of scenarios, which can be related to regulations, change requests, firing of an employee, logging an HR compliant, or anything else that needs a structured workflow.
But when do you need to do this, and how do you go about it?
In this webinar, Tom Totenberg, our Helix ALM senior solutions engineer, clarifies workflow enforcement SOPs, along with a walkthrough of how Perforce manages GDPR (General Data Protection Regulation) requests. He’ll cover:
-What are SOPs?
-Why is it important to have this documentation?
-Example: walking through our internal Perforce GDPR process.
-What to beware of.
-Building the workflow in ALM.
Branching Out: How To Automate Your Development ProcessPerforce
If you could ship 20% faster, what would it mean for your business? What could you build? Better question, what’s slowing your teams down?
Teams struggle to manage branching and merging. For bigger teams and projects, it gets even more complex. Tracking development using a flowchart, team wiki, or a white board is ineffective. And attempts to automate with complex scripting are costly to maintain.
Remove the bottlenecks and automate your development your way with Perforce Streams –– the flexible branching model in Helix Core.
Join Brad Hart, Chief Technology Officer and Brent Schiestl, Senior Product Manager for Perforce version control to learn how Streams can:
-Automate and customize development and release processes.
-Easily track and propagate changes across teams.
-Boost end user efficiency while reducing errors and conflicts.
-Support multiple teams, parallel releases, component-based development, and more.
How to Do Code Reviews at Massive Scale For DevOpsPerforce
Code review is a critical part of your build process. And when you do code review right, you can streamline your build process and achieve DevOps.
Most code review tools work great when you have a team of 10 developers. But what happens when you need to scale code review to 1,000s of developers? Many will struggle. But you don’t need to.
Join our experts Johan Karlsson and Robert Cowham for a 30-minute webinar. You’ll learn:
-The problems with scaling code review from 10s to 100s to 1,000s of developers along with other dimensions of scale (files, reviews, size).
-The solutions for dealing with all dimensions of scale.
-How to utilize Helix Swarm at massive scale.
Ready to scale code review and streamline your build process? Get started with Helix Swarm, a code review tool for Helix Core.
By now many of us have had plenty of time to clean and tidy up our homes. But have you given your product backlog and task tracking software as much attention?
To keep your digital tools organized, it is important to avoid hoarding on to inefficient processes. By removing the clutter in your product backlog, you can keep your teams focused.
It’s time to spark joy by cleaning up your planning tools!
Join Johan Karlsson — our Agile and backlog expert — to learn how to:
-Apply digital minimalism to your tracking and planning.
-Organize your work by category.
-Motivate teams by transitioning to a cleaner way of working.
TRY HANSOFT FREE
Going Remote: Build Up Your Game Dev Team Perforce
Everyone’s working remote as a result of the coronavirus (COVID-19). And while game development has always been done with remote teams, there’s a new challenge facing the industry.
Your audience has always been mostly at home – now they may be stuck there. And they want more games to stay happy and entertained.
So, how can you enable your developers to get files and feedback faster to meet this rapidly growing demand?
In this webinar, you’ll learn:
-How to meet the increasing demand.
-Ways to empower your remote teams to build faster.
-Why Helix Core is the best way to maximize productivity.
Plus, we’ll share our favorite games keeping us happy in the midst of a pandemic.
Shift to Remote: How to Manage Your New WorkflowPerforce
The spread of coronavirus has fundamentally changed the way people work. Companies around the globe are making an abrupt shift in how they manage projects and teams to support their newly remote workers.
Organizing suddenly distributed teams means restructuring more than a standup. To facilitate this transition, teams need to update how they collaborate, manage workloads, and maintain projects.
At Perforce, we are here to help you maintain productivity. Join Johan Karlsson — our Agile expert — to learn how to:
Keep communication predictable and consistent.
-Increase visibility across teams.
-Organize projects, sprints, Kanban boards and more.
-Empower and support your remote workforce.
Hybrid Development Methodology in a Regulated WorldPerforce
In a regulated industry, collaboration can be vital to building quality products that meet compliance. But when an Agile team and a Waterfall team need to work together, it can feel like mixing oil with water.
If you're used to Agile methods, Waterfall can feel slow and unresponsive. From a Waterfall perspective, pure Agile may lack accountability and direction. Misaligned teams can slow progress, and expose your development to mistakes that undermine compliance.
It's possible to create the best of both worlds so your teams can operate together harmoniously. This is how to develop products quickly, and still make regulators happy.
Join ALM Solutions Engineer Tom Totenberg in this webinar to learn how teams can:
- Operate efficiently with differing methodologies.
- Glean best practices for their tailored hybrid.
- Work together in a single environment.
Watch the webinar, and when you're ready for a tool to help you with the hybrid, know that you can try Helix ALM for free.
Better, Faster, Easier: How to Make Git Really Work in the EnterprisePerforce
There's a lot of reasons to love Git. (Git is awesome at what it does.) Let’s look at the 3 major use cases for Git in the enterprise:
1. You work with third party or outsourced development teams.
2. You use open source in your products.
3. You have different workflow needs for different teams.
Making the best of Git can be difficult in an enterprise environment. Trying to manage all the moving parts is like herding cats.
So, how do you optimize your teams’ use of Git — and make it all fit into your vision of the enterprise SDLC?
You’ll learn about:
-The challenges that accompany each use case — third parties, open source code, different workflows.
-Ways to solve these problems.
-How to make Git better, faster, and easier — with Perforce
Easier Requirements Management Using Diagrams In Helix ALMPerforce
Sometimes requirements need visuals. Whether it’s a diagram that clarifies an idea or a screenshot to capture information, images can help you manage requirements more efficiently. And that means better quality products shipped faster.
In this webinar, Helix ALM Professional Services Consultant Gerhard Krüger will demonstrate how to use visuals in ALM to improve requirements. Learn how to:
-Share information faster than ever.
-Drag and drop your way to better teamwork.
-Integrate various types of visuals into your requirements.
-Utilize diagram and flowchart software for every need.
-And more!
Immediately apply the information in this webinar for even better requirements management using Helix ALM.
It’s common practice to keep a product backlog as small as possible, probably just 10-20 items. This works for single teams with one Product Owner and perhaps a Scrum Master.
But what if you have 100 Scrum teams managing a complex system of hardware and software components? What do you need to change to manage at such a massive scale?
Join backlog expert Johan Karlsson to learn how to:
-Adapt Agile product backlog practices to manage many backlogs.
-Enhance collaboration across disciplines.
-Leverage backlogs to align teams while giving them flexibility.
Achieving Software Safety, Security, and Reliability Part 3: What Does the Fu...Perforce
In Part 3, we will look at what the future might hold for embedded programming languages and development tools. And, we will look at the future for software safety and security standards.
How to Scale With Helix Core and Microsoft Azure Perforce
Microsoft Azure helps teams increase their speed, gain flexibility, and save time. Using Helix Core with Azure you maximizes cloud benefits. You can scale to meet both current and future deployment demands. And this powerful combination helps secure your most valuable IP assets.
So, where do you start? What do you need to set up your teams for success? How can you expedite your pipelines to deliver ahead of your competitors?
Join Chuck Gehman from Perforce to learn more about:
-Compute, storage, and security options from Azure.
-Strategies that boost your cloud investment.
-Tips to secure your data.
-Best practices for global deployments.
Achieving Software Safety, Security, and Reliability Part 2Perforce
In Part 2, we will focus on the automotive industry, as it leads the way in enforcing safety, security, and reliability standards as well as best practices for software development. We will then examine how other industries could adopt similar practices.
Modernizing an application’s architecture is often a necessary multi-year project in the making. The goal –– to stabilize code, detangle dependencies, and adopt a toolset that ignites innovation.
Moving your monolith repository to a microservices/component based development model might be on trend. But is it right for you?
Before you break up with anything, it is vital to assess your needs and existing environment to construct the right plan. This can minimize business risks and maximize your development potential.
Join Tom Tyler and Chuck Gehman to learn more about:
-Why you need to plan your move with the right approach.
-How to reduce risk when refactoring your monolithic repository.
-What you need to consider before migrating code.
Achieving Software Safety, Security, and Reliability Part 1: Common Industry ...Perforce
In part one of our three-part webinar series, we examine common software development challenges, review the safety and security standards adopted by different industries, and examine the best practices that can be applied to any software development team.
The features you’ve been waiting for! Helix ALM’s latest update expands usability and functionality to bring solid improvements to your processes.
Watch Helix ALM Senior Product Manager Paula Rome demonstrate how new features:
-Simplify workflows.
-Expand report analysis.
-Boost productivity in the Helix ALM web client.
All this and MORE packed into an exciting 30 minutes! Get inspired. Be extraordinary with the new Helix ALM.
Companies that track requirements, create traceability matrices, and complete audits - especially for compliance - run into many problems using only Word and Excel to accomplish these tasks.
Most notably, manual processes leave employees vulnerable to making costly mistakes and wasting valuable time.
These outdated tracking procedures rob organizations of benefiting from four keys to productivity and efficiency:
-Automation
-Collaboration
-Visibility
-Traceability
However, modern application lifecycle management (ALM) tools solve all of these problems, linking and organizing information into a single source of truth that is instantly auditable.
Gerhard Krüger, senior consultant for Helix ALM, explains how the right software supports these fundamentals, generating improvements that save time and money.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
How Recreation Management Software Can Streamline Your Operations.pptxwottaspaceseo
Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteGoogle
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-pilot-review/
AI Pilot Review: Key Features
✅Deploy AI expert bots in Any Niche With Just A Click
✅With one keyword, generate complete funnels, websites, landing pages, and more.
✅More than 85 AI features are included in the AI pilot.
✅No setup or configuration; use your voice (like Siri) to do whatever you want.
✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It…
✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again.
✅ZERO Limits On Features Or Usages
✅Use Our AI-powered Traffic To Get Hundreds Of Customers
✅No Complicated Setup: Get Up And Running In 2 Minutes
✅99.99% Up-Time Guaranteed
✅30 Days Money-Back Guarantee
✅ZERO Upfront Cost
See My Other Reviews Article:
(1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
How to Reduce Database Load with Sparse Branches
1. How to Reduce Database
Load with Sparse Branches
John LoVerso
Software Archeologist
2. 2
MathWorks
We are a 3500+ person company dedicated to accelerating
the pace of engineering and science
We have ~90 products based upon our core platforms
• MATLAB – The Language of Technical Computing
• Simulink – Simulation and Model-Based Design
3. 3
Technical Challenges
Unified code base from which full product family is released
twice a year
Managing an over 1 million file code base
• 5,000+ components, acyclic dependencies
Integrating changes from ~1500 developers
on ~270 active branches (streams)
5. 5
Merge Down and Merge Up
Build and Test infrastructure blesses submitted changes
• Qualify changes already submitted to the branch
Merge down from the last blessed change-level
• Almost never merge from the tip
Cannot copy up; must merge
6. 6
Componentized Streams
Our branches are built out of components
We use pairs of stream specs for each branch
For details, see
• Moving 1000 Users and 100 Branches Into Streams
- from MERGE2014
• Outsmarting Merge Edge Cases in Component Based Development
- from MERGE2016
7. 7
Pair of Streams
Wide-open development stream specs
Stream: //mw/product
Parent: //mw/integ
Paths:
share …
Virtual streams computed from component information
Stream: //mw/product~CTB
Parent: //mw/product
Paths:
share matlab/toolbox/prodA/...
share matlab/toolbox/prodB/...
8. 8
Average Week
Changes
Files
Min Average Max Total
New Work 3328 1 8 52 25641
Merges 610 1 1890 39305 1153239
Changes submitted over one week in our main product depot
9. 9
2015.1 Merge Meltdown
Many regressions in integration engine
Lost weeks of developer time
All because we merge all the time
10. 10
What If We Didn’t Need to Merge?
At any time, the number of files on a branch with unique
changes is small
The rest of the files are the same as on the parent branch
11. 11
What is Sparse Branching?
Sparse branching, a.k.a. lightweight or just-in-time
branching, is a strategy where files are only branched when
modified and are otherwise just a reference to a file on the
corresponding parent branch
Initial creation of a sparse branch is an O(1) operation akin
to creating a clone or snapshot in a copy-on-write filesystem
like ZFS
12. 12
Why not simply use Task Streams?
They are database expensive
• Order(n) rather than Order(1) to create
• They consume database space until deleted
They require unnecessary merging in order to be kept up-to-
date with their parent
• Still exposes the user to the vagaries of complex merges
They have limitations
• You can only merge from their parent
• Can’t recreate them if they are destroyed
• No virtual stream support
13. 13
Some Terms
Winked-in file
• a file mapped to a revision on an ancestor branch (lazy copy)
Active file
• a file with changes that have not yet propagated to the parent branch
Make concrete
• the act of branching a winked-in file in order to make it active
14. 14
Our Approach to Sparse Branches
A sparse branch is defined by the stream Paths:
• Winked-in files use “import path@change” to map the paths from an
ancestor branch
• Files are made active on the branch by inserting “share” lines for each file
after populating it on the sparse branch
Moving ahead to a new parent change involves
• Advancing the change number in the “import path@change” directives
• Merging changes down from the parent into active files
15. 15
Pair of Streams
Wide-open development stream specs
Stream: //mw/product
Parent: //mw/integ
Paths:
share …
Virtual streams computed from component information
Stream: //mw/product~CTB
Parent: //mw/product
Paths:
share matlab/toolbox/prodA/...
share matlab/toolbox/prodB/...
16. 16
Pair of Streams – Sparse Branch
Wide-open development stream specs
Stream: //mw/product
Parent: //mw/integ
Paths:
share …
Virtual streams computed from component information
Stream: //mw/product~CTB
Parent: //mw/product
Paths:
import matlab/toolbox/prodA/... //mw/integ/matlab/toolbox/prodA/…@1000
import matlab/toolbox/prodB/... //mw/integ/matlab/toolbox/prodB/…@1000
17. 17
Pair of Streams – Sparse Branch
With Active Files
Wide-open development stream specs
Stream: //mw/product
Parent: //mw/integ
Paths:
share …
Virtual streams computed from component information
Stream: //mw/product~CTB
Parent: //mw/product
Paths:
import matlab/toolbox/prodA/... //mw/integ/matlab/toolbox/prodA/…@1000
import matlab/toolbox/prodB/... //mw/integ/matlab/toolbox/prodB/…@1000
share matlab/toolbox/prodA/file1
19. 20
Transparent to the User
Once a sparse branch is created, user commands should be
entirely agnostic to the nature of the branch
• add/edit/delete/move/unshelve/merge should all just work
Updating to a newer change level of the parent is special
• merge + revert of newly branched files
20. 21
But what about new work?
We have explored two approaches:
Branch On Edit
• Just-in-time branching of a file as soon as “p4 edit” happens
Branch On Submit
• Files are opened on the parent branch and are only branched at submit
time
21. 22
Branch On Edit
Has the benefit of being easier to implement
Broker wrapper to intercept operations that open files
• edit, delete, move, open, unshelve, merge
• Compute all the files about to be operated on
• Invoke ‘p4 populate’ to make concrete revisions on the sparse branch of
winked-in files
• Invoke ‘p4 flush’ to switch have revision
• Let operation complete to affect opened revision
Trigger (change-content) to add files to active list on submit
22. 23
How It Looks
$ p4 stream -t sparse_v1 -P //mw/robotics@1705498 //pdb/jloverso/robotics/demo
Stream //pdb/jloverso/robotics/demo created.
$ p4 stream -o //pdb/jloverso/robotics/demo~CTB | tail -2
import matlab/toolbox/robotics/... //mw/robotics/matlab/toolbox/robotics/...@1705498
$ p4 files //pdb/jloverso/robotics/demo/...
//pdb/jloverso/robotics/demo/... - no such file(s).
24. 25
How It Looks - Edit
$ p4 have matlab/toolbox/robotics/Makefile
//mw/robotics/matlab/toolbox/robotics/Makefile#3 -
/sandbox/jloverso.demo/matlab/toolbox/robotics/Makefile
$ p4 edit matlab/toolbox/robotics/Makefile
//pdb/jloverso/robotics/demo/matlab/toolbox/robotics/Makefile#1 - opened for edit
$ p4 have matlab/toolbox/robotics/Makefile
//pdb/jloverso/robotics/demo/matlab/toolbox/robotics/Makefile#1 -
/sandbox/jloverso.demo/matlab/toolbox/robotics/Makefile
25. 26
How It Looks – Dynamically Branched
$ p4 files //pdb/jloverso/robotics/demo/...
//pdb/jloverso/robotics/demo/matlab/toolbox/robotics/Makefile#1 - branch change
1722340 (text)
$ p4 filelog //pdb/jloverso/robotics/demo/matlab/toolbox/robotics/Makefile
//pdb/jloverso/robotics/demo/matlab/toolbox/robotics/Makefile
... #1 change 1722340 branch on 2016/03/31 by jloverso@jloverso.demo (text)
'Dynamically branched'
... ... branch from //mw/robotics/matlab/toolbox/robotics/Makefile#1,#4
27. 28
Branch On Submit
Provides a truer version of copy-on-write semantics
Pending changes are fully discardable with no remnants or
commit server impact
Requires the ability to re-base (reopen) the files in a pending
change from one branch to another
• Can be done by creating journal entries in order to modify entries in
db.have, db.working, and db.locks tables
• This is known in 2015.2 as “p4 sync –r”
28. 29
Status
Branch-on-edit in production use for
• private developer branches
• “fixes” branches that are created on-the-fly for each change that is
processed by our Build & Test automation
We have a prototype of branch-on-submit
• Some limitations; being worked on
Both versions support multilevel hierarchies
29. 30
Future Plans
Our goal is to get all non-mainline branches to be sparse
• This is where we can truly reduce database sizes
Possibility of open-source release of broker and trigger logic
• Some internal dependencies need to be eliminated