Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2KhnfRX.
Howard Chu talks about both naive approaches to leveraging NVRAM, and reasons to avoid those approaches, as well as optimal, proven methods for building systems around persistent memory. Filmed at qconlondon.com.
Howard Chu’s work has spanned a wide range of computing topics, including most of the GNU utilities, networking protocols and tools, kernel and filesystem drivers, and focused on maximizing the useful work from a system. He founded Symas Corp. with 5 other partners and serves as its CTO. His current focus is database oriented, covering LDAP, LMDB, and other non-relational database technologies.
Webinar: OpenEBS - Still Free and now FASTEST Kubernetes storageMayaData Inc
Webinar Session - https://youtu.be/_5MfGMf8PG4
In this webinar, we share how the Container Attached Storage pattern makes performance tuning more tractable, by giving each workload its own storage system, thereby decreasing the variables needed to understand and tune performance.
We then introduce MayaStor, a breakthrough in the use of containers and Kubernetes as a data plane. MayaStor is the first containerized data engine available that delivers near the theoretical maximum performance of underlying systems. MayaStor performance scales with the underlying hardware and has been shown, for example, to deliver in excess of 10 million IOPS in a particular environment.
Linux Memory Basics for SysAdmins - ChinaNetCloud TrainingChinaNetCloud
ChinaNetCloud training for Linux Memory Basics for SysAdmins.
This is an introduction to general Linux memory for troubleshooting, monitoring, and basic understanding.
2689 - Exploring IBM PureApplication System and IBM Workload Deployer Best Pr...Hendrik van Run
IBM IMPACT 2013 presentation
This lecture will provide an overview of a combination of design, development, configuration and deployment best practices for IBM PureApplication System and IBM Workload Deployer captured from customer engagement experiences.
Coherence and consistency models in multiprocessor architectureUniversity of Pisa
Cache coherence and consistency model in multiprocessor architecture. These slide show the introduction of multiprocessor and cache multilevel and then describe the basic mechanism of coherence and consistency protocols. In particular the protocols describe are the following: snooping and directory protocols for the coherence part and sequential protocol for the consistency part. There are also example of (in)consistency and (in)coherence.
Internet of Things (IoT) data frequently has a location and time component. Getting value out of this "geotemporal" data can be tricky. We'll explore when and how to leverage Cassandra, DSE Search and DSE Analytics to surface meaningful information from your geotemporal data.
Red Hat Enterprise Linux: Open, hyperconverged infrastructureRed_Hat_Storage
The next generation of IT will be built around flexible infrastructures and operational efficiencies, lowering costs and increasing overall business value in the organization.
A hyperconverged infrastructure that's built on Red Hat supported technologies--including Linux, Gluster storage, and oVirt virtualization manager--will run on commodity x86 servers using the performance of local storage, to deliver a cost-effective, modular, highly scalable, and secure hyperconverged solution.
C++ Programming and the Persistent Memory Developers KitIntel® Software
Topics
Introduction to Persistent Memory
Introduction to Persistent Memory Developers Kit (PMDK)
Working with PMDK
Persistent Memory Programming with PMDK C++ Bindings
Streaming a Million Likes/Second: Real-Time Interactions on Live VideoC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/39NIjLV.
Akhilesh Gupta does a technical deep-dive into how Linkedin uses the Play/Akka Framework and a scalable distributed system to enable live interactions like likes/comments at massive scale at extremely low costs across multiple data centers. Filmed at qconlondon.com.
Akhilesh Gupta is the technical lead for LinkedIn's Real-time delivery infrastructure and LinkedIn Messaging. He has been working on the revamp of LinkedIn’s offerings to instant, real-time experiences. Before this, he was the head of engineering for the Ride Experience program at Uber Technologies in San Francisco.
Next Generation Client APIs in Envoy MobileC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2x0Fav8.
Jose Nino guides the audience through the journey of Mobile APIs at Lyft. He focuses on how the team has reaped the benefits of API generation to experiment with the network transport layer. He also discusses recent developments the team has made with Envoy Mobile and the roadmap ahead. Filmed at qconlondon.com.
Jose Nino works as a Software Engineer at Lyft.
More Related Content
Similar to Software Design for Persistent Memory Systems
Webinar: OpenEBS - Still Free and now FASTEST Kubernetes storageMayaData Inc
Webinar Session - https://youtu.be/_5MfGMf8PG4
In this webinar, we share how the Container Attached Storage pattern makes performance tuning more tractable, by giving each workload its own storage system, thereby decreasing the variables needed to understand and tune performance.
We then introduce MayaStor, a breakthrough in the use of containers and Kubernetes as a data plane. MayaStor is the first containerized data engine available that delivers near the theoretical maximum performance of underlying systems. MayaStor performance scales with the underlying hardware and has been shown, for example, to deliver in excess of 10 million IOPS in a particular environment.
Linux Memory Basics for SysAdmins - ChinaNetCloud TrainingChinaNetCloud
ChinaNetCloud training for Linux Memory Basics for SysAdmins.
This is an introduction to general Linux memory for troubleshooting, monitoring, and basic understanding.
2689 - Exploring IBM PureApplication System and IBM Workload Deployer Best Pr...Hendrik van Run
IBM IMPACT 2013 presentation
This lecture will provide an overview of a combination of design, development, configuration and deployment best practices for IBM PureApplication System and IBM Workload Deployer captured from customer engagement experiences.
Coherence and consistency models in multiprocessor architectureUniversity of Pisa
Cache coherence and consistency model in multiprocessor architecture. These slide show the introduction of multiprocessor and cache multilevel and then describe the basic mechanism of coherence and consistency protocols. In particular the protocols describe are the following: snooping and directory protocols for the coherence part and sequential protocol for the consistency part. There are also example of (in)consistency and (in)coherence.
Internet of Things (IoT) data frequently has a location and time component. Getting value out of this "geotemporal" data can be tricky. We'll explore when and how to leverage Cassandra, DSE Search and DSE Analytics to surface meaningful information from your geotemporal data.
Red Hat Enterprise Linux: Open, hyperconverged infrastructureRed_Hat_Storage
The next generation of IT will be built around flexible infrastructures and operational efficiencies, lowering costs and increasing overall business value in the organization.
A hyperconverged infrastructure that's built on Red Hat supported technologies--including Linux, Gluster storage, and oVirt virtualization manager--will run on commodity x86 servers using the performance of local storage, to deliver a cost-effective, modular, highly scalable, and secure hyperconverged solution.
C++ Programming and the Persistent Memory Developers KitIntel® Software
Topics
Introduction to Persistent Memory
Introduction to Persistent Memory Developers Kit (PMDK)
Working with PMDK
Persistent Memory Programming with PMDK C++ Bindings
Streaming a Million Likes/Second: Real-Time Interactions on Live VideoC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/39NIjLV.
Akhilesh Gupta does a technical deep-dive into how Linkedin uses the Play/Akka Framework and a scalable distributed system to enable live interactions like likes/comments at massive scale at extremely low costs across multiple data centers. Filmed at qconlondon.com.
Akhilesh Gupta is the technical lead for LinkedIn's Real-time delivery infrastructure and LinkedIn Messaging. He has been working on the revamp of LinkedIn’s offerings to instant, real-time experiences. Before this, he was the head of engineering for the Ride Experience program at Uber Technologies in San Francisco.
Next Generation Client APIs in Envoy MobileC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2x0Fav8.
Jose Nino guides the audience through the journey of Mobile APIs at Lyft. He focuses on how the team has reaped the benefits of API generation to experiment with the network transport layer. He also discusses recent developments the team has made with Envoy Mobile and the roadmap ahead. Filmed at qconlondon.com.
Jose Nino works as a Software Engineer at Lyft.
Software Teams and Teamwork Trends Report Q1 2020C4Media
How do we cope with an environment that has been radically disrupted, where people are suddenly thrust into remote work in a chaotic state? What are the emerging good practices and new ideas that are shaping the way in which software development teams work? What can we do to make the workplace a more secure and diverse one while increasing the productivity of our teams? This report aims to assist technical leaders in making mid- to long-term decisions that will have a positive impact on their organisations and teams and help individual contributors find the practices, approaches, tools, techniques, and frameworks that can help them get a better experience at work - irrespective of where they are working from.
Understand the Trade-offs Using Compilers for Java ApplicationsC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2QCmmJ0.
Mark Stoodley examines some of the strengths and weaknesses of the different Java compilation technologies, if one was to apply them in isolation. Stoodley discusses how production JVMs are assembling a combination of these tools that work together to provide excellent performance across the large spectrum of applications written in Java and JVM based languages. Filmed at qconsf.com.
Mark Stoodley joined IBM Canada to build Java JIT compilers for production use and led the team that delivered AOT compilation in the IBM SDK for Java 6. He spent the last five years leading the effort to open source nearly 4.3 million lines of source code from the IBM J9 Java Virtual Machine to create the two open source projects Eclipse OMR and Eclipse OpenJ9, and now co-leads both projects.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2y2yPiS.
Colin McCabe talks about the ongoing effort to replace the use of Zookeeper in Kafka: why they want to do it and how it will work. He discusses the limitations they have found and how Kafka benefits both in terms of stability and scalability by bringing consensus in house. He talks about their progress, what work is remaining, and how contributors can help. Filmed at qconsf.com.
Colin McCabe is a Kafka committer at Confluent, working on the scalability and extensibility of Kafka. Previously, he worked on the Hadoop Distributed Filesystem and the Ceph Filesystem.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2SXXXiD.
Katharina Probst talks about what it means to act like an owner and why teams need ownership to be high-performing. When team members, regardless of whether they have a formal leadership role or not, act like owners, magical things can happen. She shares ideas that we can apply to our own work, and talks about how to recognize when we don’t live up to our own expectations of acting like an owner. Filmed at qconsf.com.
Katharina Probst is a Senior Engineering Leader, Kubernetes & SaaS at Google. Before this, she was leading engineering teams at Netflix, being responsible for the Netflix API, which helps bring Netflix streaming to millions of people around the world. Prior to joining Netflix, she was in the cloud computing team at Google, where she saw cloud computing from the provider side.
Does Java Need Inline Types? What Project Valhalla Can Bring to JavaC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2T04Lw4.
Sergey Kuksenko talks about the performance benefits inline types bring to Java and how to exploit them. Inline/value types are the key part of experimental project Valhalla, which should bring new abilities to the Java language. Filmed at qconsf.com.
Sergey Kuksenko is a Java Performance Engineer at Oracle working on a variety of Java and JVM performance enhancements. He started working as Java Engineer in 1996 and as Java Performance Engineer in 2005. He has had a passion for exploring how Java works on modern hardware.
Do you need service meshes in your tech stack?
This on-line guide aims to answer pertinent questions for software architects and technical leaders, such as: what is a service mesh?, do I need a service mesh?, how do I evaluate the different service mesh offerings? In software architecture, a service mesh is a dedicated infrastructure layer for facilitating service-to-service communications between microservices, often using a sidecar proxy.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2UgQ3BU.
Christie Wilson describes what to expect from CI/CD in 2019, and how Tekton is helping bring that to as many tools as possible, such as Jenkins X and Prow. Wilson talks about Tekton itself and performs a live demo that shows how cloud native CI/CD can help debug, surface and fix mistakes faster. Filmed at qconsf.com.
Christie Wilson is a software engineer at Google, currently leading the Tekton project. Over the past decade, she has worked in the mobile, financial and video game industries. Prior to working at Google she led a team of software developers to build load testing tools for AAA video game titles, and founded the Vancouver chapter of PyLadies.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2S7lDiS.
Sasha Rosenbaum shows how a CI/CD pipeline for Machine Learning can greatly improve both productivity and reliability. Filmed at qconsf.com.
Sasha Rosenbaum is a Program Manager on the Azure DevOps engineering team, focused on improving the alignment of the product with open source software. She is a co-organizer of the DevOps Days Chicago and the DeliveryConf conferences, and recently published a book on Serverless computing in Azure with .NET.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/36epVKg.
Todd Montgomery discusses the techniques and lessons learned from implementing Aeron Cluster. His focus is on how Raft can be implemented on Aeron, minimizing the network round trip overhead, and comparing single process to a fully distributed cluster. Filmed at qconsf.com.
Todd Montgomery is a networking hacker who has researched, designed, and built numerous protocols, messaging-oriented middleware systems, and real-time data systems, done research for NASA, contributed to the IETF and IEEE, and co-founded two startups. He currently works as an independent consultant and is active in several open source projects.
Architectures That Scale Deep - Regaining Control in Deep SystemsC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2FWc5Sk.
Ben Sigelman talks about "Deep Systems", their common properties and re-introduces the fundamentals of control theory from the 1960s, including the original conceptualizations of Observability & Controllability. He uses examples from Google & other companies to illustrate how deep systems have damaged people's ability to observe software, and what needs to be done in order to regain control. Filmed at qconsf.com.
Ben Sigelman is a co-founder and the CEO at LightStep, a co-creator of Dapper (Google’s distributed tracing system), and co-creator of the OpenTracing and OpenTelemetry projects (both part of the CNCF). His work and interests gravitate towards observability, especially where microservices, high transaction volumes, and large engineering organizations are involved.
ML in the Browser: Interactive Experiences with Tensorflow.jsC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/39SddUL.
Victor Dibia provides a friendly introduction to machine learning, covers concrete steps on how front-end developers can create their own ML models and deploy them as part of web applications. He discusses his experience building Handtrack.js - a library for prototyping real time hand tracking interactions in the browser. Filmed at qconsf.com.
Victor Dibia is a Research Engineer with Cloudera’s Fast Forward Labs. Prior to this, he was a Research Staff Member at the IBM TJ Watson Research Center, New York. His research interests are at the intersection of human computer interaction, computational social science, and applied AI.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2s9T3Vl.
Colin Eberhardt looks at some of the internals of WebAssembly, explores how it works “under the hood”, and looks at how to create a (simple) compiler that targets this runtime. Filmed at qconsf.com.
Colin Eberhardt is the Technology Director at Scott Logic, a UK-based software consultancy where they create complex application for their financial services clients. He is an avid technology enthusiast, spending his evenings contributing to open source projects, writing blog posts and learning as much as he can.
User & Device Identity for Microservices @ Netflix ScaleC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2S9tOgy.
Satyajit Thadeshwar provides useful insights on how Netflix implemented a secure, token-agnostic, identity solution that works with services operating at a massive scale. He shares some of the lessons learned from this process, both from architectural diagrams and code. Filmed at qconsf.com.
Satyajit Thadeshwar is an engineer on the Product Edge Access Services team at Netflix, where he works on some of the most critical services focusing on user and device authentication. He has more than a decade of experience building fault-tolerant and highly scalable, distributed systems.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2Ezs08q.
Justin Ryan talks about Netflix’ scalability issues and some of the ways they addressed it. He shares successes they’ve had from unintuitively partitioning computation into multiple services to get better runtime characteristics. He introduces us to useful probabilistic data structures, innovative bi-directional data passing, open-source projects available from Netflix that make this all possible. Filmed at qconsf.com.
Justin Ryan is Playback Edge Engineering at Netflix. He works on some of the most critical services at Netflix, specifically focusing on user and device authentication. Years of building developer tools has also given him a healthy set of opinions on developer productivity.
Make Your Electron App Feel at Home EverywhereC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2Z4ZJjn.
Kilian Valkhof discusses the process of making an Electron app feel at home on all three platforms: Windows, MacOS and Linux, making devs aware of the pitfalls and how to avoid them. Filmed at qconsf.com.
Kilian Valkhof is a Front-end Developer & User-experience Designer at Firstversionist. He writes about various topics, from design to machine learning, on his personal website, kilianvalkhof.com and is a frequent contributer to open source software. He is part of the Electron governance team that oversees the development of the Electron framework.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/344PnB1.
Steve Klabnik goes over the deep details of how async/await works in Rust, covering concepts like coroutines, generators, stack-less vs stack-ful, "pinning", and more. Filmed at qconsf.com.
Steve Klabnik is on the core team of Rust, leads the documentation team, and is an author of "The Rust Programming Language." He is a frequent speaker at conferences and is a prolific open source contributor, previously working on projects such as Ruby and Ruby on Rails.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2OUz6dt.
Chris Riccomini talks about the current state-of-the-art in data pipelines and data warehousing, and shares some of the solutions to current problems dealing with data streaming and warehousing. Filmed at qconsf.com.
Chris Riccomini works as a Software Engineer at WePay.
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2rm4hFD.
Yevgeniy Brikman talks about how to write automated tests for infrastructure code, including the code written for use with tools such as Terraform, Docker, Packer, and Kubernetes. Topics covered include: unit tests, integration tests, end-to-end tests, dependency injection, test parallelism, retries and error handling, static analysis, property testing and CI / CD for infrastructure code. Filmed at qconsf.com.
Yevgeniy Brikman is the co-founder of Gruntwork, a company that provides DevOps as a Service. He is the author of two books published by O'Reilly Media: Hello, Startup and Terraform: Up & Running. Previously, he worked as a software engineer at LinkedIn, TripAdvisor, Cisco Systems, and Thomson Financial.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
GridMate - End to end testing is a critical piece to ensure quality and avoid...ThomasParaiso2
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
1. Software Design for Persistent
Memory Systems
Howard Chu
CTO, Symas Corp. hyc@symas.com
2018-03-07
2. InfoQ.com: News & Community Site
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
nvram-systems
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com
4. 2
Personal Intro
● Howard Chu
– Founder and CTO Symas Corp.
– Developing Free/Open Source software since
1980s
● GNU compiler toolchain, e.g. "gmake -j", etc.
● Many other projects...
● I never use a software package without contributing to it
– Worked for NASA/JPL, wrote software for Space
Shuttle, etc.
5. 3
Personal Intro
● Career Highlights
– 2011- Author of LMDB, world's smallest, fastest, and most
reliable embedded database engine
– 1998- Main developer of OpenLDAP, world's most
scalable distributed data store
– 1995 Author of PC-Enterprise/Mac, world's fastest
AppleTalk stack and Appleshare file server
– 1993 Author of faster-than-realtime speech recognition
using Motorola 68030
– 1991 Inventor of parallel make support in GNU make
6. 4
Topics
● What is Persistent Memory?
● What system-level support exists?
● How do we leverage this in applications?
7. 5
What is Persistent Memory
● Non-volatile, doesn't lose contents when system
is powered off
● Can be thought of as battery-backed DRAM
– billed as byte-addressable storage, but really is still
constrained to cacheline granularity
– being used as a new layer in system memory
hierarchy, between regular DRAM and secondary
storage (SSD, HDD)
– ideally, will replace regular DRAM completely
9. 7
What is Persistent Memory
● STT-MRAM is the leading technology for now
– performance equivalent to DRAM
– endurance approaching DRAM (10^12 vs 10^15 writes)
– ST-DDR3, ST-DDR4 DIMMs available - drop-in compatible
with DDR3/DDR4
– Still lags in density, 256Mbit parts reaching market now
● Fabricated on 40nm process
● Compared to 8Gbit DDR4 DRAM chips already mainstream, on
10nm process
– Production on 22nm process expected later this year
10. 8
What is Persistent Memory
● Other possibilities exist
– actual battery-backed DRAM DIMMs (BBU DIMM)
● offered up to 72 hours of persistence
● deprecated, no longer marketed
– Flash-backed DRAM DIMMs (NVDIMM)
● typically with a super-capacitor onboard
● copies DRAM to flash on system shutdown
● All of these are more expensive than regular
DRAM
11. 9
System-Level Support
● Requires both BIOS and OS support
– POST must use non-destructive memory test, or
just skip memory test
– Kernel must recognize NV memory
– Linux kernel boot args can be used to explicitly
mark memory as persistent
– Current state of OS support is extremely primitive
12. 10
System-Level Support
● Kernel treats persistent memory as a block device
– you can create a filesystem on top and use it as a glorified
RAMdisk
● Congratulations, welcome to the state of the art of 1986.
– you can use it as cache dedicated to a particular set of
devices
● using dm-cache, bcache, flashcache, etc.
● but these solutions are written for Flash SSDs, and aren't optimal
for persistent RAM
– current designs assume only a small subset of system
memory is persistent
13. 11
System-Level Support
● Future support must account for systems with
100% persistent memory
– Kernel page cache manager must be modified to
utilize hot cache contents left by previous bootup
– "persistent memory" must become just "memory" -
used for system-wide device caching, instead of
isolated in its own block device
14. 12
System-Level Support
● Whether system is 100% persistent RAM or not,
memory should be managed by kernel and not require
direct management at user level
– current usage as distinct block device requires a user to
manually manage it
● explicitly copy files to it
● when the space gets full the user must choose some files to
delete, in order to make room for new files
– instead, used as part of the system cache, the OS can
page data in and out as needed, without any user
intervention
15. 13
Application Design
● Mindset
● Design Concepts
● Implementation Choices
● Other Details
– Concurrency Control
– Free Space Management
– Byte Addressability
● Endgame
16. 14
Application Design
● Requires a different mindset
– Should not view "memory" and "storage" as distinct
concepts - must adopt "single-level store"
● Storage and RAM are interchangeable, via memory-
mapping
– Data structures that are intended to be persistent
must be written atomically - interruption of updates
must not leave corrupt or inconsistent states
– Avoid temptation to take "memory-only" / "main
memory" design approach
17. 15
Application Design
● Problems with "main memory" approach
– A law of computing: data always grows to exceed
the size of available space
– There will always be larger/slower/cheaper memory
in addition to fast in-core memory: there will always
be a hierarchy of storage
– You must design for growth, and take this hierarchy
into account
18. 16
Design Concepts
● Essentially, persistent data structures must
provide ACID transaction semantics
– persistent RAM gives Durability, implicitly
– the rest is up to you
● Atomicity can be actual, or effective
– Actual: you only support modifications that can be
performed with a single atomic update
– Effective: you use undo/redo logs to allow recovery
from interrupted updates
19. 17
Design Concepts
● If you go for "effective atomicity" you'll need to have
complex locking mechanisms to protect intermediate
update states
● Once you go down the path of complex locking, you
also have to deal with deadlocks, backoffs, and retries
● All of this involves a great deal of additional code on
top of the actual data structure code
● Complex locking will not scale well across multiple
CPU sockets
20. 18
Design Concepts
● If you use undo/redo logs you'll need to build a robust
crash detection mechanism, as well as a crash recovery
procedure to recover from incomplete transactions
● The undo log will also be needed to execute transaction
abort/rollback in normal (non-crashed) operation
● The log will be a central bottleneck in all write
operations
● Logs will need explicit management - pruning/etc
21. 19
Design Concepts
● Better approach is to use MVCC (Multi-Version
Concurrency Control) with a single pointer to
the current version
– Once a new version has been constructed, a single
atomic write to the version pointer can be used to
make it visible
– Since each transaction operates on its own version
of the data structure, transactions have perfect
Isolation
22. 20
Design Concepts
● Best solution, based on constraints so far:
– data structure must be storage oriented, for growth - not a
memory-only structure
– data structure must have atomic update visibility
● Use a B+tree
– inherently suited to caching, memory hierarchy
– using Copy-on-Write, can expose a new modification simply
by updating a pointer to the root of a new tree version
● a new update can be simply aborted/rolled back just by omitting
the pointer update, no undo/redo logs needed
23. 21
Implementation
● Successful implementation requires explicit
control over memory layout of data structures
– structures must be CPU cacheline aligned, both for
performance and for integrity
– this precludes implementing in most higher level
languages
24. 22
Implementation
● We're now clearly talking about a storage library
– there's a lot of details to manage, but they can be
hidden in a library
– written in a low level language
– should use something like C
● easily callable from any other language
● mature, portable, flexible
● direct control over memory layout
– allows identical layout for "in-memory" and "on-disk" representation
25. 23
More Design Choices
● Multi-process concurrency, or just multi-thread?
– Multi-thread in a single process is simpler
● doesn't require shared memory for interprocess coordination
– Multi-process concurrency is more flexible
● allows administrative tools to query and operate regardless of whether
the main application is running
● Single-writer or multiple writer?
– Single-writer is simpler, eliminates possibility of deadlocks
– Multi-writer requires complex locking, conflict detection
● and still boils down to single-writer anyway, given the requirement of
atomic visibility
26. 24
Implementation
● Use mmap to expose data to callers
– Use a read-only mmap, otherwise random
overwrites will be persisted, causing unrecoverable
corruption
– Pointers to data in map can be returned directly to
callers on data fetch requests, thus avoiding
expensive malloc/copy operations
● This requires that data values are always stored
contiguously, even if values are larger than B+tree page
size
27. 25
Implementation
● Can optionally use writable mmap
– Opens a window to corruption vulnerability
– Requires explicit cache flush instructions, to ensure
writes are pushed from CPU cache out to RAM (if not
using msync)
– No performance benefit over readonly mmap
● writing a page requires that it first get faulted in, wasted effort if
the entire page is going to be overwritten
– May not be worth the cost in reliability and portability
● forcing a CPU cache flush is highly system-dependent
28. 26
Concurrency Control
● Systems commonly offer reader/writer semantics
– 1 writer can operate exclusively, or arbitrary number
of readers
– writer and readers cannot operate simultaneously
● Done properly, an MVCC-based design allows
readers to run wait-free, taking no locks
– writer should be able to operate concurrently with
arbitrary number of readers
29. 27
Free Space Management
● With MVCC, storage space rapidly fills up with
old/obsolete versions of data
● Most applications will have no use for the old
versions
● Reclaiming space from obsolete versions will
be critical for long term usability
● "Background" garbage collection (GC) is a
commonly practiced approach but is not viable
30. 28
Free Space Management
● Background GC assumes there's always spare CPU and
I/O capacity
– GC can consume more CPU and I/O bandwidth than the actual
user workload
● which then leads to requiring complex runtime profiling and throttling
implementations
– Thus it will either require over-provisioning of system resources,
or GC will always cause user-visible pauses in processing
● Better to track page usage in foreground and reuse old
pages when they become available
– Yields consistent write throughput without any pauses
31. 29
Free Space Management
● Tracking page availability has a direct impact on
concurrency
– Must record which readers are referencing which old versions,
to know which old versions can be purged/reclaimed
– Could just use a simple counter, recording the oldest version
still in use
● but accessing the counter becomes a bottleneck for readers
– Better to use an array with one slot per reader
● array slots must be cacheline aligned
● slots can be updated by readers and checked by writers without
taking any locks
32. 30
Byte Addressability
● Highly touted feature of NVRAM-based storage
● Largely a red herring
– Can be useful for current RAMdisk-style approaches,
but these are evolutionary dead ends
– Eventually the industry will wake up to the fact that
reinventing reset-survivable RAMdisks was a waste of
time and money
– NVRAM will eventually be integral to the system cache,
and the system cache is necessarily page-based
33. 31
Endgame
● Based on the given design constraints:
– atomicity, persistence, robustness, simplicity,
efficiency
– single-level store, blurring the line between memory
and storage
● You'll end up with something that looks a lot like
LMDB
34. 32
LMDB Overview
● LMDB "Lightning Memory-Mapped Database"
● embedded key/value store implemented with a
B+tree
● as the name indicates, it uses memory mapped
data
– defaults to read-only mmap
– zero-copy reads: retrieved data points directly into
mmap
– zero-copy writes: optionally supports writable mmap
35. 33
LMDB Overview
● full ACID transaction semantics
● MVCC concurrency control
– writers don't block readers, readers don't block
writers
– a pair of page pointers are used to point to the
current tree version
● single writer
– no need for callers to handle deadlocks or retries
36. 34
LMDB Overview
● No undo/redo logs
– Uses Copy-on-Write
– Intermediate tree states are never visible, cannot be corrupted
by system crashes
● No garbage collection
– space freed by a transaction is recorded in a 2nd B+tree living
in the same space
– writers reuse whatever available free space as needed
● No tuning or administrative overhead
– zero-config
37. 35
LMDB Overview
● Unrivalled read performance on any hardware and any data volume
– 1 billion record DB, ~120GB, on HP DL585 G5 with 128GB RAM, 16 cores
– 16 read threads concurrent with 1 write thread
38. 36
Summary
● Persistent RAM is approaching price parity with
regular DRAM, will be more common soon
● Current OS support is primitive and needs further
improvement
● If you enjoy low level programming, the design
constraints of writing an always-consistent data
structure may be interesting to explore
● Otherwise, just use LMDB and don't worry about it