The document discusses Java memory allocation profiling using the Aprof tool. It explains that Aprof works by instrumenting bytecode to inject calls that count and track object allocations. This allows it to provide insights on where memory is being allocated and identify potential performance bottlenecks related to garbage collection.
Join operations in Apache Spark is often the biggest source of performance problems and even full-blown exceptions in Spark. After this talk, you will understand the two most basic methods Spark employs for joining DataFrames – to the level of detail of how Spark distributes the data within the cluster. You’ll also find out how to work out common errors and even handle the trickiest corner cases we’ve encountered! After this talk, you should be able to write performance joins in Spark SQL that scale and are zippy fast!
This session will cover different ways of joining tables in Apache Spark.
Speaker: Vida Ha
This talk was originally presented at Spark Summit East 2017.
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudNoritaka Sekiyama
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud (Hadoop / Spark Conference Japan 2019)
# English version #
http://hadoop.apache.jp/hcj2019-program/
Bucketing 2.0: Improve Spark SQL Performance by Removing ShuffleDatabricks
Bucketing is commonly used in Hive and Spark SQL to improve performance by eliminating Shuffle in Join or group-by-aggregate scenario. This is ideal for a variety of write-once and read-many datasets at Bytedance.
Optimizing Delta/Parquet Data Lakes for Apache SparkDatabricks
This talk outlines data lake design patterns that can yield massive performance gains for all downstream consumers. We will talk about how to optimize Parquet data lakes and the awesome additional features provided by Databricks Delta. * Optimal file sizes in a data lake * File compaction to fix the small file problem * Why Spark hates globbing S3 files * Partitioning data lakes with partitionBy * Parquet predicate pushdown filtering * Limitations of Parquet data lakes (files aren't mutable!) * Mutating Delta lakes * Data skipping with Delta ZORDER indexes
Speaker: Matthew Powers
Accelerating Apache Spark Shuffle for Data Analytics on the Cloud with Remote...Databricks
The increasing challenge to serve ever-growing data driven by AI and analytics workloads makes disaggregated storage and compute more attractive as it enables companies to scale their storage and compute capacity independently to match data & compute growth rate. Cloud based big data services is gaining momentum as it provides simplified management, elasticity, and pay-as-you-go model.
Dive deep into some of the key innovations behind Amazon Aurora, discuss best practices and configurations, and share early customer experience from the field.
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Databricks
Spark SQL is a highly scalable and efficient relational processing engine with ease-to-use APIs and mid-query fault tolerance. It is a core module of Apache Spark. Spark SQL can process, integrate and analyze the data from diverse data sources (e.g., Hive, Cassandra, Kafka and Oracle) and file formats (e.g., Parquet, ORC, CSV, and JSON). This talk will dive into the technical details of SparkSQL spanning the entire lifecycle of a query execution. The audience will get a deeper understanding of Spark SQL and understand how to tune Spark SQL performance.
Join operations in Apache Spark is often the biggest source of performance problems and even full-blown exceptions in Spark. After this talk, you will understand the two most basic methods Spark employs for joining DataFrames – to the level of detail of how Spark distributes the data within the cluster. You’ll also find out how to work out common errors and even handle the trickiest corner cases we’ve encountered! After this talk, you should be able to write performance joins in Spark SQL that scale and are zippy fast!
This session will cover different ways of joining tables in Apache Spark.
Speaker: Vida Ha
This talk was originally presented at Spark Summit East 2017.
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudNoritaka Sekiyama
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud (Hadoop / Spark Conference Japan 2019)
# English version #
http://hadoop.apache.jp/hcj2019-program/
Bucketing 2.0: Improve Spark SQL Performance by Removing ShuffleDatabricks
Bucketing is commonly used in Hive and Spark SQL to improve performance by eliminating Shuffle in Join or group-by-aggregate scenario. This is ideal for a variety of write-once and read-many datasets at Bytedance.
Optimizing Delta/Parquet Data Lakes for Apache SparkDatabricks
This talk outlines data lake design patterns that can yield massive performance gains for all downstream consumers. We will talk about how to optimize Parquet data lakes and the awesome additional features provided by Databricks Delta. * Optimal file sizes in a data lake * File compaction to fix the small file problem * Why Spark hates globbing S3 files * Partitioning data lakes with partitionBy * Parquet predicate pushdown filtering * Limitations of Parquet data lakes (files aren't mutable!) * Mutating Delta lakes * Data skipping with Delta ZORDER indexes
Speaker: Matthew Powers
Accelerating Apache Spark Shuffle for Data Analytics on the Cloud with Remote...Databricks
The increasing challenge to serve ever-growing data driven by AI and analytics workloads makes disaggregated storage and compute more attractive as it enables companies to scale their storage and compute capacity independently to match data & compute growth rate. Cloud based big data services is gaining momentum as it provides simplified management, elasticity, and pay-as-you-go model.
Dive deep into some of the key innovations behind Amazon Aurora, discuss best practices and configurations, and share early customer experience from the field.
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Databricks
Spark SQL is a highly scalable and efficient relational processing engine with ease-to-use APIs and mid-query fault tolerance. It is a core module of Apache Spark. Spark SQL can process, integrate and analyze the data from diverse data sources (e.g., Hive, Cassandra, Kafka and Oracle) and file formats (e.g., Parquet, ORC, CSV, and JSON). This talk will dive into the technical details of SparkSQL spanning the entire lifecycle of a query execution. The audience will get a deeper understanding of Spark SQL and understand how to tune Spark SQL performance.
Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...Spark Summit
What if you could get the simplicity, convenience, interoperability, and storage niceties of an old-fashioned CSV with the speed of a NoSQL database and the storage requirements of a gzipped file? Enter Parquet.
At The Weather Company, Parquet files are a quietly awesome and deeply integral part of our Spark-driven analytics workflow. Using Spark + Parquet, we’ve built a blazing fast, storage-efficient, query-efficient data lake and a suite of tools to accompany it.
We will give a technical overview of how Parquet works and how recent improvements from Tungsten enable SparkSQL to take advantage of this design to provide fast queries by overcoming two major bottlenecks of distributed analytics: communication costs (IO bound) and data decoding (CPU bound).
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin HuaiDatabricks
Catalyst is becoming one of the most important components of Apache Spark, as it underpins all the major new APIs in Spark 2.0 and later versions, from DataFrames and Datasets to Streaming. At its core, Catalyst is a general library for manipulating trees.
In this talk, Yin explores a modular compiler frontend for Spark based on this library that includes a query analyzer, optimizer, and an execution planner. Yin offers a deep dive into Spark SQL’s Catalyst optimizer, introducing the core concepts of Catalyst and demonstrating how developers can extend it. You’ll leave with a deeper understanding of how Spark analyzes, optimizes, and plans a user’s query.
Optimizing Performance in Rust for Low-Latency Database DriversScyllaDB
The process of optimizing shard-aware drivers for ScyllaDB has involved several initiatives, often necessitating a complete rewrite from the ground up. Discover the efforts put into enhancing the performance of ScyllaDB drivers with a focus on Rust, and how its code base will serve as a foundation for drivers using other language bindings in the future. This session emphasizes the performance gains achieved by harnessing the power of the asynchronous Tokio framework as the backbone of a new, high-performance driver while thoughtfully architecting and optimizing various components of the driver.
Using Spark to Load Oracle Data into CassandraJim Hatcher
This presentation describes how you can use Spark as an ETL tool to get data from a relational database into Cassandra. I go through the concept in general and then talk about some specific issues you might run into and how to fix them.
Threading Made Easy! A Busy Developer’s Guide to Kotlin CoroutinesLauren Yew
Kotlin Coroutines is a powerful threading library for Kotlin, released by JetBrains in 2018. At The New York Times, we recently migrated our core libraries and parts of our News app from RxJava to Kotlin Coroutines. In this talk we’ll share lessons learned and best practices to understand, migrate to, and use Kotlin Coroutines & Flows.
In this presentation, you will learn:
What Coroutines are and how they function
How to use Kotlin Coroutines & Flows (with real world examples and demos)
Where and why you should use Coroutines & Flows in your app
How to avoid the pitfalls of Coroutines
Kotlin Coroutines vs. RxJava
Lessons learned from migrating to Kotlin Coroutines from RxJava in large legacy projects & libraries
By the end of this talk, you will be able to apply Kotlin Coroutines to your own app, run the provided sample code yourself, and convince your team to give Kotlin Coroutines a try!
Whoops, The Numbers Are Wrong! Scaling Data Quality @ NetflixDataWorks Summit
Netflix is a famously data-driven company. Data is used to make informed decisions on everything from content acquisition to content delivery, and everything in-between. As with any data-driven company, it’s critical that data used by the business is accurate. Or, at worst, that the business has visibility into potential quality issues as soon as they arise. But even in the most mature data warehouses, data quality can be hard. How can we ensure high quality in a cloud-based, internet-scale, modern big data warehouse employing a variety of data engineering technologies?
In this talk, Michelle Ufford will share how the Data Engineering & Analytics team at Netflix is doing exactly that. We’ll kick things off with a quick overview of Netflix’s analytics environment, then dig into details of our data quality solution. We’ll cover what worked, what didn’t work so well, and what we plan to work on next. We’ll conclude with some tips and lessons learned for ensuring data quality on big data.
Optimizing spark jobs through a true understanding of spark core. Learn: What is a partition? What is the difference between read/shuffle/write partitions? How to increase parallelism and decrease output files? Where does shuffle data go between stages? What is the "right" size for your spark partitions and files? Why does a job slow down with only a few tasks left and never finish? Why doesn't adding nodes decrease my compute time?
Using Apache Spark to analyze large datasets in the cloud presents a range of challenges. Different stages of your pipeline may be constrained by CPU, memory, disk and/or network IO. But what if all those stages have to run on the same cluster? In the cloud, you have limited control over the hardware your cluster runs on.
You may have even less control over the size and format of your raw input files. Performance tuning is an iterative and experimental process. It’s frustrating with very large datasets: what worked great with 30 billion rows may not work at all with 400 billion rows. But with strategic optimizations and compromises, 50+ TiB datasets can be no big deal.
By using Spark UI and simple metrics, explore how to diagnose and remedy issues on jobs:
Sizing the cluster based on your dataset (shuffle partitions)
Ingestion challenges – well begun is half done (globbing S3, small files)
Managing memory (sorting GC – when to go parallel, when to go G1, when offheap can help you)
Shuffle (give a little to get a lot – configs for better out of box shuffle) – Spill (partitioning for the win)
Scheduling (FAIR vs FIFO, is there a difference for your pipeline?)
Caching and persistence (it’s the cost of doing business, so what are your options?)
Fault tolerance (blacklisting, speculation, task reaping)
Making the best of a bad deal (skew joins, windowing, UDFs, very large query plans)
Writing to S3 (dealing with write partitions, HDFS and s3DistCp vs writing directly to S3)
Amazon DynamoDB is a fully managed NoSQL database service for applications that need consistent, single-digit millisecond latency at any scale. This talk explores DynamoDB capabilities and benefits in detail and discusses how to get the most out of your DynamoDB database. We go over schema design best practices with DynamoDB across multiple use cases, including gaming, AdTech, IoT, and others. We also explore designing efficient indexes, scanning, and querying, and go into detail on a number of recently released features, including JSON document support, Streams, and more.
Frame - Feature Management for Productive Machine LearningDavid Stein
Presented at the ML Platforms Meetup at Pinterest HQ in San Francisco on August 16, 2018.
Abstract: At LinkedIn we observed that much of the complexity in our machine learning applications was in their feature preparation workflows. To address this problem, we built Frame, a shared virtual feature store that provides a unified abstraction layer for accessing features by name. Frame removes the need for feature consumers to deal directly with underlying data sources, which are often different across computing environments. By simplifying feature preparation, Frame has made ML applications at LinkedIn easier to build, modify, and understand.
Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...Spark Summit
What if you could get the simplicity, convenience, interoperability, and storage niceties of an old-fashioned CSV with the speed of a NoSQL database and the storage requirements of a gzipped file? Enter Parquet.
At The Weather Company, Parquet files are a quietly awesome and deeply integral part of our Spark-driven analytics workflow. Using Spark + Parquet, we’ve built a blazing fast, storage-efficient, query-efficient data lake and a suite of tools to accompany it.
We will give a technical overview of how Parquet works and how recent improvements from Tungsten enable SparkSQL to take advantage of this design to provide fast queries by overcoming two major bottlenecks of distributed analytics: communication costs (IO bound) and data decoding (CPU bound).
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin HuaiDatabricks
Catalyst is becoming one of the most important components of Apache Spark, as it underpins all the major new APIs in Spark 2.0 and later versions, from DataFrames and Datasets to Streaming. At its core, Catalyst is a general library for manipulating trees.
In this talk, Yin explores a modular compiler frontend for Spark based on this library that includes a query analyzer, optimizer, and an execution planner. Yin offers a deep dive into Spark SQL’s Catalyst optimizer, introducing the core concepts of Catalyst and demonstrating how developers can extend it. You’ll leave with a deeper understanding of how Spark analyzes, optimizes, and plans a user’s query.
Optimizing Performance in Rust for Low-Latency Database DriversScyllaDB
The process of optimizing shard-aware drivers for ScyllaDB has involved several initiatives, often necessitating a complete rewrite from the ground up. Discover the efforts put into enhancing the performance of ScyllaDB drivers with a focus on Rust, and how its code base will serve as a foundation for drivers using other language bindings in the future. This session emphasizes the performance gains achieved by harnessing the power of the asynchronous Tokio framework as the backbone of a new, high-performance driver while thoughtfully architecting and optimizing various components of the driver.
Using Spark to Load Oracle Data into CassandraJim Hatcher
This presentation describes how you can use Spark as an ETL tool to get data from a relational database into Cassandra. I go through the concept in general and then talk about some specific issues you might run into and how to fix them.
Threading Made Easy! A Busy Developer’s Guide to Kotlin CoroutinesLauren Yew
Kotlin Coroutines is a powerful threading library for Kotlin, released by JetBrains in 2018. At The New York Times, we recently migrated our core libraries and parts of our News app from RxJava to Kotlin Coroutines. In this talk we’ll share lessons learned and best practices to understand, migrate to, and use Kotlin Coroutines & Flows.
In this presentation, you will learn:
What Coroutines are and how they function
How to use Kotlin Coroutines & Flows (with real world examples and demos)
Where and why you should use Coroutines & Flows in your app
How to avoid the pitfalls of Coroutines
Kotlin Coroutines vs. RxJava
Lessons learned from migrating to Kotlin Coroutines from RxJava in large legacy projects & libraries
By the end of this talk, you will be able to apply Kotlin Coroutines to your own app, run the provided sample code yourself, and convince your team to give Kotlin Coroutines a try!
Whoops, The Numbers Are Wrong! Scaling Data Quality @ NetflixDataWorks Summit
Netflix is a famously data-driven company. Data is used to make informed decisions on everything from content acquisition to content delivery, and everything in-between. As with any data-driven company, it’s critical that data used by the business is accurate. Or, at worst, that the business has visibility into potential quality issues as soon as they arise. But even in the most mature data warehouses, data quality can be hard. How can we ensure high quality in a cloud-based, internet-scale, modern big data warehouse employing a variety of data engineering technologies?
In this talk, Michelle Ufford will share how the Data Engineering & Analytics team at Netflix is doing exactly that. We’ll kick things off with a quick overview of Netflix’s analytics environment, then dig into details of our data quality solution. We’ll cover what worked, what didn’t work so well, and what we plan to work on next. We’ll conclude with some tips and lessons learned for ensuring data quality on big data.
Optimizing spark jobs through a true understanding of spark core. Learn: What is a partition? What is the difference between read/shuffle/write partitions? How to increase parallelism and decrease output files? Where does shuffle data go between stages? What is the "right" size for your spark partitions and files? Why does a job slow down with only a few tasks left and never finish? Why doesn't adding nodes decrease my compute time?
Using Apache Spark to analyze large datasets in the cloud presents a range of challenges. Different stages of your pipeline may be constrained by CPU, memory, disk and/or network IO. But what if all those stages have to run on the same cluster? In the cloud, you have limited control over the hardware your cluster runs on.
You may have even less control over the size and format of your raw input files. Performance tuning is an iterative and experimental process. It’s frustrating with very large datasets: what worked great with 30 billion rows may not work at all with 400 billion rows. But with strategic optimizations and compromises, 50+ TiB datasets can be no big deal.
By using Spark UI and simple metrics, explore how to diagnose and remedy issues on jobs:
Sizing the cluster based on your dataset (shuffle partitions)
Ingestion challenges – well begun is half done (globbing S3, small files)
Managing memory (sorting GC – when to go parallel, when to go G1, when offheap can help you)
Shuffle (give a little to get a lot – configs for better out of box shuffle) – Spill (partitioning for the win)
Scheduling (FAIR vs FIFO, is there a difference for your pipeline?)
Caching and persistence (it’s the cost of doing business, so what are your options?)
Fault tolerance (blacklisting, speculation, task reaping)
Making the best of a bad deal (skew joins, windowing, UDFs, very large query plans)
Writing to S3 (dealing with write partitions, HDFS and s3DistCp vs writing directly to S3)
Amazon DynamoDB is a fully managed NoSQL database service for applications that need consistent, single-digit millisecond latency at any scale. This talk explores DynamoDB capabilities and benefits in detail and discusses how to get the most out of your DynamoDB database. We go over schema design best practices with DynamoDB across multiple use cases, including gaming, AdTech, IoT, and others. We also explore designing efficient indexes, scanning, and querying, and go into detail on a number of recently released features, including JSON document support, Streams, and more.
Frame - Feature Management for Productive Machine LearningDavid Stein
Presented at the ML Platforms Meetup at Pinterest HQ in San Francisco on August 16, 2018.
Abstract: At LinkedIn we observed that much of the complexity in our machine learning applications was in their feature preparation workflows. To address this problem, we built Frame, a shared virtual feature store that provides a unified abstraction layer for accessing features by name. Frame removes the need for feature consumers to deal directly with underlying data sources, which are often different across computing environments. By simplifying feature preparation, Frame has made ML applications at LinkedIn easier to build, modify, and understand.
ZeroSpace is a part of Woxa Technology where industrial trainer are ready to trained you.Take a demo class and choose your future in the programming field. We offer to learn in latest technology with live project which is demanding.
feel free to contact us-: 8471003400
How do we go from your Java code to the CPU assembly that actually runs it? Using high level constructs has made us forget what happens behind the scenes, which is however key to write efficient code.
Starting from a few lines of Java, we explore the different layers that constribute to running your code: JRE, byte code, structure of the OpenJDK virtual machine, HotSpot, intrinsic methds, benchmarking.
An introductory presentation to these low-level concerns, based on the practical use case of optimizing 6 lines of code, so that hopefully you to want to explore further!
Presentation given at the Toulouse (FR) Java User Group.
Video (in french) at https://www.youtube.com/watch?v=rB0ElXf05nU
Slideshow with animations at https://docs.google.com/presentation/d/1eIcROfLpdTU2_Z_IKiMG-AwqZGZgbN1Bs2E0nGShpbk/pub?start=true&loop=false&delayms=60000
Woxa Technologies have great industrial exoerts in java field they work on the live projects with students they are not teacher they are industrial trainer.
for more information 8471003400
Inside the JVM - Follow the white rabbit! / Breizh JUGSylvain Wallez
Presentation given at the Rennes (FR) Java User Group in Feb 2019.
How do we go from your Java code to the CPU assembly that actually runs it? Using high level constructs has made us forget what happens behind the scenes, which is however key to write efficient code.
Starting from a few lines of Java, we explore the different layers that constribute to running your code: JRE, byte code, structure of the OpenJDK virtual machine, HotSpot, intrinsic methds, benchmarking.
An introductory presentation to these low-level concerns, based on the practical use case of optimizing 6 lines of code, so that hopefully you to want to explore further!
Compromising Linux Virtual Machines with Debugging MechanismsRussell Sanford
This presentation covers utilizing VMwares (GDB) debugging protocol to invasive inject commands into a Linux-x64 target. Automatic detection of kernel API is performed to locate _vmalloc & call_usermodehelper* functions across all 3x and 4x kernels.
http://imatge-upc.github.io/telecombcn-2016-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak PROIDEA
Speaker: Andrzej Dyjak
Language: English
In recent years security industry started to grow fond of Apple’s iOS and OS X platforms. This talk will cover one of XNU's flagship debugging utilities: DTrace, a dynamic tracing framework for troubleshooting kernel and application problems on production systems in real time. It will be shown how it can be used in order to ease various tasks within the realm of dynamic binary analysis and beyond.
CONFidence: http://confidence.org.pl/
Quantifying the Performance of Garbage Collection vs. Explicit Memory ManagementEmery Berger
This talk answers an age-old question: is garbage collection faster/slower/the same speed as malloc/free? We introduce oracular memory management, an approach that lets us measure unaltered Java programs as if they used malloc and free. The result: a good GC can match the performance of a good allocator, but it takes 5X more space. If physical memory is tight, however, conventional garbage collectors suffer an order-of-magnitude performance penalty.
Deep Learning in Spark with BigDL by Petar Zecevic at Big Data Spain 2017Big Data Spain
BigDL is a deep learning framework modeled after Torch and open-sourced by Intel in 2016. BigDL runs on Apache Spark, a fast, general, distributed computing platform that is widely used for Big Data processing and machine learning tasks.
https://www.bigdataspain.org/2017/talk/deep-learning-in-spark-with-bigdl
Big Data Spain 2017
16th -17th November Kinépolis Madrid
Lock-free algorithms for Kotlin CoroutinesRoman Elizarov
Presentation for SPTCC 2017 - http://neerc.ifmo.ru/sptcc/
Video (part 1): https://www.youtube.com/watch?v=W2dOOBN1OQI
Video (part 2): https://www.youtube.com/watch?v=iQsN_IDUTSc
This is a detailed review of ACM International Collegiate Programming Contest (ICPC) Northeastern European Regional Contest (NEERC) 2016 Problems. It includes a summary of problem and names of problem authors and detailed runs statistics for each problem.
Problem statements are available here:
http://neerc.ifmo.ru/information/problems.pdf
The beginning of the following video has the actual review (in Russian): https://www.youtube.com/watch?v=fN25KkNYsjA
Многопоточное Программирование - Теория и ПрактикаRoman Elizarov
Многоядерные процессоры используются во всех серверах, рабочих станциях и мобильных устройствах. Написание многопоточных программ необходимо для обеспечения вертикальной масштабируемости, но, в отличие от однопоточных программ, их намного сложней отладить и протестировать, чтобы убедиться в корректности. Важно понимать какие именно гарантии дают те или иные конструкции языка и библиотеки при их многопоточном исполнении и какие подводные камни могут нарушить корректность кода. Доклад будет содержать краткое введение в теорию многопоточного программирования. Мы рассмотрим теоретические модели, которые используются для описания поведения многопоточных программ. Будут рассмотрены понятия последовательной согласованности и линеаризуемости (с примерами) и объяснено зачем это все-нужно программисту-практику. Будет показано как эти понятия применяются в модели памяти Java с примерами кода приводящего к неожиданным результатам с точки зрения человека, который с ней не знаком.
Доклад сделан для конференции Java Point Student Day 2016.
Slides from tech talk about the art of non-blocking waiting in Java with LockSupport.park/unpark and AbstractQueuedSynchronizer. Presented on JPoint 2016 Conference.
This is a detailed review of ACM International Collegiate Programming Contest (ICPC) Northeastern European Regional Contest (NEERC) 2015 Problems. It includes a summary of problem and names of problem authors and detailed runs statistics for each problem. Video of the actual presentation that was recorded during NEERC is here https://www.youtube.com/watch?v=vn7v1MuWXdU (in Russian)
Note: there were only preliminary stats avaialble, because problems review was happening before before the closing ceremony. This published presentation has full stats.
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdfJay Das
With the advent of artificial intelligence or AI tools, project management processes are undergoing a transformative shift. By using tools like ChatGPT, and Bard organizations can empower their leaders and managers to plan, execute, and monitor projects more effectively.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/
A Comprehensive Look at Generative AI in Retail App Testing.pdfkalichargn70th171
Traditional software testing methods are being challenged in retail, where customer expectations and technological advancements continually shape the landscape. Enter generative AI—a transformative subset of artificial intelligence technologies poised to revolutionize software testing.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Into the Box Keynote Day 2: Unveiling amazing updates and announcements for modern CFML developers! Get ready for exciting releases and updates on Ortus tools and products. Stay tuned for cutting-edge innovations designed to boost your productivity.
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
1. Why GC is eating all my CPU?
Aprof - Java Memory Allocation Profiler
Roman Elizarov, Devexperts
Joker Conference, St. Petersburg, 2014
2. Java Memory Allocation Profiler
Why it is needed?
When to use it?
How it works?
How to use it?
DISCLAIMER Herein lots of:
omissions,
simplifications,
half-truths,
intentionally misleading information. [Almost] all the truth is at the end of presentation!
3. Java
•Does not have stack-allocation, does not have structs, does not have tuples…
•Promotes the “Object Oriented” style of programming with lots of object allocations
–Most of which are only temporary → garbage
10. …or use the logs and APIs
•Always use the following settings in prod: -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
– So, you can always figure out % time spent in GC by looking at your stdout as last resort.
•To figure it out programmatically, see java.lang.management.GarbageCollectorMXBean
Shameless commercial plug: That is how our MARS monitoring product does it
11. How much is much?
•10%+ time in GC – you start worrying
•30%+ time in GC – you do something about it
http://upload.wikimedia.org/wikipedia/commons/4/4f/Scheibenbremse-magura.jpg
12.
13. C/C++
Java
Use CPU Profiler
Find hot spots
Fix
Use CPU Profiler
Find hot spots
Wait! Allocation is FAST! (does not consume CPU)
Troubleshooting
15. Memory allocation profiling
•“-Xaprof” option in old JVM (<= 1.6)
–Prints something like this on process termination:
Allocation profile (sizes in bytes, cutoff = 0 bytes):
___________Size__Instances__Average__Class________________
555807584 34737974 16 java.lang.Integer
321112 5844 55 [I
106104 644 165 [C
37144 63 590 [B
13744 325 42 [Ljava.lang.Object;
… <the rest>
Top alloc’d
That is where aprof project got its name from
17. Where memory is allocated? (1)
•Java
–new CName(…)
–new <prim>[…]
–new CName[...]
–new CName[…][…]…[…]
•Bytecode
–new
–newarray
–anewarray
–multianewarray
Fundamental ways to allocate memory
Quiz for audience: What else?
Allocate memory
Since Java 1.0
18. Where memory is allocated? (2)
–Integer i = 1234;
–Integer i = Integer.valueOf(1234);
Boxing – syntactic sugar to allocate memory
Quiz for audience: What else?
Since Java 5
“Enhanced for loop” also desugars to method calls
19. Where memory is allocated? (3)
•java.lang.Object.clone
–Yet another true way of object allocation
•Reflection & deserialization
–Gets compiled into bytecode after a few invokes
but Array.newInstance needs separate care
•sun.misc.Unsafe.allocateInstance
–Could be tracked, but is not in current aprof
•JNI
–Can be tracked via native JVMTI
aprof is a pure Java tool now, does not track it
Quiz for audience: What else?
20. Where memory is allocated? (4)
Captured
In new lambda
Since Java 8
Capturing Java 8 Lambda Closure
21. Let’s instrument bytecodes
java –javaagent:aprof.jar …
JVM API to write bytecode instrumenting agents in Java
24. Manipulate bytecode with ASM
•ObjectWeb ASM is an open source lib to help
–Easy to use for bytecode manipulation
–Extremely fast (suited to on-the-fly manipulation)
ClassReader
ClassVisitor
ClassWriter
MethodVisitor
26. Generate calls to profiling methods
New array bytecode
Needs length to compute object size
Index of location
invoke static profiling method
27. Generate calls to profiling methods
Regular new object bytecode
Needs class to compute object size
28. Example transformation
public int[] newArray(int);
Code:
0: iload_1
1: newarray int
3: areturn
public int[] newArray(int); Code: 0: iload_1 1: dup 2: sipush 4137 5: invokestatic #31 8: newarray int 10: areturn
java –javaagent:aprof.jar=dump.classes.dir=dump …
Method com/devexperts/aprof/AProfOps. intAllocateArraySize
Assigned index
return new int[n];
29. We cannot measure the system without affecting it
Need to minimize measurement effect
Measure garbage without producing garbage (do not allocate memory)
30. Garbage-free code?
[ ] Class-transformation
Only during load (don’t care)
[ ] Object-class size computation
Only once per class (don’t care)
[x] Count individual allocations
It has to be garbage-free
and it is (once all allocation locations were visited)
31. Count all allocations
Uses array access, because index enumerates (location, type) pairs consecutively
33. Compute object size
Regular object size
… and cache the size for class
Precise and actual size (!)
34. Let’s try it
http://3.bp.blogspot.com/-vUgjzK5P-kQ/ToYOTLjqBTI/AAAAAAAAFu0/2ZGrGV9y0c4/s640/tumblr_komtamvGna1qzvjtno1_500.jpg
35. Big business app
Top allocated data types with locations
-------------------------------------------------------------
char[]: 330,657,880 bytes in 5,072,613 objects
java.util.Arrays.copyOfRange: 131,299,568 bytes in …
aprof.txt
Oops! That is not informative
We knew that it will produce a lot of char[] garbage in strings!
aprof agent
36. Need allocation context
Call stack:
•MyDataClass.getData(int)
•java.lang.StringBuilder.toString()
•Java.lang.String.String(char[], int, int)
•Java.util.Arrays.copyOfRange(char[], int, int[])
•new char[]
But it is allocated here
We want this location
38. How it is done?
•Tracked via thread-local instance of class com.devexperts.aprof.LocationStack
•Local variable is injected into
–methods that allocate memory
–methods that invoke tracked methods
–tracked methods themselves
•Local variable is initialized on first use
39. public int[] newArray(int);
Code:
0: aconst_null
1: astore_2
2: iload_1
3: dup
4: aload_2
5: dup
6: ifnonnull 15
9: pop
10: invokestatic #25
13: dup
14: astore_2
15: sipush 4137
18: invokestatic #31
21: newarray int
23: areturn
Example transformation (actual)
LocationStack stack = null; // local var
If (stack == null)
stack = LocationStack.get();
return new int[n];
40. Looks scary
… But fast in practice
–Long-running methods retrieve LocationStack from ThreadLocal only once
–HotSpot optimizes this ugly code quite well
–It only affects code that does memory allocations or uses tracked (memory allocating!) methods
–Does not allocate memory during stable operation
It has no performance impact on dense garbage-free computation code, various getters, etc
41. Big business app
Top allocated data types with reverse location traces
-----------------------------------------------------
char[]: 330,657,880 bytes in 5,072,613 objects
java.util.Arrays.copyOfRange: 131,299,568 bytes in …
java.lang.StringBuilder.toString: …
MyBusinessMethod: …
… (more!)
aprof.txt
aprof agent
Got you!
42. Actual call stack: …
•MyBusinessMethod
•java.lang.StringBuilder.toString()
•Java.lang.String.String(char[], int, int)
•Java.util.Arrays.copyOfRange(char[], int, int[])
•new char[]
Aprof dump vs actual call stack
Top allocated data types with reverse location traces ----------------------------------------------------- char[]: 330,657,880 bytes in 5,072,613 objects java.util.Arrays.copyOfRange: 131,299,568 bytes in … java.lang.StringBuilder.toString: … MyBusinessMethod: … … (more!)
At most 4 items are displayed in aprof dump
(1) Type that was allocated
(2) Where it was allocated
(1)
(2)
(3) Outermost tracked method
(4) Caller of the outermost tracked method
(3)
(4)
43. But what if
… caught MyFrameworkMethod allocating memory instead?
Add it to tracked methods list
java –javaagent:aprof.jar=track=MyFrameworkMethod …
java –javaagent:aprof.jar=track.file=details.config …
a)
b)
Repeat
44. Poor man’s catch-all
java –javaagent:aprof.jar=+unknown …
Injects the profiling code right into java.lang.Object constructor
•Does some mental accounting to exclude allocation that were already counted, reports the difference
•The difference can appear from JNI object allocations. But location is unknown anyway.
•Does not help with tracking JNI array allocations at all (no constructor invocation)
45.
46. Top allocated data types with reverse location traces
-----------------------------------------------------
java.util.ArrayList$Itr: 32,000,006,272 bytes …
java.util.ArrayList.iterator: 32,000,006,272 bytes … IterateALot.sumList: 32,000,000,000 bytes …
Iterator is allocated here
47. java -XX:+PrintGCDetails -XX:+PrintGCTimeStamps …
Shouldn’t GC work like hell?
[PSYoungGen: 512K->400K(1024K)] 512K->408K(126464K)
[PSYoungGen: 912K->288K(1024K)] 920K->296K(126464K)
[PSYoungGen: 800K->352K(1024K)] 808K->360K(126464K)
[PSYoungGen: 864K->336K(1536K)] 872K->344K(126976K)
[PSYoungGen: 1360K->352K(1536K)] 1368K->360K(126976K)
[PSYoungGen: 1376K->352K(2560K)] 1384K->360K(128000K)
[PSYoungGen: 2400K->0K(2560K)] 2408K->284K(128000K)
[PSYoungGen: 2048K->0K(4608K)] 2332K->284K(130048K)
And that is it! No more GC!
What is going on here?
49. Aprof allocation elimination checking
java –javaagent:aprof.jar:+check.eliminate.allocation -XX:+UnlockDiagnosticVMOptions -XX:+LogCompilation …
Top allocated data types with reverse location traces
-----------------------------------------------------
java.util.ArrayList$Itr: 32,000,006,272 bytes …
java.util.ArrayList.iterator: 32,000,006,272 bytes … IterateALot.sumList: … ; possibly eliminated
1.HotSpot writes hs_xxx.log files
2.Aprof parses them to learn eliminated allocation locations
50. Some additional options/features
•histogram – track array allocation separately for different size brackets
•file – configure dump file, use #### in file name to auto-number files
•file.append – append dump to file every, instead of overwriting
•time – time period to write dumps,
defaults to a minute
java –jar aprof.jar
To get help
51. Advanced topics (1)
•Transforming classes that were already loaded
–… before aprof Pre-Main had even got control
•Profiling the profiler
–aprof records and reports its own allocations
•Caching class meta-info for each class-loader
–for fast class transformation
•Aggregating classes with dynamic names like “sun.reflect.GeneratedConstructorAccessorX”
–to avoid memory leaks (out of memory)
52. Advanced topics (2)
•Handling HotSpot intrinsics
–Arrays.copyOf and Arrays.copyOfRange do not actually run their bytecode in HotSpot (intrinsic)
aprof has to treat them as first-class allocations
•Avoiding class-name clashes with target app
–asm and aprof transformer are actually loaded from a separate inner class loader
•Accessing index of counters via fast hash
–http://elizarov.livejournal.com/tag/hash
•Writing big reports without tons of garbage
54. Known issues
•Lambda capture memory allocations are not tracked nor reported in any way
–Need to track metafactory calls and unnamed classes it generates
•Java 8 library methods (collections, streams, etc) are not included into default list of tracked methods
–Need to work through them and include them
•JNI allocations are not properly tracked
–Need to write native JVMTI agent to track them
Where you can help
Mostly done in aprof v30
To get actual code location