Ian wcre2011
•Download as PPTX, PDF•
0 likes•121 views
This study examined how communicated information about the execution of large software systems evolves over time. The researchers analyzed log data from multiple releases of an enterprise application to identify six types of modifications to communicated information: rephrasing, adding information, deleting information, diverging, merging, and redundant information. They found that most modifications could have been avoided and that short-lived communicated information often contained low-level implementation details. The study concluded that communicated information grows and changes over releases, and documenting and tracking these evolution is important for maintaining log processing applications.
Report
Share
Report
Share
Recommended
Ian wcre2011
This study examines how communicated information (CI), such as log messages and system alerts, evolves over time in large software systems. The researchers analyzed CI from multiple releases of an enterprise application. They found that CI grows with each release, even when system size decreases. CI is also impacted by code refactoring and re-engineering. Six types of CI modifications were identified: rephrasing, adding information, deleting information, diverging, merging, and redundant information. Most modifications can be avoided. Short-lived CI often contains low-level implementation details that require maintenance effort over time.
RedisDay London 2018 - Graph Launch
The document describes the key features and capabilities of RedisGraph including:
- High performance and linear scalability for graph queries.
- Active-active geo distribution and high availability with instant failover.
- Durability at memory speeds with data stored in RAM and flash memory.
- Built-in high performance search capabilities.
- Flexible deployment options across cloud, on-premises and hybrid environments.
Icsm2011 syer
This document explores the development of micro-apps across different mobile platforms. It finds that Android apps are typically smaller in size than equivalent apps on BlackBerry, but Android apps also depend more on third-party libraries and the platform itself. This high platform dependency for Android apps can reduce development effort due to platform support but also increases maintenance effort and makes porting apps to other platforms more difficult. The document discusses how these findings for micro-apps may differ compared to desktop applications.
Career & Competence - exercise package
The document provides information and exercises for a career coaching group for international degree students. The VALOA project aims to increase employment opportunities for international students in Finland by helping employers recruit foreign graduates and helping universities support international students.
The document describes a career coaching group piloted at the University of Oulu to help international students understand the Finnish job market and improve their career planning. The group covered topics like career goals, skills assessment, networking, and cultural aspects of work.
Various exercises used in the coaching group are presented, related to topics like envisioning one's future career, skills assessment, personal networks, resume building, and practicing interviews. The exercises provide ways for students to reflect on work experiences, values, strengths,
Choosing a College Major
This document provides guidance on choosing a college major, noting that while a major must eventually be selected, schools differ on timing and some allow custom majors. It recommends considering interests, skills, goals, and discussing options with counselors when selecting a major, and notes majors and minors don't have to relate but can complement each other, and double majors don't necessarily extend time in college.
Career Coaching for Students
Career Coaching for Students is a program for teen career exploration. High school students enjoy highly validated and reliable assessments (which means it doesn't take an hour to complete), research strategies and decision making tools.
Career Coaching for Students™
Career Coaching for Students™ is a program for high school freshman through senior years. Focus is on self awareness, research strategies, planning and life skill development.
The college version is perfect for Freshman through Senior college students wanting to gain clarity around career options.
Job hunting in the 21st century for students and recent grads
This document provides guidance on effective job hunting strategies, especially in challenging economic times. It emphasizes the importance of networking, developing contacts, and taking personal responsibility for one's job search over more passive methods. Key recommendations include actively participating in industry groups, volunteering, engaging contacts through questions, maintaining a contact database, and following up consistently.
Recommended
Ian wcre2011
This study examines how communicated information (CI), such as log messages and system alerts, evolves over time in large software systems. The researchers analyzed CI from multiple releases of an enterprise application. They found that CI grows with each release, even when system size decreases. CI is also impacted by code refactoring and re-engineering. Six types of CI modifications were identified: rephrasing, adding information, deleting information, diverging, merging, and redundant information. Most modifications can be avoided. Short-lived CI often contains low-level implementation details that require maintenance effort over time.
RedisDay London 2018 - Graph Launch
The document describes the key features and capabilities of RedisGraph including:
- High performance and linear scalability for graph queries.
- Active-active geo distribution and high availability with instant failover.
- Durability at memory speeds with data stored in RAM and flash memory.
- Built-in high performance search capabilities.
- Flexible deployment options across cloud, on-premises and hybrid environments.
Icsm2011 syer
This document explores the development of micro-apps across different mobile platforms. It finds that Android apps are typically smaller in size than equivalent apps on BlackBerry, but Android apps also depend more on third-party libraries and the platform itself. This high platform dependency for Android apps can reduce development effort due to platform support but also increases maintenance effort and makes porting apps to other platforms more difficult. The document discusses how these findings for micro-apps may differ compared to desktop applications.
Career & Competence - exercise package
The document provides information and exercises for a career coaching group for international degree students. The VALOA project aims to increase employment opportunities for international students in Finland by helping employers recruit foreign graduates and helping universities support international students.
The document describes a career coaching group piloted at the University of Oulu to help international students understand the Finnish job market and improve their career planning. The group covered topics like career goals, skills assessment, networking, and cultural aspects of work.
Various exercises used in the coaching group are presented, related to topics like envisioning one's future career, skills assessment, personal networks, resume building, and practicing interviews. The exercises provide ways for students to reflect on work experiences, values, strengths,
Choosing a College Major
This document provides guidance on choosing a college major, noting that while a major must eventually be selected, schools differ on timing and some allow custom majors. It recommends considering interests, skills, goals, and discussing options with counselors when selecting a major, and notes majors and minors don't have to relate but can complement each other, and double majors don't necessarily extend time in college.
Career Coaching for Students
Career Coaching for Students is a program for teen career exploration. High school students enjoy highly validated and reliable assessments (which means it doesn't take an hour to complete), research strategies and decision making tools.
Career Coaching for Students™
Career Coaching for Students™ is a program for high school freshman through senior years. Focus is on self awareness, research strategies, planning and life skill development.
The college version is perfect for Freshman through Senior college students wanting to gain clarity around career options.
Job hunting in the 21st century for students and recent grads
This document provides guidance on effective job hunting strategies, especially in challenging economic times. It emphasizes the importance of networking, developing contacts, and taking personal responsibility for one's job search over more passive methods. Key recommendations include actively participating in industry groups, volunteering, engaging contacts through questions, maintaining a contact database, and following up consistently.
Icse2013 shang
1. The document discusses an approach for assisting developers in deploying big data analytics applications on Hadoop clouds by analyzing execution logs and sequences.
2. The approach involves abstracting, linking, and simplifying logs to reduce the amount of data for manual examination and provide context about execution events.
3. An evaluation on sample systems found the approach reduced the logs requiring manual inspection by over 86% and had fewer false positives than a traditional keyword search approach.
Trivento summercamp masterclass 9/9/2016
This document provides a high-level summary of streaming data processing and the Lambda architecture. It begins with a brief history of batch and streaming systems for big data. It then introduces the Lambda architecture as a way to handle both batch and streaming data using separate batch and speed layers. The document discusses advantages and disadvantages of the Lambda architecture, as well as use cases, implementation tips, and approaches that have emerged beyond the Lambda architecture like Kappa and FastData architectures.
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Flink Forward San Francisco 2022.
Being in the payments space, Stripe requires strict correctness and freshness guarantees. We rely on Flink as the natural solution for delivering on this in support of our Change Data Capture (CDC) infrastructure. We heavily rely on CDC as a tool for capturing data change streams from our databases without critically impacting database reliability, scalability, and maintainability. Data derived from these streams is used broadly across the business and powers many of our critical financial reporting systems totalling over $640 Billion in payment volume annually. We use many components of Flink’s flexible DataStream API to perform aggregations and abstract away the complexities of stream processing from our downstreams. In this talk, we’ll walk through our experience from the very beginning to what we have in production today. We’ll share stories around the technical details and trade-offs we encountered along the way.
by
Jeff Chao
2015-09-16 IoT in Oil and Gas Conference
The IoT is a game-changer opening the Systems Engineer and the Data Scientist to the O&G Development teams. Upstream, mid-stream, and downstream segments of the market are confronted by the big question “Now what?”
Time series databases
Slides from a talk delivered during 4Developers conference in Warsaw covering concepts of time-series databases.
4Developers: Time series databases
Proces implementacji systemu IT nie kończy się na zatwierdzeniu zmian w repozytorium i wydaniu kodu na serwery produkcyjne. W tym miejscu zaczyna się jego nowy etap, obserwacji zachowania aplikacji w realnym środowisku, zbierania i analizowania wszelkich informacji czy też wreszcie reakcji na wykryte problemy. Od niskopoziomowych danych systemowych o wykorzystaniu CPI czy karty sieciowej, przez informacje o procesach wewnątrz aplikacji po wskaźniki biznesowe PKI, każdy fragment informacji się liczy. Wykorzystując dedykowane silniki dla danych typu time-series każdy zespół w projekcie może otrzymać dashboardy z kluczowymi informacjami, planowanie zasobów staje się łatwiejsze, możliwe jest również wykrywanie różnego rodzaju anomalii w zachowaniu całego systemu. Podczas tej sesji zapoznasz się z kluczowymi konceptami związane z gromadzeniem i przetwarzaniem danych metrycznych, a także z dostępnymi stosami technologicznymi. Nauczyć się jak identyfikować, tworzyć i używać kluczowych metryk dla twojej aplikacji, by mieć pewność, że zachowuje się on zgodnie z oczekiwaniami.
Health & Status Monitoring (2010-v8)
This document describes two case studies of health and status monitoring systems used to monitor large, complex datasets and detect anomalies. In the first case study, a system monitored thousands of servers in a data center and detected dead or slow nodes that reduced overall performance. The second case study monitored billions of payment card transactions and developed over 15,500 statistical models to detect data quality issues and interoperability problems, improving approval rates and saving millions. Both cases highlighted the importance of executive support, dashboards, governance programs, and developing numerous statistical models tailored to different data segments.
Elections Canada
Mr. Gagnon was the Business Intelligence Architect for Elections Canada from January 2015 to present. His primary roles were to assist with an agency-wide information management and business intelligence solution to report on the status of operational systems and integrate data to detect trends during electoral events. He delivered a functional data mart for the 2015 General Election and recommendations for a long-term BI architecture.
The Connected Data Imperative: Why Graphs? at Neo4j GraphDay New York City
Philip Rathle, VP of Product at Neo4j, presents on the Connected Data Imperative at Neo4j GraphDay NYC
The future Proof Financial: Fintech
The document provides an overview of financial technology (fintech) and how it is impacting and transforming the finance industry. It discusses how fintech startups are developing technologies and solutions that can perform various banking and financial services functions. The document outlines the different areas of finance that fintech is influencing, including transaction processing, external reporting, purchase to payment, and order to cash processes. It also examines how fintech is enabling more strategic functions like planning, control, management reporting, and specialized expertise. Finally, the document explores some of the key companies and technologies that are driving innovation in fintech.
Trivento summercamp fast data 9/9/2016
This document discusses fast data and streaming systems. It provides a history of big data processing from MapReduce to streaming. Fast data refers to data in motion that is processed in real-time from streaming sources. Streaming systems allow for processing unbounded datasets using techniques like windows, watermarks and triggers. The document discusses streaming architectures and the SMACK stack (Spark, Mesos, Akka, Cassandra and Kafka) that provides technologies for building high performing streaming systems. It provides an example IoT application and how machine learning could be added. Streaming systems like Flink and Spark Streaming are compared.
The Machine Learning Audit. MIS ITAC 2017 Keynote
As it gains wider adoption, what does machine learning mean for internal auditors and their organizations? With the proliferation of buzzwords and the black box nature of machine learning, Mr. Clark will help you cut through the noise and understand what fundamental changes are occurring and what is still more hype than reality. The session will include an overview of what machine learning is, examine its current and potential impact on industries and organizations, and explain the need for an objective audit. The presentation will conclude with an example of what a machine learning audit would consist of, and what steps would be required to perform one.
Streaming HYpothesis REasoning
Addresses streaming data challenges in sampling rates, cache maintenance, deductive reasoning, and the surrounding Semantic Web framework. Using a fixed-size cache, the challenge is to identify and preserve assertions within a stream. Deductive reasoning will continuously be performed over the cache to draw relevant conclusions as quickly as possible. The use of a cache differentiates our work from state-of-the-art works in deductive stream reasoning in that the cache enables us to temporarily store propositions that are no longer in the stream window.
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Prometheus is a next-generation monitoring system with a time series database at it's core. Once you have a time series database, what do you do with it though? This talk will look at getting data in, and more importantly how to use the data you collect productively.
Contact us at prometheus@robustperception.io
MADES - A Multi-Layered, Adaptive, Distributed Event Store
This demo was presented at DEBS'13 on July 1, 2013 in Arlington, Texas, USA.
The full paper and more information is available at:
http://msrg.org/papers/DEBS13Mades
Abstract:
Application performance monitoring (APM) is shifting towards capturing and analyzing every event that arises in an enterprise infrastructure. Current APM systems, for example, make it possible to monitor enterprise applications at the granularity of tracing each method invocation (i.e., an event). Naturally, there is great interest in monitoring these events in real-time to react to system and application failures and in storing the captured information for an extended period of time to enable detailed system analysis, data analytics, and future auditing of trends in the historic data. However, the high insertion-rates (up to millions of events per second) and the purposely limited resource, a small fraction of all enterprise resources (i.e., 1-2% of the overall system resources), dedicated to APM are the key challenges for applying current data management solutions in this context. Emerging distributed key-value stores, often positioned to operate at this scale, induce additional storage overhead when dealing with relatively small data points (e.g., method invocation events) inserted at a rate of millions per second. Thus, they are not a promising solution for such an important class of workloads given APM's highly constrained resource budget. In this paper, to address these shortcomings, we present Multilayered, Adaptive, Distributed Event Store (MADES): a massively distributed store for collecting, querying, and storing event data at a rate of millions of events per second.
Systems analysis and design (abe)
This document provides an overview of information and systems concepts. It defines data and information, explaining that information is data that has been processed to convey meaning. It discusses the different types of information needed in organizations, including operating, management, trigger, and background information. It also covers systems theory concepts like objectives, properties, and types of information systems.
EDA Meets Data Engineering – What's the Big Deal?
Presenter: Guru Sattanathan, Systems Engineer, Confluent
Event-driven architectures have been around for many years, much like Apache Kafka®, which first open sourced in 2011. The reality is that the true potential of Kafka is only being realised now. Kafka is becoming the central nervous system of many of today’s enterprises. It is bringing a profound paradigm shift to the way we think about enterprise IT. What has changed in Kafka to enable this paradigm shift? Is it not just a message broker, and how are enterprises using it today? This session will explore these key questions.
Sydney: https://content.deloitte.com.au/20200221-tel-event-tech-community-syd-registration
Melbourne: https://content.deloitte.com.au/20200221-tel-event-tech-community-mel-registration
Finding bad apples early: Minimizing performance impact
The big data era is characterized by the ever-increasing velocity and volume of data. In order to store and analyze the ever-growing data, the operational footprint of data stores and Hadoop have also grown over time. (As per a recent report from IDC, the spending on big data infrastructure is expected to reach $41.5 billion by 2018.) The clusters comprise several thousands of nodes. The high performance of such clusters is vital for delivering the best user experience and productivity of teams.
The performance of such clusters is often limited by slow/bad nodes. Finding slow nodes in large clusters is akin to finding a needle in a haystack; hence, manual identification of slow/bad nodes is not practical. To this end, we developed a novel statistical technique to automatically detect slow/bad nodes in clusters comprising hundreds to thousands of nodes. We modeled the problem as a classification problem and employed a simple, yet very effective, distance measure to determine slow/bad nodes. The key highlights of the proposed technique are the following:
# Robustness against anomalies (note that anomalies may occur, for example, due to an ad-hoc heavyweight job on a Hadoop cluster)
# Given the varying data characteristics of different services, no one model fits all. Consequently, we parameterized the threshold used for classification
The proposed technique works well with both hourly and daily data, and has been in use in production by multiple services. This has not only eliminated manual investigation efforts, but has also mitigated the impact of slow nodes, which used to get detected after several weeks/months of lag!
We shall walk the audience through how the techniques are being used with REAL data.
OpsRamp Platform Winter 2020 Release
Simplify and transform IT operations management.
The OpsRamp Winter 2020 Release advances AIOps, cloud, and synthetic monitoring. Watch this on-demand webinar and get to know all the new IT Ops management platform enhancements including:
- OpsQ Recommend Mode. Stay in control by using explainable and transparent analytical recommendations for first-response and incident creation.
- Deeper Cloud Monitoring. Support for 4 AWS, 7 Azure, and 8 GCP cloud services.
- Synthetic Monitoring. Deeper insights and analysis for troubleshooting multi-step transactions.
- Agentless Discovery and Monitoring for Windows Servers. Manage distributed Windows infrastructure in a secure and frictionless way with agentless monitoring.
Hear from the OpsRamp product management team and dive into comprehensive demonstrations.
Learn more at https://www.opsramp.com
Also, follow us on social media channels to learn about product highlights, news, announcements, events, conferences and more:
Twitter - https://www.twitter.com/OpsRamp
LinkedIn - https://www.linkedin.com/company/opsramp
Facebook - https://www.facebook.com/OpsRampHQ/
Bhawani prasad data integration-ppt
The document provides an overview of an organization's data integration strategy. It discusses the scope of integration, including defining IT enablement needs, standards for data definition and information flow, and infrastructure requirements. It also outlines key focus areas such as process integration, data integration, and data management. The document summarizes the organization's business requirements for integration and provides examples of system data flows and interfaces between systems. Finally, it compares different integration technologies such as EII, EAI, ETL, and CDC.
Studying the Integration Practices and the Evolution of Ad Libraries in the G...
In-app advertisements have become a major revenue for app developers in the mobile app economy. Ad libraries play an integral part in this ecosystem as app
developers integrate these libraries into their apps to display ads. However, little is known about how app developers integrate these libraries with their apps and how these libraries have evolved over time.
In this thesis, we study the ad library integration practices and the evolution of such libraries. To understand the integration practices of ad libraries, we manually study apps and derive a set of rules to automatically identify four strategies for integrating
multiple ad libraries. We observe that integrating multiple ad libraries commonly occurs in apps with a large number of downloads and ones in categories with a high percentage of apps that display ads. We also observe that app developers prefer to manage their own integrations instead of using off the shelf features of ad libraries for integrating multiple ad libraries.
To study the evolution of ad libraries, we conduct a longitudinal study of the 8 most popular ad libraries. In particular, we look at their evolution in terms of size, the main drivers for releasing a new ad library version, and their architecture. We observe that ad libraries are continuously evolving with a median release interval of 34 days. Some ad libraries have grown exponentially in size (e.g., Facebook Audience Network ad library), while other libraries have worked to reduce their size. To study the main drivers for releasing an ad library version, we manually study the release notes of the eight studied ad libraries. We observe that ad library developers continuously update their ad libraries to support a wider range of Android versions (i.e., to ensure that more devices can use the libraries without errors). Finally, we derive a reference architecture for ad libraries and study how the studied ad libraries diverged from this architecture during our study period.
Our findings can assist ad library developers to understand the challenges for developing ad libraries and the desired features of these libraries.
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
Presenting our paper "Studying the Dialogue Between Users and Developers of Free Apps in the Google Play Store" at ICSE 2018
More Related Content
Similar to Ian wcre2011
Icse2013 shang
1. The document discusses an approach for assisting developers in deploying big data analytics applications on Hadoop clouds by analyzing execution logs and sequences.
2. The approach involves abstracting, linking, and simplifying logs to reduce the amount of data for manual examination and provide context about execution events.
3. An evaluation on sample systems found the approach reduced the logs requiring manual inspection by over 86% and had fewer false positives than a traditional keyword search approach.
Trivento summercamp masterclass 9/9/2016
This document provides a high-level summary of streaming data processing and the Lambda architecture. It begins with a brief history of batch and streaming systems for big data. It then introduces the Lambda architecture as a way to handle both batch and streaming data using separate batch and speed layers. The document discusses advantages and disadvantages of the Lambda architecture, as well as use cases, implementation tips, and approaches that have emerged beyond the Lambda architecture like Kappa and FastData architectures.
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Flink Forward San Francisco 2022.
Being in the payments space, Stripe requires strict correctness and freshness guarantees. We rely on Flink as the natural solution for delivering on this in support of our Change Data Capture (CDC) infrastructure. We heavily rely on CDC as a tool for capturing data change streams from our databases without critically impacting database reliability, scalability, and maintainability. Data derived from these streams is used broadly across the business and powers many of our critical financial reporting systems totalling over $640 Billion in payment volume annually. We use many components of Flink’s flexible DataStream API to perform aggregations and abstract away the complexities of stream processing from our downstreams. In this talk, we’ll walk through our experience from the very beginning to what we have in production today. We’ll share stories around the technical details and trade-offs we encountered along the way.
by
Jeff Chao
2015-09-16 IoT in Oil and Gas Conference
The IoT is a game-changer opening the Systems Engineer and the Data Scientist to the O&G Development teams. Upstream, mid-stream, and downstream segments of the market are confronted by the big question “Now what?”
Time series databases
Slides from a talk delivered during 4Developers conference in Warsaw covering concepts of time-series databases.
4Developers: Time series databases
Proces implementacji systemu IT nie kończy się na zatwierdzeniu zmian w repozytorium i wydaniu kodu na serwery produkcyjne. W tym miejscu zaczyna się jego nowy etap, obserwacji zachowania aplikacji w realnym środowisku, zbierania i analizowania wszelkich informacji czy też wreszcie reakcji na wykryte problemy. Od niskopoziomowych danych systemowych o wykorzystaniu CPI czy karty sieciowej, przez informacje o procesach wewnątrz aplikacji po wskaźniki biznesowe PKI, każdy fragment informacji się liczy. Wykorzystując dedykowane silniki dla danych typu time-series każdy zespół w projekcie może otrzymać dashboardy z kluczowymi informacjami, planowanie zasobów staje się łatwiejsze, możliwe jest również wykrywanie różnego rodzaju anomalii w zachowaniu całego systemu. Podczas tej sesji zapoznasz się z kluczowymi konceptami związane z gromadzeniem i przetwarzaniem danych metrycznych, a także z dostępnymi stosami technologicznymi. Nauczyć się jak identyfikować, tworzyć i używać kluczowych metryk dla twojej aplikacji, by mieć pewność, że zachowuje się on zgodnie z oczekiwaniami.
Health & Status Monitoring (2010-v8)
This document describes two case studies of health and status monitoring systems used to monitor large, complex datasets and detect anomalies. In the first case study, a system monitored thousands of servers in a data center and detected dead or slow nodes that reduced overall performance. The second case study monitored billions of payment card transactions and developed over 15,500 statistical models to detect data quality issues and interoperability problems, improving approval rates and saving millions. Both cases highlighted the importance of executive support, dashboards, governance programs, and developing numerous statistical models tailored to different data segments.
Elections Canada
Mr. Gagnon was the Business Intelligence Architect for Elections Canada from January 2015 to present. His primary roles were to assist with an agency-wide information management and business intelligence solution to report on the status of operational systems and integrate data to detect trends during electoral events. He delivered a functional data mart for the 2015 General Election and recommendations for a long-term BI architecture.
The Connected Data Imperative: Why Graphs? at Neo4j GraphDay New York City
Philip Rathle, VP of Product at Neo4j, presents on the Connected Data Imperative at Neo4j GraphDay NYC
The future Proof Financial: Fintech
The document provides an overview of financial technology (fintech) and how it is impacting and transforming the finance industry. It discusses how fintech startups are developing technologies and solutions that can perform various banking and financial services functions. The document outlines the different areas of finance that fintech is influencing, including transaction processing, external reporting, purchase to payment, and order to cash processes. It also examines how fintech is enabling more strategic functions like planning, control, management reporting, and specialized expertise. Finally, the document explores some of the key companies and technologies that are driving innovation in fintech.
Trivento summercamp fast data 9/9/2016
This document discusses fast data and streaming systems. It provides a history of big data processing from MapReduce to streaming. Fast data refers to data in motion that is processed in real-time from streaming sources. Streaming systems allow for processing unbounded datasets using techniques like windows, watermarks and triggers. The document discusses streaming architectures and the SMACK stack (Spark, Mesos, Akka, Cassandra and Kafka) that provides technologies for building high performing streaming systems. It provides an example IoT application and how machine learning could be added. Streaming systems like Flink and Spark Streaming are compared.
The Machine Learning Audit. MIS ITAC 2017 Keynote
As it gains wider adoption, what does machine learning mean for internal auditors and their organizations? With the proliferation of buzzwords and the black box nature of machine learning, Mr. Clark will help you cut through the noise and understand what fundamental changes are occurring and what is still more hype than reality. The session will include an overview of what machine learning is, examine its current and potential impact on industries and organizations, and explain the need for an objective audit. The presentation will conclude with an example of what a machine learning audit would consist of, and what steps would be required to perform one.
Streaming HYpothesis REasoning
Addresses streaming data challenges in sampling rates, cache maintenance, deductive reasoning, and the surrounding Semantic Web framework. Using a fixed-size cache, the challenge is to identify and preserve assertions within a stream. Deductive reasoning will continuously be performed over the cache to draw relevant conclusions as quickly as possible. The use of a cache differentiates our work from state-of-the-art works in deductive stream reasoning in that the cache enables us to temporarily store propositions that are no longer in the stream window.
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Prometheus is a next-generation monitoring system with a time series database at it's core. Once you have a time series database, what do you do with it though? This talk will look at getting data in, and more importantly how to use the data you collect productively.
Contact us at prometheus@robustperception.io
MADES - A Multi-Layered, Adaptive, Distributed Event Store
This demo was presented at DEBS'13 on July 1, 2013 in Arlington, Texas, USA.
The full paper and more information is available at:
http://msrg.org/papers/DEBS13Mades
Abstract:
Application performance monitoring (APM) is shifting towards capturing and analyzing every event that arises in an enterprise infrastructure. Current APM systems, for example, make it possible to monitor enterprise applications at the granularity of tracing each method invocation (i.e., an event). Naturally, there is great interest in monitoring these events in real-time to react to system and application failures and in storing the captured information for an extended period of time to enable detailed system analysis, data analytics, and future auditing of trends in the historic data. However, the high insertion-rates (up to millions of events per second) and the purposely limited resource, a small fraction of all enterprise resources (i.e., 1-2% of the overall system resources), dedicated to APM are the key challenges for applying current data management solutions in this context. Emerging distributed key-value stores, often positioned to operate at this scale, induce additional storage overhead when dealing with relatively small data points (e.g., method invocation events) inserted at a rate of millions per second. Thus, they are not a promising solution for such an important class of workloads given APM's highly constrained resource budget. In this paper, to address these shortcomings, we present Multilayered, Adaptive, Distributed Event Store (MADES): a massively distributed store for collecting, querying, and storing event data at a rate of millions of events per second.
Systems analysis and design (abe)
This document provides an overview of information and systems concepts. It defines data and information, explaining that information is data that has been processed to convey meaning. It discusses the different types of information needed in organizations, including operating, management, trigger, and background information. It also covers systems theory concepts like objectives, properties, and types of information systems.
EDA Meets Data Engineering – What's the Big Deal?
Presenter: Guru Sattanathan, Systems Engineer, Confluent
Event-driven architectures have been around for many years, much like Apache Kafka®, which first open sourced in 2011. The reality is that the true potential of Kafka is only being realised now. Kafka is becoming the central nervous system of many of today’s enterprises. It is bringing a profound paradigm shift to the way we think about enterprise IT. What has changed in Kafka to enable this paradigm shift? Is it not just a message broker, and how are enterprises using it today? This session will explore these key questions.
Sydney: https://content.deloitte.com.au/20200221-tel-event-tech-community-syd-registration
Melbourne: https://content.deloitte.com.au/20200221-tel-event-tech-community-mel-registration
Finding bad apples early: Minimizing performance impact
The big data era is characterized by the ever-increasing velocity and volume of data. In order to store and analyze the ever-growing data, the operational footprint of data stores and Hadoop have also grown over time. (As per a recent report from IDC, the spending on big data infrastructure is expected to reach $41.5 billion by 2018.) The clusters comprise several thousands of nodes. The high performance of such clusters is vital for delivering the best user experience and productivity of teams.
The performance of such clusters is often limited by slow/bad nodes. Finding slow nodes in large clusters is akin to finding a needle in a haystack; hence, manual identification of slow/bad nodes is not practical. To this end, we developed a novel statistical technique to automatically detect slow/bad nodes in clusters comprising hundreds to thousands of nodes. We modeled the problem as a classification problem and employed a simple, yet very effective, distance measure to determine slow/bad nodes. The key highlights of the proposed technique are the following:
# Robustness against anomalies (note that anomalies may occur, for example, due to an ad-hoc heavyweight job on a Hadoop cluster)
# Given the varying data characteristics of different services, no one model fits all. Consequently, we parameterized the threshold used for classification
The proposed technique works well with both hourly and daily data, and has been in use in production by multiple services. This has not only eliminated manual investigation efforts, but has also mitigated the impact of slow nodes, which used to get detected after several weeks/months of lag!
We shall walk the audience through how the techniques are being used with REAL data.
OpsRamp Platform Winter 2020 Release
Simplify and transform IT operations management.
The OpsRamp Winter 2020 Release advances AIOps, cloud, and synthetic monitoring. Watch this on-demand webinar and get to know all the new IT Ops management platform enhancements including:
- OpsQ Recommend Mode. Stay in control by using explainable and transparent analytical recommendations for first-response and incident creation.
- Deeper Cloud Monitoring. Support for 4 AWS, 7 Azure, and 8 GCP cloud services.
- Synthetic Monitoring. Deeper insights and analysis for troubleshooting multi-step transactions.
- Agentless Discovery and Monitoring for Windows Servers. Manage distributed Windows infrastructure in a secure and frictionless way with agentless monitoring.
Hear from the OpsRamp product management team and dive into comprehensive demonstrations.
Learn more at https://www.opsramp.com
Also, follow us on social media channels to learn about product highlights, news, announcements, events, conferences and more:
Twitter - https://www.twitter.com/OpsRamp
LinkedIn - https://www.linkedin.com/company/opsramp
Facebook - https://www.facebook.com/OpsRampHQ/
Bhawani prasad data integration-ppt
The document provides an overview of an organization's data integration strategy. It discusses the scope of integration, including defining IT enablement needs, standards for data definition and information flow, and infrastructure requirements. It also outlines key focus areas such as process integration, data integration, and data management. The document summarizes the organization's business requirements for integration and provides examples of system data flows and interfaces between systems. Finally, it compares different integration technologies such as EII, EAI, ETL, and CDC.
Similar to Ian wcre2011 (20)
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...
The Connected Data Imperative: Why Graphs? at Neo4j GraphDay New York City
The Connected Data Imperative: Why Graphs? at Neo4j GraphDay New York City
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
MADES - A Multi-Layered, Adaptive, Distributed Event Store
MADES - A Multi-Layered, Adaptive, Distributed Event Store
Finding bad apples early: Minimizing performance impact
Finding bad apples early: Minimizing performance impact
More from SAIL_QU
Studying the Integration Practices and the Evolution of Ad Libraries in the G...
In-app advertisements have become a major revenue for app developers in the mobile app economy. Ad libraries play an integral part in this ecosystem as app
developers integrate these libraries into their apps to display ads. However, little is known about how app developers integrate these libraries with their apps and how these libraries have evolved over time.
In this thesis, we study the ad library integration practices and the evolution of such libraries. To understand the integration practices of ad libraries, we manually study apps and derive a set of rules to automatically identify four strategies for integrating
multiple ad libraries. We observe that integrating multiple ad libraries commonly occurs in apps with a large number of downloads and ones in categories with a high percentage of apps that display ads. We also observe that app developers prefer to manage their own integrations instead of using off the shelf features of ad libraries for integrating multiple ad libraries.
To study the evolution of ad libraries, we conduct a longitudinal study of the 8 most popular ad libraries. In particular, we look at their evolution in terms of size, the main drivers for releasing a new ad library version, and their architecture. We observe that ad libraries are continuously evolving with a median release interval of 34 days. Some ad libraries have grown exponentially in size (e.g., Facebook Audience Network ad library), while other libraries have worked to reduce their size. To study the main drivers for releasing an ad library version, we manually study the release notes of the eight studied ad libraries. We observe that ad library developers continuously update their ad libraries to support a wider range of Android versions (i.e., to ensure that more devices can use the libraries without errors). Finally, we derive a reference architecture for ad libraries and study how the studied ad libraries diverged from this architecture during our study period.
Our findings can assist ad library developers to understand the challenges for developing ad libraries and the desired features of these libraries.
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
Presenting our paper "Studying the Dialogue Between Users and Developers of Free Apps in the Google Play Store" at ICSE 2018
Improving the testing efficiency of selenium-based load tests
Slides for a paper published at AST 2019:
Shahnaz M. Shariff, Heng Li, Cor-Paul Bezemer, Ahmed E. Hassan, Thanh H. D. Nguyen, and Parminder Flora. 2019. Improving the testing efficiency of selenium-based load tests. In Proceedings of the 14th International Workshop on Automation of Software Test (AST '19). IEEE Press, Piscataway, NJ, USA, 14-20. DOI: https://doi.org/10.1109/AST.2019.00008
Studying User-Developer Interactions Through the Distribution and Reviewing M...
This document discusses studying user-developer interactions through the distribution and reviewing mechanisms of the Google Play Store. It analyzes emergency updates made by developers to fix issues, the dialogue between users and developers through reviews and responses, and how the reviewing mechanism can help identify good and bad updates. The study found that responding to reviews is six times more likely to increase an app's rating, with 84% of rating increases going to four or five stars. Three common patterns of developer responses were identified: responding to negative or long reviews, only negative reviews, and reviews shortly after an update.
Studying online distribution platforms for games through the mining of data f...
Our studies of Steam platform data provided insights into online game distribution:
1) Urgent game updates were used to fix crashes, balance issues, and functionality; frequent updaters released more 0-day patches.
2) The Early Access model attracted indie developers and increased game participation; reviews were more positive during Early Access.
3) Game reviews were typically short and in English; sales increased review volume more than new updates; negative reviews came after longer play.
Understanding the Factors for Fast Answers in Technical Q&A Websites: An Empi...
This study analyzed factors that impact the speed of questions receiving accepted answers on four popular Stack Exchange websites: Stack Overflow, Mathematics, Ask Ubuntu, and Super User. The researchers examined question, answerer, asker, and answer factors from over 150,000 questions. They built classification models and found that key factors for fast answers included the past speed of answerers, length of the question, and past speed of answers for the question's tags. The models achieved AUCs of 0.85-0.95. Fast answers relied heavily on answerers, especially frequent answerers. The study suggests improving incentives for non-frequent and more difficult questions to attract diverse answerers.
Investigating the Challenges in Selenium Usage and Improving the Testing Effi...
Selenium is a popular tool for browser-based automation testing. The author analyzes challenges in using Selenium by mining Selenium questions on Stack Overflow. Programming language-related questions, especially for Java and Python, are most common and growing fastest. Less than half of questions receive accepted answers, and questions about browsers and components take longest. In the second part, the author develops an approach to improve efficiency of Selenium-based load testing by sharing browsers among user instances. This increases the number of error-free users by 20-22% while reducing memory usage.
Mining Development Knowledge to Understand and Support Software Logging Pract...
This document summarizes Heng Li's PhD thesis on mining development knowledge to understand and support software logging practices. It discusses how logging code is used to record runtime information but can be difficult for developers to maintain. The thesis aims to understand current logging practices and develop tools by mining change history, source code, issue reports, and other development knowledge. It presents research that analyzes logging-related issues to identify developers' logging concerns, uses code topics and structure to predict where logging statements should be added, leverages code changes to suggest when logging code needs updating, and applies machine learning models to recommend appropriate log levels.
Which Log Level Should Developers Choose For a New Logging Statement?
The document discusses choosing an appropriate log level when adding a new logging statement. It finds that an ordinal regression model can effectively model log levels, achieving an AUC of 0.76-0.81 in within-project evaluation and 0.71-0.8 in cross-project evaluation. The most influential factors for determining log levels vary between projects and include metrics related to the logging statement, containing code block, and file as well as code change and historical change metrics.
Towards Just-in-Time Suggestions for Log Changes
The document presents a study on providing just-in-time suggestions for log changes when developers make code changes. The researchers analyzed over 32,000 log changes from 4 systems. They found 20 reasons for log changes that fall into 4 categories: block changes, log improvements, dependence-driven changes, and logging issues. A random forest classifier using 25 software metrics related to code changes, history, and complexity achieved 0.84-0.91 AUC in predicting whether a log change is needed. Change metrics and product metrics were the most influential factors. The study aims to help developers make better logging decisions for failure diagnosis.
The Impact of Task Granularity on Co-evolution Analyses
The document discusses how task granularity at different levels (e.g. commits, pull requests, work items) can impact analyses of co-evolution in software projects. It finds that analyzing at the commit-level can overlook relationships between tasks that span multiple commits. Work item level analysis is recommended to provide a more complete view of co-evolution, as median of 29% of work items consist of multiple commits, and analyzing at the commit level would miss 24% of co-changed files and inability to group 83% of related commits.
A Framework for Evaluating the Results of the SZZ Approach for Identifying Bu...
Presented by Daniel da Costa, Shane McIntosh, Weiyi (Ian) Shang, Uira Kulesza, Roberta Coelho, and Ahmed E. Hassan
How are Discussions Associated with Bug Reworking? An Empirical Study on Open...
1) Initial bug fix discussions with more comments and more developers participating are more likely to experience later bug reworking through re-opening or re-patching of the bug.
2) Manual analysis found that defective initial fixes and failure to reach consensus in discussions contributed to later reworking.
3) For re-opened bugs, initial discussions focused on addressing a particular problem through a burst of comments, while re-patched bugs lacked thorough code review and testing during the initial fix period.
A Study of the Relation of Mobile Device Attributes with the User-Perceived Q...
This study examined the relationship between mobile device attributes and user-perceived quality of Android apps. The researchers analyzed 150,373 star ratings from Google Play across 30 devices and 280 apps. They found that the perceived quality of apps varies across devices, and having better characteristics of an attribute does not necessarily correlate with higher quality. Device OS version, resolution, and CPU showed significant relationships with ratings, as did some app attributes like lines of code and number of inputs. However, some device attributes had stronger relationships than app attributes.
A Large-Scale Study of the Impact of Feature Selection Techniques on Defect C...
This document presents the results of a large-scale study on the impact of feature selection techniques on defect classification models. The study used expanded scopes including multiple datasets from NASA and PROMISE with different feature types, more classification techniques from different paradigms, and additional feature selection techniques. The results show that correlation-based feature subset selection techniques like FS1 and FS2 consistently appear in the top ranks across most of the datasets, projects within the datasets, and classification techniques. The document concludes that future defect classification studies should consider applying correlation-based feature selection techniques.
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
The study analyzes user-developer interactions through reviews and responses on the Google Play Store. It finds that responding to reviews has a significant positive impact, with 84% of rating increases due to the developer addressing the issue or providing guidance. Three common response patterns were identified: only negative reviews, negative or longer reviews, and reviews shortly after an update. Developers most often thank the user, ask for details, provide guidance, or ask for an endorsement. Guidance responses can address common issues through FAQs. The analysis considered over 2,000 apps, 355,000 review changes, 128,000 responses, and 4 million reviews.
What Do Programmers Know about Software Energy Consumption?
This document summarizes the results of a survey of 122 programmers about their knowledge of software energy consumption. The survey found that programmers have limited awareness of energy consumption and how to reduce it. They were unaware of the main causes of high energy usage. Programmers lacked knowledge about how to properly rank the energy consumption of different hardware components and were unfamiliar with strategies to improve efficiency, such as minimizing I/O and avoiding polling. The study concludes that programmers would benefit from more education on software energy usage and its causes.
Threshold for Size and Complexity Metrics: A Case Study from the Perspective ...
Presented by Kazuhiro Yamashita, Changyun Huang, Meiyappan Nagappan, Yasutaka Kamei, Audris Mockus, Ahmed E. Hassan, and Naoyasu Ubayashi
Revisiting the Experimental Design Choices for Approaches for the Automated R...
Prior research on automated duplicate issue report retrieval focused on improving performance metrics like recall rate. The author revisits experimental design choices from four perspectives: needed effort, data changes, data filtration, and evaluation process.
The thesis contributions are: 1) Showing the importance of considering needed effort in performance measurement. 2) Proposing a "realistic evaluation" approach and analyzing prior findings with it. 3) Developing a genetic algorithm to filter old issue reports and improve performance. 4) Highlighting the impact of "just-in-time" features on evaluation. The findings help better understand benefits and limitations of prior work in this area.
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
The document summarizes a large-scale field study that tracked the program comprehension activities of 78 professional developers over 3,148 hours. The study found that:
1) Program comprehension accounted for approximately 58% of developers' time on average, with navigation and editing making up the remaining portions.
2) Developers frequently used web browsers and document editors to aid comprehension beyond just IDEs.
3) Interviews and observations revealed that insufficient documentation, unclear code, and complex inheritance hierarchies contributed to long comprehension sessions.
More from SAIL_QU (20)
Studying the Integration Practices and the Evolution of Ad Libraries in the G...
Studying the Integration Practices and the Evolution of Ad Libraries in the G...
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
Improving the testing efficiency of selenium-based load tests
Improving the testing efficiency of selenium-based load tests
Studying User-Developer Interactions Through the Distribution and Reviewing M...
Studying User-Developer Interactions Through the Distribution and Reviewing M...
Studying online distribution platforms for games through the mining of data f...
Studying online distribution platforms for games through the mining of data f...
Understanding the Factors for Fast Answers in Technical Q&A Websites: An Empi...
Understanding the Factors for Fast Answers in Technical Q&A Websites: An Empi...
Investigating the Challenges in Selenium Usage and Improving the Testing Effi...
Investigating the Challenges in Selenium Usage and Improving the Testing Effi...
Mining Development Knowledge to Understand and Support Software Logging Pract...
Mining Development Knowledge to Understand and Support Software Logging Pract...
Which Log Level Should Developers Choose For a New Logging Statement?
Which Log Level Should Developers Choose For a New Logging Statement?
The Impact of Task Granularity on Co-evolution Analyses
The Impact of Task Granularity on Co-evolution Analyses
A Framework for Evaluating the Results of the SZZ Approach for Identifying Bu...
A Framework for Evaluating the Results of the SZZ Approach for Identifying Bu...
How are Discussions Associated with Bug Reworking? An Empirical Study on Open...
How are Discussions Associated with Bug Reworking? An Empirical Study on Open...
A Study of the Relation of Mobile Device Attributes with the User-Perceived Q...
A Study of the Relation of Mobile Device Attributes with the User-Perceived Q...
A Large-Scale Study of the Impact of Feature Selection Techniques on Defect C...
A Large-Scale Study of the Impact of Feature Selection Techniques on Defect C...
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
What Do Programmers Know about Software Energy Consumption?
What Do Programmers Know about Software Energy Consumption?
Threshold for Size and Complexity Metrics: A Case Study from the Perspective ...
Threshold for Size and Complexity Metrics: A Case Study from the Perspective ...
Revisiting the Experimental Design Choices for Approaches for the Automated R...
Revisiting the Experimental Design Choices for Approaches for the Automated R...
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
Ian wcre2011
- 1. An Exploratory Study of the Evolution of Communicated Information about the Execution of Large Software Systems Weiyi Shang Zhen Ming Jiang Bram Adams Ahmed E. Hassan Michael W. Godfrey University of WaterlooQueen’s University Mohamed Nasser Parminder Flora Research In Motion (RIM)
- 2. 2 What run-time actions cause the failure?
- 3. Automated profiling & instrumentation Detail No domain knowledgeLarge scale 3
- 4. Communicated information (CI) Execution Logs System Alerts Code Comments /* … */ StaticDynamic 4 Field experienceDeveloper experience
- 5. CI forms basis of Ecosystem of Log Processing Apps Workload recoveryAnomaly detection Capacity planning System monitoring Performance analysis 5 Failure diagnosis
- 6. How to keep Log Processing Apps in sync with CI? Release 1 Release 2 Release 3 6
- 7. Our Study Dimensions What types of modifications happen to CI? What information is conveyed by the short-lived CI? Quantity Type Content How does CI evolve over time? 7
- 8. Case Study Setup Data Collection Log Abstraction System Deployment time=1, Trying to launch, TaskID=01A time=$t, Trying to launch, TaskID=$id Enterprise Application (EA) 8 Log Events
- 9. Our Study Dimensions What types of modifications happen to CI? What information is conveyed by the short-lived CI? Quantity Type Content How does CI evolve over time? 9
- 10. CI keeps on growing over time 0 20 40 60 80 100 120 140 160 180 0.14.0 0.15.0 0.16.0 0.17.0 0.18.0 0.19.0 0.20.0 0.20.1 0.20.2 0.21.0 releases # execution events 10
- 11. …even when system size decreases # K SLOC # Execution log events 0.19.0 293 113 0.20.0 250 121 11
- 12. CI is impacted by re-engineering 0.00% 20.00% 40.00% 60.00% 80.00% 100.00% 0.15.0 0.16.0 0.17.0 0.18.0 0.19.0 0.20.0 0.20.1 0.20.2 0.21.0 Unchanged CI Large amounts of implementation changes 12
- 13. How does CI evolve over time? 13 Growing & changing Document & track What types of modifications happen to CI? What information is conveyed by the short-lived CI? Quantity Type Content
- 14. Six types of modification exist Rephrasing Redundant information Adding information Deleting information Diverging Merging 14
- 15. Six types of modification exist Rephrasing Redundant information Adding information Deleting information Diverging Merging Hadoop mapred Reduce task fetch n bytes Hadoop MapReduce task Reduce fetch n bytes 15
- 16. Six types of modification exist Rephrasing Redundant information Adding information Deleting information Diverging Merging ShuffleRamManager memory limit n MaxSingleShuffleLimit m ShuffleRamManager memory limit n MaxSingleShuffleLimit m mergeThreshold Q 16
- 17. Six types of modification exist Rephrasing Redundant information Adding information Deleting information Diverging Merging Adding task to tasktracker Adding Map Task to tasktracker Adding Reduce Task to tasktracker 17
- 18. Six types of modification exist Rephrasing Redundant information Adding information Deleting information Diverging Merging Avoidable 18
- 19. Six types of modification exist Rephrasing Redundant information Adding information Deleting information Diverging Merging Recoverable 19
- 20. Six types of modification exist Rephrasing Redundant information Adding information Deleting information Diverging Merging Unavoidable 20
- 21. Most modifications can be avoided 9.86% 61.97% 14.08% 7.04% 7.04% 2.82% 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% redundant info rephrasing adding info deleting info diverging merging avoidable recoverable unavoidable 21
- 22. How does CI evolve over time? 22 Growing & changing Document & track What types of modifications happen to CI? What information is conveyed by the short-lived CI? Quantity Type Content 6 types Are mostly avoidable
- 23. Short-lived CI contains implementation details Hadoop saves output to a machine. Hadoop assigns a reduce task to a machine. Map task updates its progress. Hadoop reads from a local file. Hadoop Attempt saves its output and reports to the task tracker. 23 Node name Local path Using ipc Output file name
- 24. How does CI evolve over time? 24 Growing & changing Document & track What types of modifications happen to CI? What information is conveyed by the short-lived CI? Quantity Type Content 6 types Are mostly avoidable Implementation- level details Fragile Maintenance effort
- 25. 25
- 26. 26
- 27. 27 What happened in the game? My favorite soccer team lost
- 28. Let’s try to find out by ourselves • All game details • 90 minutes • Have to form my own interpretation • Only essential events • 5 minutes • Provided by experts of the domain 28
Editor's Notes
- neon
- Apps make use of these logs
- Font bigger
- Fixing bugs then gone
- News paper