Azure Data Explorer deep dive - review 04.2020Riccardo Zamana
Full review 04.2020 about Azure Data Explorer service. Slide Desk is a sort of review od Kusto, in terms of usage, ingestion techniques, querying and exporting data, using anomaly detection and clustering methods.
Azure Data Explorer deep dive - review 04.2020Riccardo Zamana
Full review 04.2020 about Azure Data Explorer service. Slide Desk is a sort of review od Kusto, in terms of usage, ingestion techniques, querying and exporting data, using anomaly detection and clustering methods.
Development of Software for scalable anomaly detection modeling of time-series data using Apache Spark.
私たちはこれまで、様々な機器類を監視するセンサーの時系列データを分析し、異常を検知する手法およびソフトウェアの研究開発を行ってきた。
今回紹介するソフトウェアでは、バッチ処理で複数のセンサーから得られた高次元の時系列データから線形のLASSO回帰により学習、モデル化し、異常時を識別する。
しかし学習時間やメモリー使用量の増大が課題になってきたため、Sparkを活用し並列分散化を行った。
SparkにはMLlibという汎用的な機械学習ライブラリが存在するが、今回は使用するアルゴリズムの特殊性を考慮し、既存実装を基に新規に開発した。
本講演では当開発におけるデザインチョイスや性能計測結果について報告する。
a
With a long history of open innovation with Hadoop, Yahoo continues to invest in and expand the platform capabilities by pushing the boundaries of what the platform can accomplish for the entire organization. In this talk, Sumeet Singh will present some of the recent innovations, open source contributions, and where things are headed when it comes to Hadoop at Yahoo.
Hive is the main data transformation tool at Criteo, and hundreds of analysts and thousands of automated jobs run Hive queries every day. We evolved Criteo’s Hive platform from an error-prone add-on installed on some spare machines to a best-in-class installation capable of self-healing and automatically scaling to handle its growing load. The resulting platform is based on Mesos. Mesos has allowed Criteo to scale per demand and better utilize resources, iterate on development much faster than on bare metal, and roll out new versions seamlessly without downtime for our users.
Let's make a brief introduction to Azure Data eXplorer, with many examples using Kusto dialect and C# client.
With a particular focus on IIoT contexts and proces control data, let's discover how to implement time series analysis in terms of pattern recognition, and trend correlation.
The Fundamentals Guide to HDP and HDInsightGert Drapers
This session will give you the architectural overview and introduction in to inner workings of HDP 2.0 (http://hortonworks.com/products/hdp-windows/) and HDInsight. The world has embraced the Hadoop toolkit to solve their data problems from ETL, data warehouses to event processing pipelines. As Hadoop consists of many components, services and interfaces, understanding its architecture is crucial, before you can successfully integrate it in to your own environment.
Analytics at the Real-Time Speed of Business: Spark Summit East talk by Manis...Spark Summit
Redis accelerates Apache Spark execution by 45 times, when used as a shared distributed in-memory datastore for Spark in analyses like time series data range queries. With the redis module for machine learning, redis-ml, implementation of spark-ml models gains a new real time serving layer that offloads processing of models directly in Redis, allows multiple applications to reuse the same models and speeds up classification and execution of these models by 13x. Join this session to learn more about the Redis Labs’ connector for Apache Spark that enhances production implementations of real-time big data processing.
Hopsworks in the cloud Berlin Buzzwords 2019 Jim Dowling
This talk, given at Berlin Buzzwords 2019, describes the recent progress in making Hopsworks a cloud-native platform, with HA data-center support added for HopsFS.
Just-in-Time Analytics and the Need for Autonomous Database Administration wi...Databricks
Spark makes possible just-in-time analytics – moving the data warehouse into the same environment that supports ETL and non-SQL analytics. This results in the benefits of elastic compute, schema-on-read, and Spark’s unified API for graph, streaming, and machine learning. However, even with this capability, challenges for interactivity, efficiency, and scalability remain. As just-in-time analytics becomes the norm, data scientists and engineers have had to take on the capacity planning, configuration, and performance tuning roles of the DBA as well.
The Algebraix Query Accelerator for Spark shims the existing Spark DataFrames and SQL APIs so that it can unobtrusively build a model of how the users’ queries relate to the data and to each other. The AQA uses this model to predict future query characteristics and deploy optimizations to speed up future queries. SQL queries and DataFrame programs are translated into SQL-DA, a data algebra representation, and stored together in a graph-like data structure called the algebraic cache, which serves as the core data structure in the model. An exemplary use case of materializing views into common expression patterns is discussed.
AQA for Spark is an artificially intelligent agent that helps data scientists and engineers focus on the analysis task by automating the performance tuning and resource management tasks of the DBA. The AQA functions as an inter-query optimizer that complements a traditional query optimizer (Catalyst) by creating additional speed-up, which is demonstrated with a benchmark analysis.
Mobile Game Architectures on AWS (MBL201) | AWS re:Invent 2013Amazon Web Services
The gaming industry is undergoing massive changes, and AWS offers unique capabilities that game developers can use to succeed. In this session, we cover cloud gaming architectural patterns you can use to create highly available and scalable online games. We discuss games-as-APIs, database design considerations, decoupled architectures, and the best instance types for mobile, social, and AAA online games. By the end of this session, you’ll understand the different architecture patterns for the major classes of online games, as well as understand which AWS technologies will help you meet the unique challenges of each one.
Development of Software for scalable anomaly detection modeling of time-series data using Apache Spark.
私たちはこれまで、様々な機器類を監視するセンサーの時系列データを分析し、異常を検知する手法およびソフトウェアの研究開発を行ってきた。
今回紹介するソフトウェアでは、バッチ処理で複数のセンサーから得られた高次元の時系列データから線形のLASSO回帰により学習、モデル化し、異常時を識別する。
しかし学習時間やメモリー使用量の増大が課題になってきたため、Sparkを活用し並列分散化を行った。
SparkにはMLlibという汎用的な機械学習ライブラリが存在するが、今回は使用するアルゴリズムの特殊性を考慮し、既存実装を基に新規に開発した。
本講演では当開発におけるデザインチョイスや性能計測結果について報告する。
a
With a long history of open innovation with Hadoop, Yahoo continues to invest in and expand the platform capabilities by pushing the boundaries of what the platform can accomplish for the entire organization. In this talk, Sumeet Singh will present some of the recent innovations, open source contributions, and where things are headed when it comes to Hadoop at Yahoo.
Hive is the main data transformation tool at Criteo, and hundreds of analysts and thousands of automated jobs run Hive queries every day. We evolved Criteo’s Hive platform from an error-prone add-on installed on some spare machines to a best-in-class installation capable of self-healing and automatically scaling to handle its growing load. The resulting platform is based on Mesos. Mesos has allowed Criteo to scale per demand and better utilize resources, iterate on development much faster than on bare metal, and roll out new versions seamlessly without downtime for our users.
Let's make a brief introduction to Azure Data eXplorer, with many examples using Kusto dialect and C# client.
With a particular focus on IIoT contexts and proces control data, let's discover how to implement time series analysis in terms of pattern recognition, and trend correlation.
The Fundamentals Guide to HDP and HDInsightGert Drapers
This session will give you the architectural overview and introduction in to inner workings of HDP 2.0 (http://hortonworks.com/products/hdp-windows/) and HDInsight. The world has embraced the Hadoop toolkit to solve their data problems from ETL, data warehouses to event processing pipelines. As Hadoop consists of many components, services and interfaces, understanding its architecture is crucial, before you can successfully integrate it in to your own environment.
Analytics at the Real-Time Speed of Business: Spark Summit East talk by Manis...Spark Summit
Redis accelerates Apache Spark execution by 45 times, when used as a shared distributed in-memory datastore for Spark in analyses like time series data range queries. With the redis module for machine learning, redis-ml, implementation of spark-ml models gains a new real time serving layer that offloads processing of models directly in Redis, allows multiple applications to reuse the same models and speeds up classification and execution of these models by 13x. Join this session to learn more about the Redis Labs’ connector for Apache Spark that enhances production implementations of real-time big data processing.
Hopsworks in the cloud Berlin Buzzwords 2019 Jim Dowling
This talk, given at Berlin Buzzwords 2019, describes the recent progress in making Hopsworks a cloud-native platform, with HA data-center support added for HopsFS.
Just-in-Time Analytics and the Need for Autonomous Database Administration wi...Databricks
Spark makes possible just-in-time analytics – moving the data warehouse into the same environment that supports ETL and non-SQL analytics. This results in the benefits of elastic compute, schema-on-read, and Spark’s unified API for graph, streaming, and machine learning. However, even with this capability, challenges for interactivity, efficiency, and scalability remain. As just-in-time analytics becomes the norm, data scientists and engineers have had to take on the capacity planning, configuration, and performance tuning roles of the DBA as well.
The Algebraix Query Accelerator for Spark shims the existing Spark DataFrames and SQL APIs so that it can unobtrusively build a model of how the users’ queries relate to the data and to each other. The AQA uses this model to predict future query characteristics and deploy optimizations to speed up future queries. SQL queries and DataFrame programs are translated into SQL-DA, a data algebra representation, and stored together in a graph-like data structure called the algebraic cache, which serves as the core data structure in the model. An exemplary use case of materializing views into common expression patterns is discussed.
AQA for Spark is an artificially intelligent agent that helps data scientists and engineers focus on the analysis task by automating the performance tuning and resource management tasks of the DBA. The AQA functions as an inter-query optimizer that complements a traditional query optimizer (Catalyst) by creating additional speed-up, which is demonstrated with a benchmark analysis.
Mobile Game Architectures on AWS (MBL201) | AWS re:Invent 2013Amazon Web Services
The gaming industry is undergoing massive changes, and AWS offers unique capabilities that game developers can use to succeed. In this session, we cover cloud gaming architectural patterns you can use to create highly available and scalable online games. We discuss games-as-APIs, database design considerations, decoupled architectures, and the best instance types for mobile, social, and AAA online games. By the end of this session, you’ll understand the different architecture patterns for the major classes of online games, as well as understand which AWS technologies will help you meet the unique challenges of each one.
Amazon Web Services offers a comprehensive suite of products and services for video game developers across every major platform: mobile, console, PC and online. From AAA console and PC games, to educational and serious games, AWS provides the back end servers and hosting services for your game studio. Build, deploy, distribute, analyze and monetize with AWS. Pay as you go, and only pay for what you use. Focus on your game, not your infrastructure.
Join this session to learn more about how games studios and operators use the AWS cloud to support their titles. We will also dive deep into the recently announced Amazon Lumberyard and Gamelift services and explore the technical features of these services and the resources available to help game developers get started building their next creations with on AWS.
As the amount of data being created by applications increases, the requirement to keep pace in this space becomes increasingly difficult. This session covers how to properly collect, manage and present the data usefully using service offerings from Amazon Web Services such as Amazon RedShift and Amazon Kinesis. At this session we will include live demo on parsing the data generated and management of the data.
Alex Smith, Solutions Architect, Amazon Web Services, ASEAN
In this talk from the Dublin Websummit 2014 AWS Technical Evangelist Ian Massingham discusses the major trends that are changing the gaming market today and how using the cloud as a development and delivery platform for gaming products and services can help meet the challenges that these trends present.
Includes examples of gaming customers running on the AWS cloud today as well as a discussion of how you might build and scaling a gaming back-end on AWS using AWS services to enable low cost and pain free scaling of your gaming infrastructure.
In this talk from the Dublin Websummit 2014 AWS Technical Evangelist Ian Massingham discusses the major trends that are changing the gaming market today and how using the cloud as a development and delivery platform for gaming products and services can help meet the challenges that these trends present.
Includes examples of gaming customers running on the AWS cloud today as well as a discussion of how you might build and scaling a gaming back-end on AWS using AWS services to enable low cost and pain free scaling of your gaming infrastructure.
Innovation dank DevOps (DevOpsCon Berlin 2015)Wooga
“You build it, you run it!” - Wenn Du als Entwickler weisst, dass Du Deine Software selbst betreiben musst, was bist bereit zu tun, um den späteren Betrieb zu vereinfach?
Bei Wooga haben Dutzende von Teams ihre eigene Antwort auf die Frage gesucht und dabei von den Erfahrungen der anderen Teams gelernt. Herausgekommen ist ein großes Experimentierfeld beim Betrieb von Web Services - und eine technologische Innovation, die uns innerhalb weniger Iterationen von einem simplen LAMP-Stack zu lastabhängig skalierenden stateful Servern auf Basis von Erlang oder Akka gebracht hat.
Apache Spark v3 is a new milestone for the Big Data framework. In this session, you will (re)discover what Spark is, learn about the new features in its third major version, and go through a complete end-to-end project.
I like to call Spark an Analytics Operating Systems. It is offering far more than just a framework or a library. I will explain why. Spark v3 is the latest major evolution. It was released mid-June 2020 and adds impressive new features. After looking at them from a high level, I will detail a few of my favorites.
Finally, as we all like code (well, at least I do), I will demonstrate a complete data & AI pipeline looking at Covid-19 data.
Key takeaways: Spark as an Analytics OS, Spark v3 highlights, building data/AI pipelines/models with Spark.
Audience: software engineers, data engineers, architects, data scientists.
How to Build a Big Data Application: Serverless Editionecobold
Come learn how to build, launch, and scale a Big Data application in a serverless context. This is going to be an information packed meetup around Big Data processing, Lambda functions, Lambda Step functions, and everything that ties them together.
Big Data is something we're very passionate about. As the cost of servers have come down and the cost of software has become free, using data to drive your business has become much more obtainable to a larger group of companies. The serverless methodology has recently come in the scene, and it's proving to be just as transformational as cloud has been to the Big Data analytics space. We will be sharing some of our learnings and experiences over the last two years of working with Big Data in a serverless context. We will cover one or two examples of eventful Big Data processing, and the impact it can have on your business in terms of speed of analytics and cost savings to the bottom line.
Introducing Amazon RDS for PostgreSQL (DAT210) | AWS re:Invent 2013Amazon Web Services
AWS customers have been asking us for Amazon RDS for PostgreSQL, and we’re excited to announce its immediate availability. Learn how you can offload the management of your PostgreSQL database instances to Amazon RDS using automated backups and point-in-time recovery, Multi-AZ deployments for high availability, and provisioned IOPS for fast and predictable performance. Also learn how to take advantage of familiar PostgreSQL features such as PostGIS with Amazon RDS for PostgreSQL.
Geosophic is a platform of online services for mobile games. It offers both user engagement features (such as leaderboards and achievements) and analytics.
In this talk I would like to share with you our experience building this platform with Grails with the goal to support a big load. I’ll focus on the system architecture, the problems we have found and how we are solving them.
A paper introduction of "Fabian M. Suchanek, Serge Abiteboul, Pierre Senellart: PARIS: Probabilistic Alignment of Relations, Instances, and Schema. PVLDB 5(3): 157-168 (2011)" at SWIM seminar.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.