Do web services written in Clojure perform as well as those written in Java? We put this modest 5 node cluster to the load test on EC2 just to see what it could take.
Максим Барышиков-«WoT: Geographically distributed cluster of clusters»Tanya Denisyuk
"World of Tanks — MMOG с зарегистрированным пиковым онлайном превышающем 1 миллион пользователей. Авторитарная архитектура сервера (весь расчет физического мира происходит на стороне сервера и затем транслируется подклчюенным клиентам) позволяет эффективно защититься от нечестной игры, но вместе с тем создает значительную дополнительную нагрузку на серверную инфраструктуру.
В докладе рассказывается об инженерных решениях, принятых на разных стадиях разработки игры, их последствиях, и о том, как в конечном итоге организован географически распределенный «кластер кластеров» World of Tanks."
Putting the Go in MongoDB: How We Rebuilt The MongoDB Tools in GoMongoDB
With the release of MongoDB 3.0, the tools (mongodump, mongoimport, mongotop, etc) have been completely re-designed and re-written in Go to improve maintainability and performance. In this section, we'll give an architectural overview of the tools, describe how we used Go's native capacities to improve their parallelism, and also take a deep technical dive into their internals. We'll discuss performance, usability and integration improvements and share advanced techniques for power users. With a better understanding of how the tools work, you should feel comfortable effectively using and contributing to the tools.
Atmosphere 2014: Centralized log management based on Logstash and Kibana - ca...PROIDEA
Nowadays cloud enviroments are primary platform for applications. We no longer have multipurpose machines, rather multiple smaller virtual servers with dedicated roles. Therefore there is a need to have one place where we can manage applications and system logs. I wish to share my experience gained while building centralized log managment system using Nxlog, Logstash and Kibana. With that tools we are building cost effective and scalable log managment platform.
Dariusz Eliasz - Works in Allegro Group as a Solution Architect and is responsible for organizing cooperation with infrastructure teams, also leads some of the infrastructure projects. Earlier as an Expert System Administratorhe was related with building and maintaining the infrastructure shared services (i.e. image hosting platform) within Allegro Group.
Most database products have their own auditing functionalities or plugins but they always involve overhead which means they end up having them turned off or with the bare minimum enabled.
In this workshop we will show how to get reliable logging for mysql and mongodb servers in a scalable and non intrusive way, its drawbacks and how we can build our own open source tools to achieve results similar to most commercial products.
Tools to sniff, process and act upon queries will be shared and we will show how simple is to set up and monitor a database environment so it can be replicated and grow horizontally. All the code needed will be published.
Presentation at NYC Storm Meetup #1 on the Kafka-Storm implementation used in production at Outbrain Engage to track thousands of web traffic pings per second.
Максим Барышиков-«WoT: Geographically distributed cluster of clusters»Tanya Denisyuk
"World of Tanks — MMOG с зарегистрированным пиковым онлайном превышающем 1 миллион пользователей. Авторитарная архитектура сервера (весь расчет физического мира происходит на стороне сервера и затем транслируется подклчюенным клиентам) позволяет эффективно защититься от нечестной игры, но вместе с тем создает значительную дополнительную нагрузку на серверную инфраструктуру.
В докладе рассказывается об инженерных решениях, принятых на разных стадиях разработки игры, их последствиях, и о том, как в конечном итоге организован географически распределенный «кластер кластеров» World of Tanks."
Putting the Go in MongoDB: How We Rebuilt The MongoDB Tools in GoMongoDB
With the release of MongoDB 3.0, the tools (mongodump, mongoimport, mongotop, etc) have been completely re-designed and re-written in Go to improve maintainability and performance. In this section, we'll give an architectural overview of the tools, describe how we used Go's native capacities to improve their parallelism, and also take a deep technical dive into their internals. We'll discuss performance, usability and integration improvements and share advanced techniques for power users. With a better understanding of how the tools work, you should feel comfortable effectively using and contributing to the tools.
Atmosphere 2014: Centralized log management based on Logstash and Kibana - ca...PROIDEA
Nowadays cloud enviroments are primary platform for applications. We no longer have multipurpose machines, rather multiple smaller virtual servers with dedicated roles. Therefore there is a need to have one place where we can manage applications and system logs. I wish to share my experience gained while building centralized log managment system using Nxlog, Logstash and Kibana. With that tools we are building cost effective and scalable log managment platform.
Dariusz Eliasz - Works in Allegro Group as a Solution Architect and is responsible for organizing cooperation with infrastructure teams, also leads some of the infrastructure projects. Earlier as an Expert System Administratorhe was related with building and maintaining the infrastructure shared services (i.e. image hosting platform) within Allegro Group.
Most database products have their own auditing functionalities or plugins but they always involve overhead which means they end up having them turned off or with the bare minimum enabled.
In this workshop we will show how to get reliable logging for mysql and mongodb servers in a scalable and non intrusive way, its drawbacks and how we can build our own open source tools to achieve results similar to most commercial products.
Tools to sniff, process and act upon queries will be shared and we will show how simple is to set up and monitor a database environment so it can be replicated and grow horizontally. All the code needed will be published.
Presentation at NYC Storm Meetup #1 on the Kafka-Storm implementation used in production at Outbrain Engage to track thousands of web traffic pings per second.
As service providers and primary code contributors in the Islandora Community, discoverygarden encounters customers who are ingesting, accessing, and storing high volumes of data. For example, a customer who had 150,000 objects in 2012 now has three million objects and expectations to grow to five million in the very short term. This is increasingly common.
As repositories grow in size they can encounter poor performance, particularly during large ingests and derivative generation. To accommodate growing repositories caching mechanisms, infrastructure changes, and code updates are necessary.
The presentation will explore customer case studies that demonstrate interim solutions and the extensive, ongoing research and development to find long-term solutions.
Security Monitoring for big Infrastructures without a Million Dollar budgetJuan Berner
Nowadays in an increasingly more complex and dynamic network its not enough to be a regex ninja and storing only the logs you think you might need. From network traffic to custom logs you won't know which logs will be crucial to stop the next attacker, and if you are not planning to spend a half of your security budget in a commercial solution we will show you a way to building you own SIEM with open source. The talk will go from how to build a powerful logging environment for your organization to scaling on the cloud and storing everything forever. We will walk through how to build such a system with open source solutions as Elasticsearch and Hadoop, and creating your own custom monitoring rules to monitor everything you need. The talk will also include how to secure the environment and allow restricted access to other teams as well as avoiding common pitfalls and ensuring compliance standards.
Eko10 workshop - OPEN SOURCE DATABASE MONITORINGPablo Garbossa
Most database products have their own auditing functionalities or plugins but they always involve overhead which means they end up having them turned off or with the bare minimum enabled.
In this workshop we will show how to get reliable logging for mysql and mongodb servers in a scalable and non intrusive way, its drawbacks and how we can build our own open source tools to achieve results similar to most commercial products.
Tools to sniff, process and act upon queries will be shared and we will show how simple is to set up and monitor a database environment so it can be replicated and grow horizontally. All the code needed will be published.
Solving some of the scalability problems at booking.comIvan Kruglov
Booking.com uses Sereal in many applications. One of the biggest use case though is the events pipeline. It was built to delivery messages (events) from generation point to various processors in near real-time fashion. These days it servers billions of messages per day.
One of our processors recently faced scalability issues due to growth of the volume of delivered events.
In this talk I would like to share what problem we had, how we addressed it and which new features of Sereal helped us.
Containers are the future for all microservice based apps. Where do you deploy them? How do you manage them? At Digital Ocean we went through growing pains of trying out 5 of the top major Docker container schedulers, Mesos, Kubernetes, Docker Swarm, Nomad and we even tried manual scheduling of containers. Let us walk you through how we chose different schedulers for different applications, and tips and tricks for choosing a scheduler to use.
The open source motion control software Machinekit has excellent Python bindings for different API. Machinetalk, the middleware stack, pymachinetalk, the client API for Machinetalk and the Python configuration API.
Learn more at https://machinekoder.com/ and http://machinekit.io
My Learnings on Setting up a Kubernetes Cluster on AWS using Kubernetes Opera...Sathyajith Bhat
I recently setup a Kubernetes cluster on Amazon Web Services(AWS) using Kubernetes Operations(KOPS). Here's some of my findings and learnings from a out-of-box-experience of setting up a Kubernetes cluster
FaaS you like it (if Shakespeare had written Functions-as-a-Service)Ewan Slater
My slides from the London Cloud Native Meetup on 7th August 2018.
Covering serverless, FaaS (Functions-as-a-Service) and the Fn project (with a bit of help from William Shakespeare).
As service providers and primary code contributors in the Islandora Community, discoverygarden encounters customers who are ingesting, accessing, and storing high volumes of data. For example, a customer who had 150,000 objects in 2012 now has three million objects and expectations to grow to five million in the very short term. This is increasingly common.
As repositories grow in size they can encounter poor performance, particularly during large ingests and derivative generation. To accommodate growing repositories caching mechanisms, infrastructure changes, and code updates are necessary.
The presentation will explore customer case studies that demonstrate interim solutions and the extensive, ongoing research and development to find long-term solutions.
Security Monitoring for big Infrastructures without a Million Dollar budgetJuan Berner
Nowadays in an increasingly more complex and dynamic network its not enough to be a regex ninja and storing only the logs you think you might need. From network traffic to custom logs you won't know which logs will be crucial to stop the next attacker, and if you are not planning to spend a half of your security budget in a commercial solution we will show you a way to building you own SIEM with open source. The talk will go from how to build a powerful logging environment for your organization to scaling on the cloud and storing everything forever. We will walk through how to build such a system with open source solutions as Elasticsearch and Hadoop, and creating your own custom monitoring rules to monitor everything you need. The talk will also include how to secure the environment and allow restricted access to other teams as well as avoiding common pitfalls and ensuring compliance standards.
Eko10 workshop - OPEN SOURCE DATABASE MONITORINGPablo Garbossa
Most database products have their own auditing functionalities or plugins but they always involve overhead which means they end up having them turned off or with the bare minimum enabled.
In this workshop we will show how to get reliable logging for mysql and mongodb servers in a scalable and non intrusive way, its drawbacks and how we can build our own open source tools to achieve results similar to most commercial products.
Tools to sniff, process and act upon queries will be shared and we will show how simple is to set up and monitor a database environment so it can be replicated and grow horizontally. All the code needed will be published.
Solving some of the scalability problems at booking.comIvan Kruglov
Booking.com uses Sereal in many applications. One of the biggest use case though is the events pipeline. It was built to delivery messages (events) from generation point to various processors in near real-time fashion. These days it servers billions of messages per day.
One of our processors recently faced scalability issues due to growth of the volume of delivered events.
In this talk I would like to share what problem we had, how we addressed it and which new features of Sereal helped us.
Containers are the future for all microservice based apps. Where do you deploy them? How do you manage them? At Digital Ocean we went through growing pains of trying out 5 of the top major Docker container schedulers, Mesos, Kubernetes, Docker Swarm, Nomad and we even tried manual scheduling of containers. Let us walk you through how we chose different schedulers for different applications, and tips and tricks for choosing a scheduler to use.
The open source motion control software Machinekit has excellent Python bindings for different API. Machinetalk, the middleware stack, pymachinetalk, the client API for Machinetalk and the Python configuration API.
Learn more at https://machinekoder.com/ and http://machinekit.io
My Learnings on Setting up a Kubernetes Cluster on AWS using Kubernetes Opera...Sathyajith Bhat
I recently setup a Kubernetes cluster on Amazon Web Services(AWS) using Kubernetes Operations(KOPS). Here's some of my findings and learnings from a out-of-box-experience of setting up a Kubernetes cluster
FaaS you like it (if Shakespeare had written Functions-as-a-Service)Ewan Slater
My slides from the London Cloud Native Meetup on 7th August 2018.
Covering serverless, FaaS (Functions-as-a-Service) and the Fn project (with a bit of help from William Shakespeare).
Opportunity Eye is an initiative that focuses on a new class of applications known as personal assistant software. Our first project is a personal assistant for bloggers. This slide deck is an introduction to that demo.
In this modern world of global competition and modular innovation, leading edge best-of-class companies seek new ways to gain competitive advantage on a budget by adopting the distributed work environment. That is why it is in your best interest to learn how this smarter version of outsourcing is a viable alternative to having a lot of knowledge workers on payroll.
Logging at OVHcloud :
Logs Data platform est la plateforme de collecte, d'analyse et de gestion centralisée de logs d'OVHcloud. Cette plateforme a pour but de répondre aux challenges que constitue l'indexation de plus de 4000 milliards de logs par une entreprise comme OVHcloud. Cette présentation vous décrira l'architecture générale de Logs Data Platform autour de ses composants centraux Elasticsearch et Graylog et vous décrira les différentes problématiques de scalabilité, disponibilité, performance et d'évolutivité qui sont le quotidien de l'équipe Observability à OVHcloud.
USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a MonthNicolas Brousse
TubeMogul grew from few servers to over two thousands servers and handling over one trillion http requests a month, processed in less than 50ms each. To keep up with the fast growth, the SRE team had to implement an efficient Continuous Delivery infrastructure that allowed to do over 10,000 puppet deployment and 8,500 application deployment in 2014. In this presentation, we will cover the nuts and bolts of the TubeMogul operations engineering team and how they overcome challenges.
Pavel Odintsov introduces FastNetMon DDoS prevention and how they migrated analytics to ClickHouse to handle large data volumes. Pavel is CTO and co-founder of FastNetMon LTD.
FastNetMon: https://fastnetmon.com/
Meetup: https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/events/282872933/
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)Kevin Lynch
In this presentation I talk about our motivation to converting our microservices to run on Kubernetes. I discuss many of the technical challenges we encountered along the way, including networking issues, Java issues, monitoring and alerting, and managing all of our resources!
Comparison between zookeeper, etcd 3 and other distributed coordination systemsImesha Sudasingha
This is a comparison between popular distributed coordination systems including zookeeper (which powers Apache Hadoop), etcd 3 (which powers Kubernetes), consul and hazelcast. This comparison was made in second half of 2016. Therefore, please note that some of these technologies have improved immensely over the time. Anyway, this presentation will provide an initial idea of each distributed coordination systems.
Protecting the Web at a scale using consul and Elk / Valentin Chernozemski (S...Ontico
HighLoad++ 2017
Зал «Мумбай», 7 ноября, 16:00
Тезисы:
http://www.highload.ru/2017/abstracts/3065.html
Brute-force attacks against web based applications are on the raise.
You will be presented with an architecture built on top of ELK (https://www.elastic.co/products) and consul (https://www.consul.io/) that is capable of reliably detecting, analysing and mitigating large scale brute-force attacks against Wordpress, Drupal, Magento and Joomla based web sites in near real time.
...
Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015Monal Daxini
Keystone - Processing over Half a Trillion events per day with 8 million events & 17 GB per second peaks, and at-least once processing semantics. We will explore in detail how we employ Kafka, Samza, and Docker at scale to implement a multi-tenant pipeline. We will also look at the evolution to its current state and where the pipeline is headed next in offering a self-service stream processing infrastructure atop the Kafka based pipeline and support Spark Streaming.
Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022StreamNative
Starting with version 2.10, the Apache ZooKeeper dependency has been eliminated and replaced with a pluggable framework that enables you to reduce the infrastructure footprint of Apache Pulsar by leveraging alternative metadata and coordination systems based on your deployment environment. In this talk, walk through the steps required to utilize the existing etcd service running inside Kubernetes to act as Pulsar's metadata store, thereby eliminating the need to run ZooKeeper entirely, leaving you with a Zookeeper-less Pulsar.
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...javier ramirez
QuestDB es una base de datos open source de alto rendimiento. Mucha gente nos comentaba que les gustaría usarla como servicio, sin tener que gestionar las máquinas. Así que nos pusimos manos a la obra para desarrollar una solución que nos permitiese lanzar instancias de QuestDB con provisionado, monitorización, seguridad o actualizaciones totalmente gestionadas.
Unos cuantos clusters de Kubernetes más tarde, conseguimos lanzar nuestra oferta de QuestDB Cloud. Esta charla es la historia de cómo llegamos ahí. Hablaré de herramientas como Calico, Karpenter, CoreDNS, Telegraf, Prometheus, Loki o Grafana, pero también de retos como autenticación, facturación, multi-nube, o de a qué tienes que decir que no para poder sobrevivir en la nube.
An introduction to Netty. A powerful framework to develop networking applications.
This is suppose to be followed as hands on training, as the exercises on the slides imply, but can be also used an introduction guidance.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
8. Latency Overall
●
Poor at the beginning – cold cache
●
Really good 15 minutes in
●
Bad spike near the end of the first hour
●
Steady performance in the second hour
13. Summary
●
Cloud Spike
●
Postgresql latency similar to Cassandra
●
Solr indexing twice as slow
●
Between 50 and 70 requests per second
●
Clojure just as fast as Java
●
Jetty performed well