Serverless technologies and capabilities are here and are accessible now more than ever.
The power of infinite scale and system capabilities has never been more accessible. This also affects traditional front end development as serverless technologies allow for easy construction of backend support for any frontend with ease and simplicity.
In this talk, we will demonstrate how to build a fully functional Graphql endpoint for FE applications using Apollo Server and Client libraries, utilizing different cloud providers. We will also demonstrate the usage of Servless.com framework to set up the required infrastructure as code to simplify and support this setup
The video of the presentation (Hebrew):
https://youtu.be/8ba4cpdtK-8
Debugging & Profiling of AWS Lambda: ServerlessConf - IOpipeErica Windisch
Exploring the debugging and profiling tools for AWS Lambda. We use the v8-profiler for NodeJS with IOpipe to see deep into a serverless application and trace function calls. https://www.iopipe.com
Serverless technologies and capabilities are here and are accessible now more than ever.
The power of infinite scale and system capabilities has never been more accessible. This also affects traditional front end development as serverless technologies allow for easy construction of backend support for any frontend with ease and simplicity.
In this talk, we will demonstrate how to build a fully functional Graphql endpoint for FE applications using Apollo Server and Client libraries, utilizing different cloud providers. We will also demonstrate the usage of Servless.com framework to set up the required infrastructure as code to simplify and support this setup
The video of the presentation (Hebrew):
https://youtu.be/8ba4cpdtK-8
Debugging & Profiling of AWS Lambda: ServerlessConf - IOpipeErica Windisch
Exploring the debugging and profiling tools for AWS Lambda. We use the v8-profiler for NodeJS with IOpipe to see deep into a serverless application and trace function calls. https://www.iopipe.com
In 2016 Paddy Power and Betfair, two gambling giants, merged to form PPB. Each company had its own monitoring baggage, but the SRE team was tasked with cleaning up and consolidating their toolsets. This Sensu Summit 2019 talk from Artur Malinowski and Killian McHale looks at their selection process, scoring and ultimately the decisions which led them to Sensu – which now monitors over 10,000 clients across the PPB estate.
Large Scale Kubernetes on AWS at Europe's Leading Online Fashion Platform - C...Henning Jacobs
Bootstrapping a Kubernetes cluster is easy, rolling it out to nearly 200 engineering teams and operating it at scale is a challenge. In this talk, we are presenting our approach to Kubernetes provisioning on AWS, operations and developer experience for our growing Zalando Technology department. We will highlight in the context of Kubernetes: AWS service integrations, our IAM/OAuth infrastructure, cluster autoscaling, continuous delivery and general developer experience. The talk will cover our most important learnings and we will openly share failure stories.
Talk given at Container Days HH (https://containerdays.io/) on 2017-06-20.
The HERE Mobility Marketplace connects businesses & consumers with different types of transportation services.
Transporting a passenger from point A to B requires many moving parts. Multiple data points flow constantly between mobile devices & backend services.
In order to make smarter decisions we needed realtime answers to questions like:
What is the current status of a ride?
Where are the driver and passenger located?
Why didn't the driver show up?
Why was the ride cancelled?
And more...
I would like to share with you how we addressed this challenge using Flink to gain better visibility, answering those questions, for Customer Support, BI and Data Science teams.
I'll describe what issues we faced and how we solved them.
VR API Evolution, Generic Interfaces, and FactoringRyan A. Pavlik
Presented in July of 2015.
Summarizes a number of "developmental stages" seen in the wild and in academic/industrial research with VR APIs (not naming names!) and their associated, varying levels of hardware abstraction. Discusses the concept of "factoring" devices into generic interfaces and guidelines used in the OSVR project.
This is a Lightning Talk exploring the value of bringing Serverless development to your operations and developer teams, and how it fits with DevOps culture.
This talk will offer an introduction to Serverless applications, practical options for replacing infrastructure with Serverless, and will explore developer culture.
Keynote: Sensu as a multi-cloud monitoring control planeSensu Inc.
In this Sensu Summit 2019 talk, Sensu CEO Caleb Hailey explores how Sensu provides a “monitoring control plane” that is analogous to the Kubernetes control plane, and how Sensu can empower organizations by delivering consistent monitoring workflows in multi-cloud environments.
From AWS/STUPS to Kubernetes on AWS @Zalando - Berlin Kubernetes MeetupHenning Jacobs
This talk will highlight our challenges while migrating from our STUPS infrastructure (Docker on EC2, Cloud Formation) to Kubernetes on AWS.
Talk was held at Berlin Kubernetes Meetup on 2017-05-18: https://www.meetup.com/Berlin-Kubernetes-Meetup/events/239313998/
This presentation deals with logging in the course of mobile development, namely describing the open source logging environment built with ELK stack (ElasticSearch, Logstash and Kibana).
Presentation by Igor Rudyk (Software Engineer, GlobalLogic, Lviv), delivered at Mobile TechTalk Lviv on April 28, 2015.
More details - http://globallogic.com.ua/mobile-techtalk-lviv-2015-report
In this Sensu Summit 2018 talk. CTO Sean Porter shares Sensu's beliefs as a company and community, how far we’ve come in the past year, the problems we face together, what we’re doing next to solve for them.
Sandstone HPC: A Domain General Gateway for New UsersZebula Sampedro
The complexity of high-performance computing (HPC) resources poses many challenges to new users. A number of science gateways have been developed to increase the productivity of novice users by hiding the underlying infrastructure, however these solutions tend not to teach HPC skills that transfer easily outside of the gateway. In this paper we introduce a domain-general gateway, Sandstone HPC, that represents the HPC environment more naturally to novice users by abstracting the command-line interface and providing contextual help. We assess the degree to which Sandstone HPC improves upon the usability of the command-line interface by analyzing the results of a usability study conducted on both environments. We will also detail how the architecture promotes long-term sustainability and a community-development model.
Kubernetes on AWS @Zalando - Berlin AWS User Group 2017-05-09Henning Jacobs
In this talk we share our learnings from running Kubernetes on AWS in production and how we are migrating 200+ engineering teams from AWS/STUPS to Kubernetes.
This talk was given at the Berlin AWS User Group meetup on 2017-05-09 hosted by NewStore (https://www.meetup.com/aws-berlin/events/236795816/).
More information on http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html
What’s new in Microsoft ALM 2013, hosted in Windows Azure, VISUAL STUDIO ONLI...VISEO
What’s new in Microsoft ALM 2013, the parts that is 100% running on Windows Azure, now called VISUAL STUDIO ONLINE !!
by Vincent Thavonekham, Objet Direct
In 2016 Paddy Power and Betfair, two gambling giants, merged to form PPB. Each company had its own monitoring baggage, but the SRE team was tasked with cleaning up and consolidating their toolsets. This Sensu Summit 2019 talk from Artur Malinowski and Killian McHale looks at their selection process, scoring and ultimately the decisions which led them to Sensu – which now monitors over 10,000 clients across the PPB estate.
Large Scale Kubernetes on AWS at Europe's Leading Online Fashion Platform - C...Henning Jacobs
Bootstrapping a Kubernetes cluster is easy, rolling it out to nearly 200 engineering teams and operating it at scale is a challenge. In this talk, we are presenting our approach to Kubernetes provisioning on AWS, operations and developer experience for our growing Zalando Technology department. We will highlight in the context of Kubernetes: AWS service integrations, our IAM/OAuth infrastructure, cluster autoscaling, continuous delivery and general developer experience. The talk will cover our most important learnings and we will openly share failure stories.
Talk given at Container Days HH (https://containerdays.io/) on 2017-06-20.
The HERE Mobility Marketplace connects businesses & consumers with different types of transportation services.
Transporting a passenger from point A to B requires many moving parts. Multiple data points flow constantly between mobile devices & backend services.
In order to make smarter decisions we needed realtime answers to questions like:
What is the current status of a ride?
Where are the driver and passenger located?
Why didn't the driver show up?
Why was the ride cancelled?
And more...
I would like to share with you how we addressed this challenge using Flink to gain better visibility, answering those questions, for Customer Support, BI and Data Science teams.
I'll describe what issues we faced and how we solved them.
VR API Evolution, Generic Interfaces, and FactoringRyan A. Pavlik
Presented in July of 2015.
Summarizes a number of "developmental stages" seen in the wild and in academic/industrial research with VR APIs (not naming names!) and their associated, varying levels of hardware abstraction. Discusses the concept of "factoring" devices into generic interfaces and guidelines used in the OSVR project.
This is a Lightning Talk exploring the value of bringing Serverless development to your operations and developer teams, and how it fits with DevOps culture.
This talk will offer an introduction to Serverless applications, practical options for replacing infrastructure with Serverless, and will explore developer culture.
Keynote: Sensu as a multi-cloud monitoring control planeSensu Inc.
In this Sensu Summit 2019 talk, Sensu CEO Caleb Hailey explores how Sensu provides a “monitoring control plane” that is analogous to the Kubernetes control plane, and how Sensu can empower organizations by delivering consistent monitoring workflows in multi-cloud environments.
From AWS/STUPS to Kubernetes on AWS @Zalando - Berlin Kubernetes MeetupHenning Jacobs
This talk will highlight our challenges while migrating from our STUPS infrastructure (Docker on EC2, Cloud Formation) to Kubernetes on AWS.
Talk was held at Berlin Kubernetes Meetup on 2017-05-18: https://www.meetup.com/Berlin-Kubernetes-Meetup/events/239313998/
This presentation deals with logging in the course of mobile development, namely describing the open source logging environment built with ELK stack (ElasticSearch, Logstash and Kibana).
Presentation by Igor Rudyk (Software Engineer, GlobalLogic, Lviv), delivered at Mobile TechTalk Lviv on April 28, 2015.
More details - http://globallogic.com.ua/mobile-techtalk-lviv-2015-report
In this Sensu Summit 2018 talk. CTO Sean Porter shares Sensu's beliefs as a company and community, how far we’ve come in the past year, the problems we face together, what we’re doing next to solve for them.
Sandstone HPC: A Domain General Gateway for New UsersZebula Sampedro
The complexity of high-performance computing (HPC) resources poses many challenges to new users. A number of science gateways have been developed to increase the productivity of novice users by hiding the underlying infrastructure, however these solutions tend not to teach HPC skills that transfer easily outside of the gateway. In this paper we introduce a domain-general gateway, Sandstone HPC, that represents the HPC environment more naturally to novice users by abstracting the command-line interface and providing contextual help. We assess the degree to which Sandstone HPC improves upon the usability of the command-line interface by analyzing the results of a usability study conducted on both environments. We will also detail how the architecture promotes long-term sustainability and a community-development model.
Kubernetes on AWS @Zalando - Berlin AWS User Group 2017-05-09Henning Jacobs
In this talk we share our learnings from running Kubernetes on AWS in production and how we are migrating 200+ engineering teams from AWS/STUPS to Kubernetes.
This talk was given at the Berlin AWS User Group meetup on 2017-05-09 hosted by NewStore (https://www.meetup.com/aws-berlin/events/236795816/).
More information on http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html
What’s new in Microsoft ALM 2013, hosted in Windows Azure, VISUAL STUDIO ONLI...VISEO
What’s new in Microsoft ALM 2013, the parts that is 100% running on Windows Azure, now called VISUAL STUDIO ONLINE !!
by Vincent Thavonekham, Objet Direct
So, you have IoT Devices connected to IoT Hub sending telemetry data into the Microsoft Azure cloud. Now what? This session will take you through setting up real-time stream processing of IoT data. We’ll look at integrating services like Azure Stream Analytics, Azure Functions, and Cosmos DB to build a highly scalable stream processing backend for any IoT solution. You’ll leave this session better prepared to handle real-time IoT stream processing in Azure; plus you’ll do it with less code by utilizing serverless Azure Functions.
Provisioning Windows instances at scale on Azure, AWS and OpenStack - Adrian ...ITCamp
In a cloud based environment, where automation is a primary concern, guest operating systems need to be provisioned at boot time.
There are a lot of actions that need to be performed at this stage, ranging from assigning the admin user’s credentials to creating WinRM listeners, storage configurations, RDP settings, guest agent installation, custom data execution and much more.
The de-facto standard guest provisioning tools are cloud-init on Linux and cloudbase-init on Windows.
I will present how cloudbase-init runs on all the Microsoft supported Windows editions (there are quite a few) and how it supports a plethora of metadata service implementations (EC2, OpenStack, the recently added Azure).
Cloudbase-init is being run thousands of times daily all over the world’s public clouds and data centers and it has reached more than 5 million known runs to date.
We will also take an in-depth look at the Argus integration testing framework, which automates the integration testing of cloudbase-init on real world platforms, to make sure it meets a very strict set of performance, compatibility and security requirements.
At the end I will show you a live demo with a cloudbase-init bootstrapped Windows instance on Azure, and how you can benefit from the provisioning process.
IBM Think Session 8598 Domino and JavaScript Development MasterClassPaul Withers
Session from IBM Think 2018. Note: the architecture used is an extreme case of what's possible (and it could go further), rather than a real-world expectation
Presto: Query Anything - Data Engineer’s perspectiveAlluxio, Inc.
Data Orchestration Summit
www.alluxio.io/data-orchestration-summit-2019
November 7, 2019
Presto: Query Anything - Data Engineer’s perspective
Speakers:
Kamil Bajda-Pawlikowski, Starburst, Presto Company
Martin Traverso, Presto Software Foundation
For more Alluxio events: https://www.alluxio.io/events/
nuclio is iguazio's open source serverless project. nuclio is 100x faster, brings significant new functionality and works with data and event sources to accelerate performance and development.
【BS1】What’s new in visual studio 2022 and c# 10日本マイクロソフト株式会社
Watch this session to learn about the new capabilities in Visual Studio 2022 and the new C# 10 features for .NET developers. With Visual Studio 2022, you'll always get the best-in-class tools and services available for any developer, any app, and any platform. Whether you're using Visual Studio for the first time, or you've been using it for years, there's a lot to like in our newest version.
.NET 6 is the latest version of .NET that was released in Nov 2021. .NET 6 ecosystem offers simplifications on development, high performance, and ultimate productivity.
Cloud Workstations provides preconfigured, customizable, and secure managed development environments on Google Cloud. Cloud Workstations is accessible through a browser-based IDE, from multiple local code editors (such as IntelliJ IDEA Ultimate or VS Code), or through SSH. Instead of manually setting up development environments, you can create a workstation configuration specifying your environment in a reproducible way
Industrial IoT with Azure and Open SourceIvo Andreev
IIoT leverages the power of machines and realtime analytics to pick up on industrial inefficiencies and problems sooner, and save time and money in addition to supporting BI efforts. In a myriad of reference architectures it is up to experience and trial-error to find out what really works in a real life scenario..
We will review the challenges and solutions in building an IIoT platform from the ground up on the edge between Azure and open source in order to have the best from both worlds. Technical focus will be on IoT Edge, TS Insights, Stream Analytics, IoT Hub, App Insights, Event Grid, Service Bus, ARM templates, Influx DB, Grafana and more - all neatly glued together by Azure Functions.
Cloud State of the Union for Java DevelopersBurr Sutter
This presentation provides a broad overview of what is going on in the Cloud computing world - for Java developers - presented on Dec 21st 2010 at the Atlanta Java Users Group - ajug.org - no audio was recorded.
Power Platform: AI Builder la democratizzazione di AIAlessio Biasiutti
L'utilizzo dell'intelligenza artificiale fa crescere molto di più i fatturati delle aziende che la utilizzano rispetto a quelle che non lo stanno facendo. La mancanza di competenze in azienda è quindi un freno alla crescita. AI Builder è la risposta code-less alla creazione di modelli di AI
Come costruire un PoC IoT completo: gestione allarmi, device heartbeat, real-time. L'intera soluzione sarà costruita utilizzando la piattaforma Cloud Microsoft (Stream Analytics, Logic Apps, SQL Database, Power BI, Dynamics 365).
Il tutto in 1 ora e come sempre scrivendo meno codice possibile!
è possible rinoscere e contare oggetti inquadrati con una telecamera connessa ad Rasperry PI e visualizzare i risultati in una dashbaord realtime di Power BI? Certo che sì anche se non siete dei data scientist!
Structured Streaming è il modulo di Stream Processing costruito sul motore Spark SQL. In poche parole garantisce l'esecuzione di un messaggio esattamente una volta, è scalabile e fault-tolerant. È possibile definire le analisi stream nello stesso modo in cui si definirebbe un calcolo batch sui dati usando i Dataset/DataFrame API in Scala, Java, Python or R utilizzando l'engine SQL di Spark.
Durante la sessione vedremo un'overview delle funzionalità e un esempio di di come sia possibile eseguire l'ingestion dei dati con Event Hub (Kafka enabled) eseguire un'analisi con Spark e salvare i risultati su Cosmos DB.
Azure Stream Analytics è un motore serverless che permette l'analisi dei dati in tempo reale di più flussi simultanei. Durante la sessione vedremo un esempio di come sia possibile determinare situazioni di allarme e notificarle via email.
Costruire un PoC IoT completo: gestione allarmi, device heartbeat, real-time e batch analysis e integrazione con Dynamics 365. L'intera soluzione sarà costruita utilizzando la piattaforma Cloud Microsoft (Stream Analytics, Logic Apps, SQL Database, Power BI, Dynamics 365).
Il tutto in 1 ora e come sempre scrivendo meno codice possibile!
Azure Saturday Pordenone 2018 IoT real time processing for lazy developerAlessio Biasiutti
In this session I explained how to develop a IoT E2E solution writing as less code as possible. I designed the solution using Azure Event Hub, Azure Service Bus, Azure Logic Apps e Azure SQL Database
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Discussion on Vector Databases, Unstructured Data and AI
https://www.meetup.com/unstructured-data-meetup-new-york/
This meetup is for people working in unstructured data. Speakers will come present about related topics such as vector databases, LLMs, and managing data at scale. The intended audience of this group includes roles like machine learning engineers, data scientists, data engineers, software engineers, and PMs.This meetup was formerly Milvus Meetup, and is sponsored by Zilliz maintainers of Milvus.
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeWalaa Eldin Moustafa
Dynamic policy enforcement is becoming an increasingly important topic in today’s world where data privacy and compliance is a top priority for companies, individuals, and regulators alike. In these slides, we discuss how LinkedIn implements a powerful dynamic policy enforcement engine, called ViewShift, and integrates it within its data lake. We show the query engine architecture and how catalog implementations can automatically route table resolutions to compliance-enforcing SQL views. Such views have a set of very interesting properties: (1) They are auto-generated from declarative data annotations. (2) They respect user-level consent and preferences (3) They are context-aware, encoding a different set of transformations for different use cases (4) They are portable; while the SQL logic is only implemented in one SQL dialect, it is accessible in all engines.
#SQL #Views #Privacy #Compliance #DataLake
State of Artificial intelligence Report 2023kuntobimo2016
Artificial intelligence (AI) is a multidisciplinary field of science and engineering whose goal is to create intelligent machines.
We believe that AI will be a force multiplier on technological progress in our increasingly digital, data-driven world. This is because everything around us today, ranging from culture to consumer products, is a product of intelligence.
The State of AI Report is now in its sixth year. Consider this report as a compilation of the most interesting things we’ve seen with a goal of triggering an informed conversation about the state of AI and its implication for the future.
We consider the following key dimensions in our report:
Research: Technology breakthroughs and their capabilities.
Industry: Areas of commercial application for AI and its business impact.
Politics: Regulation of AI, its economic implications and the evolving geopolitics of AI.
Safety: Identifying and mitigating catastrophic risks that highly-capable future AI systems could pose to us.
Predictions: What we believe will happen in the next 12 months and a 2022 performance review to keep us honest.
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...sameer shah
"Join us for STATATHON, a dynamic 2-day event dedicated to exploring statistical knowledge and its real-world applications. From theory to practice, participants engage in intensive learning sessions, workshops, and challenges, fostering a deeper understanding of statistical methodologies and their significance in various fields."
Analysis insight about a Flyball dog competition team's performanceroli9797
Insight of my analysis about a Flyball dog competition team's last year performance. Find more: https://github.com/rolandnagy-ds/flyball_race_analysis/tree/main
5. CONCEPTS
▪ Event-processing engine
▪ In memory cluster
▪ PaaS - Serverless
▪ SQL Like programming
language
▪ JavaScript per UDF/UDA
▪ Availability 99,9%
▪ Scale-Up/Down
6. INPUT
EventHub
EventProcessedUtcTime, EventEnqueuedUtcTime, PartitionId
IoT Hub
IoTHub.MessageId, IoTHub.CorrelationId…..
BlobStorage
Blob vengono presi in considerazione ogni secondo e solo una volta
Pattern cluster1/logs/{date}/{time}/{partition}
Reference Data
Pattern products/{date}/{time}/product-list.csv
7. QUERY LANGUAGE
SELECT
Make,
System.TimeStamp AS Time,
COUNT(*) AS [Count]
INTO
AlertOutput
FROM
Input TIMESTAMP BY Time
GROUP BY
Make,
TumblingWindow(second, 10)
HAVING
[Count] >= 3
▪ Aggregate Functions (SUM, AVG…)
▪ Analytic Functions (LAG, LAST, ISFIRST…)
▪ Array Functions
▪ Conversion Functions (TRY_CAST, GetType)
▪ Date Time Functions (DATEADD, DATEDIFF, DAY, MONTH…)
▪ GeoSpatial Functions (ST_DISTANCE, ST_WITHIN…)
▪ Mathematical Functions (ABS, SIGN…)
▪ Record Functions (GetRecordPropertyValue…)
▪ String Functions (LEN, SUBSTRING, CONCAT…)
8. OUTPUT
Data Lake Store
SQL Database (NO SQL Server su VM)
Blob Storage
Table Storage
Cosmos DB
ServiceBus Queue / Topic
Event Hub
Functions
Power BI
17. STREAM ANALYTICS ON IOT EDGE
▪ Low-latency command and
control
▪ Limited connectivity to the
cloud
▪ Limited bandwidth
▪ Compliance
18. STREAM ANALYTICS ON IOT EDGE
▪ ASA on IoT Edge offers a cross-platform
solution running on Docker containers
Linux or Windows
▪ Multiple architectures (Intel or ARM
CPUs)
▪ Raspberry Pi
▪ Industrial PCs
▪ Dedicated field gateways
▪ Servers
20. QUERY LANGUAGE
▪ Most SQL query language features are already supported.
▪ Following feature are NOT yet supported:
▪ User-defined functions
▪ Azure ML functions
▪ Using more than 14 aggregates in a single step
▪ AVRO format for input/output
▪ Geospatial operators
▪ PARTITION BY
▪ GetMetadataPropertyValue